[How To] Back up your Server Data to Amazon S3

With the advent of the cloud computing, storage has become extremely cheap. We can use this storage to keep backups of our data and keep them safe from any potential data loss disaster. One such extremely cost-effective solution is Amazon S3. Some of the pros of S3 are:

  • The storage costs at S3 are very low
  • You have virtually unlimited storage capacity
  • You only pay for how much you use

In this “how to” we will see how to use Amazon S3 to backup your server data.

Pre-requisites:

  • Linux Server with root access (Assumed Debian based)
  • An Amazon S3 account i.e. S3 access key and S3 secret key (Sign Up at http://aws.amazon.com/s3/ to get it)

Applications/Softwares Used:

  • FUSE Libraries: Filesystem in User Space
  • S3FS: Filesystem based on Amazon S3
  • Rsync: Remote data synchronization tool

STEPS:

  1. Login to your server as root using SSH and install the pre-requisites and dependencies by keying the following command:
    Code   
    1. apt-get install build-essential libcurl4-openssl-dev libxml2-dev libfuse-dev comerr-dev libfuse2 libidn11-dev libkdb5-4 libkrb5-dev libldap2-dev libselinux1-dev libsepol1-dev pkg-config fuse-utils sshfs rsync mailutils
  2. Get a copy of the latest version of S3FS Source, extract it and install it:
    Code   
    1. wget -c http://s3fs.googlecode.com/files/s3fs-r191-source.tar.gz
    2. tar xvf s3fs-r191-source.tar.gz
    3. cd s3fs
    4. make&&make install
  3. To allow non-root users to the fuse drives open /etc/fuse.conf in your favorite editor, find and un-comment user_allow_other
  4. Create a bucket for your backups on S3 using the S3 console. In this case we created “server-backup-bucket” as our backup bucket.
  5. Create a mount point for our backups and mount our S3 drive:
    Code   
    1. mkdir -p /mnt/backups/
    2. /usr/bin/s3fs -o allow_other server-backup-bucket -o accessKeyId=XYZ -o secretAccessKey=ABC /mnt/backups/

    Now, the backup directory is mounted to /mnt/backups/.
  6. Create a shell script inside /opt/backup_scripts/ named mount-s3-backups which uses the above mount method:
    Code   
    1. #!/bin/bash
    2. IS_MOUNTED=`df | grep -c '/mnt/backups'` # Check if the drive is already mounted
    3. if [ "$IS_MOUNTED" -eq "0" ]; then
    4. /usr/bin/s3fs -o allow_other server-backup-bucketname -o accessKeyId=XYZ -o secretAccessKey=ABC /mnt/backups/
    5. echo "S3 Backup Drive mounted successfully!"
    6. else
    7. echo "S3 Backup Drive already mounted!"
    8. fi

    Make this script executable:
    Code   
    1. chmod +x /opt/backup_scripts/mount-s3-backups
  7. Setup a shell script in /opt/backup_scripts/s3-home-backup.sh. In this case I assume we are going to backup the /home/ directory:
    Code   
    1. #!/bin/bash
    2. modprobe fuse # Loads the Fuse Kernel module
    3. /opt/backup_scripts/mount-s3-backups # Mount S3 Backup directory if not mounted
    4. IS_MOUNTED=`df | grep -c '/mnt/backups'` # Check if the mount was successful
    5. if [ "$IS_MOUNTED" -eq "1" ]; then
    6. TIME_STARTED=`date`
    7. echo "Started Backup..."
    8. echo "Backing up to S3 Drive" > /opt/backup_scripts/tmp.log
    9. echo "START TIME: $TIME_STARTED" >> /opt/backup_scripts/tmp.log
    10. echo "-----------------------------------------" >> /opt/backup_scripts/tmp.log
    11. /usr/bin/rsync -avz --delete /home/ /mnt/backups/home/ >> /opt/backup_scripts/tmp.log # Rsync the /home directory to /mnt/backups/home --delete will ensure deleted file in the filesystem are deleted on the backup.
    12. TIME_ENDED=`date`
    13. echo "-----------------------------------------" >> /opt/backup_scripts/tmp.log
    14. echo "END TIME: $TIME_ENDED" >> /opt/backup_scripts/tmp.log
    15. echo "-----------------------------------------" >> /opt/backup_scripts/tmp.log
    16. mail logs@your-domain.com -s "[S3 Backups] Daily Home Backup Complete" < /opt/backup_scripts/tmp.log # Sends the logs of the backup to your email
    17. umount /mnt/backups # Unmount the Backups Directory
    18. echo "Backup complete!"
    19. else
    20. echo "S3Drive not Mounted" > /opt/backup_scripts/tmp.log
    21. mail logs@your-domain.com -s "[ERROR] S3Drive not Mounted!" < /opt/backup_scripts/tmp.log
    22. fi

    We are almost there. Now lets make the /opt/backup_scripts/s3-home-backup.sh executable and run it:
    Code   
    1. chmod +x /opt/backup_scripts/s3-home-backup.sh
    2. /opt/backup_scripts/s3-home-backup.sh

    If everything is OK, the script should run successfully and a mail should be delivered to logs@your-domain.com with the logs of the backup.
  8. For the last step lets add the script to crontab, open crontab in your favorite editor by running crontab -e and insert the following code:
    Code   
    1. @daily /opt/backup_scripts/s3-home-backup.sh

    The backups interval can be configured by changing the cron interval.

That’s it.. Eight simple steps to securely backup your server data on Amazon S3.

P.S. : Data Backup is a critical operation. Any script that is written must be thoroughly tested before running over the actual data to prevent data loss.

This entry was posted in How Tos and tagged , , , , , . Bookmark the permalink.

7 Responses to [How To] Back up your Server Data to Amazon S3

  1. Pingback: Tweets that mention [How To] Back up your Server Data to Amazon S3 | TechTatva -- Topsy.com

  2. Mujtaba Mir says:

    hi,
    thanks alot, for such a great tutorial.

    Dear what if i want biweekly or thrice a week backup. Mean what changes i have to do in the scirpt.
    Need to change the Start and time variable??? if yes then What to write instead of ‘date’ ?

    thanks in advance for help

  3. Darell says:

    I see you don’t use the potential of social websites like pinterest
    and facebook on your blog. You can get big traffic from social sites on autopilot using one useful app,
    for more info search in google for:
    Alufi’s Social Automation

  4. You did the a great work writing and revealing the hidden beneficial features of
    Ray Ban Wayfarer Knock Off http://www.forestjournal.jp/knockoff.php?pid=32

  5. fmvsggrakkmlotifndpsjnbeefcspaogtkob
    Knock Off Folding Ray Bans http://www.forestjournal.jp/knockoff.php?pid=139

  6. Quit smoking benefits If we see you smoking we will assume you’re on fire and take appropriate action.
    Knockoff Ray Ban Aviator Sunglasses http://www.forestjournal.jp/knockoff.php?pid=126

  7. A formidable share, I basically given this onto a colleague who was performing a bit of analysis on this. And he actually purchased me breakfast as a result of I discovered it for him.. smile. So let me reword that: Thnx for the treat! However yeah Thnkx for spending the time to discuss this, I feel strongly about it and love reading much more on this subject. If potential, as you grow to be expertise, would you mind updating your weblog with much more details? Its highly helpful for me. Huge thumb up for this blog submit!
    Ray Ban New Wayfarer Knockoffs http://www.forestjournal.jp/knockoff.php?pid=25

Leave a Reply

Your email address will not be published. Required fields are marked *