[How To] Back up your Server Data to Amazon S3

With the advent of the cloud computing, storage has become extremely cheap. We can use this storage to keep backups of our data and keep them safe from any potential data loss disaster. One such extremely cost-effective solution is Amazon S3. Some of the pros of S3 are:

  • The storage costs at S3 are very low
  • You have virtually unlimited storage capacity
  • You only pay for how much you use

In this “how to” we will see how to use Amazon S3 to backup your server data.

Pre-requisites:

  • Linux Server with root access (Assumed Debian based)
  • An Amazon S3 account i.e. S3 access key and S3 secret key (Sign Up at http://aws.amazon.com/s3/ to get it)

Applications/Softwares Used:

  • FUSE Libraries: Filesystem in User Space
  • S3FS: Filesystem based on Amazon S3
  • Rsync: Remote data synchronization tool

STEPS:

  1. Login to your server as root using SSH and install the pre-requisites and dependencies by keying the following command:
    Code   
    1. apt-get install build-essential libcurl4-openssl-dev libxml2-dev libfuse-dev comerr-dev libfuse2 libidn11-dev libkdb5-4 libkrb5-dev libldap2-dev libselinux1-dev libsepol1-dev pkg-config fuse-utils sshfs rsync mailutils
  2. Get a copy of the latest version of S3FS Source, extract it and install it:
    Code   
    1. wget -c http://s3fs.googlecode.com/files/s3fs-r191-source.tar.gz
    2. tar xvf s3fs-r191-source.tar.gz
    3. cd s3fs
    4. make&&make install
  3. To allow non-root users to the fuse drives open /etc/fuse.conf in your favorite editor, find and un-comment user_allow_other
  4. Create a bucket for your backups on S3 using the S3 console. In this case we created “server-backup-bucket” as our backup bucket.
  5. Create a mount point for our backups and mount our S3 drive:
    Code   
    1. mkdir -p /mnt/backups/
    2. /usr/bin/s3fs -o allow_other server-backup-bucket -o accessKeyId=XYZ -o secretAccessKey=ABC /mnt/backups/

    Now, the backup directory is mounted to /mnt/backups/.
  6. Create a shell script inside /opt/backup_scripts/ named mount-s3-backups which uses the above mount method:
    Code   
    1. #!/bin/bash
    2. IS_MOUNTED=`df | grep -c '/mnt/backups'` # Check if the drive is already mounted
    3. if [ "$IS_MOUNTED" -eq "0" ]; then
    4. /usr/bin/s3fs -o allow_other server-backup-bucketname -o accessKeyId=XYZ -o secretAccessKey=ABC /mnt/backups/
    5. echo "S3 Backup Drive mounted successfully!"
    6. else
    7. echo "S3 Backup Drive already mounted!"
    8. fi

    Make this script executable:
    Code   
    1. chmod +x /opt/backup_scripts/mount-s3-backups
  7. Setup a shell script in /opt/backup_scripts/s3-home-backup.sh. In this case I assume we are going to backup the /home/ directory:
    Code   
    1. #!/bin/bash
    2. modprobe fuse # Loads the Fuse Kernel module
    3. /opt/backup_scripts/mount-s3-backups # Mount S3 Backup directory if not mounted
    4. IS_MOUNTED=`df | grep -c '/mnt/backups'` # Check if the mount was successful
    5. if [ "$IS_MOUNTED" -eq "1" ]; then
    6. TIME_STARTED=`date`
    7. echo "Started Backup..."
    8. echo "Backing up to S3 Drive" > /opt/backup_scripts/tmp.log
    9. echo "START TIME: $TIME_STARTED" >> /opt/backup_scripts/tmp.log
    10. echo "-----------------------------------------" >> /opt/backup_scripts/tmp.log
    11. /usr/bin/rsync -avz --delete /home/ /mnt/backups/home/ >> /opt/backup_scripts/tmp.log # Rsync the /home directory to /mnt/backups/home --delete will ensure deleted file in the filesystem are deleted on the backup.
    12. TIME_ENDED=`date`
    13. echo "-----------------------------------------" >> /opt/backup_scripts/tmp.log
    14. echo "END TIME: $TIME_ENDED" >> /opt/backup_scripts/tmp.log
    15. echo "-----------------------------------------" >> /opt/backup_scripts/tmp.log
    16. mail logs@your-domain.com -s "[S3 Backups] Daily Home Backup Complete" < /opt/backup_scripts/tmp.log # Sends the logs of the backup to your email
    17. umount /mnt/backups # Unmount the Backups Directory
    18. echo "Backup complete!"
    19. else
    20. echo "S3Drive not Mounted" > /opt/backup_scripts/tmp.log
    21. mail logs@your-domain.com -s "[ERROR] S3Drive not Mounted!" < /opt/backup_scripts/tmp.log
    22. fi

    We are almost there. Now lets make the /opt/backup_scripts/s3-home-backup.sh executable and run it:
    Code   
    1. chmod +x /opt/backup_scripts/s3-home-backup.sh
    2. /opt/backup_scripts/s3-home-backup.sh

    If everything is OK, the script should run successfully and a mail should be delivered to logs@your-domain.com with the logs of the backup.
  8. For the last step lets add the script to crontab, open crontab in your favorite editor by running crontab -e and insert the following code:
    Code   
    1. @daily /opt/backup_scripts/s3-home-backup.sh

    The backups interval can be configured by changing the cron interval.

That’s it.. Eight simple steps to securely backup your server data on Amazon S3.

P.S. : Data Backup is a critical operation. Any script that is written must be thoroughly tested before running over the actual data to prevent data loss.

This entry was posted in How Tos and tagged , , , , , . Bookmark the permalink.

2 Responses to [How To] Back up your Server Data to Amazon S3

  1. Pingback: Tweets that mention [How To] Back up your Server Data to Amazon S3 | TechTatva -- Topsy.com

  2. Mujtaba Mir says:

    hi,
    thanks alot, for such a great tutorial.

    Dear what if i want biweekly or thrice a week backup. Mean what changes i have to do in the scirpt.
    Need to change the Start and time variable??? if yes then What to write instead of ‘date’ ?

    thanks in advance for help

Leave a Reply

Your email address will not be published. Required fields are marked *