I came across an insanely educational blog entry by Mike Rubel which described how to create automated snapshot style backups using rsync. Below is my interpretation of his method which I currently use to perform my own backups.
With this simple but powerful script you can perform incremental backups using nothing but rsync and symbolic linking.
We will go into detail within each step to describe exactly what is being done. For this post to make sense to the reader it is best read from the bottom up starting at rsync -avi –delete…
Create a folder that will store a log file and stamp it with the date of the backup
mkdir /media/greg/WD-1.8-Ext4/logs/$(date '+%d-%b-%Y')
If a folder named backup.7 is found then delete it and all of its contents. This is considered the 8th day of backups.
rm -rf /media/greg/WD-1.8-Ext4/backup.7
If the following backup folder is found move it and its contents directly into a new folder. You may add additional number sequencing for more backups or days. Ex: backup.8, backup.9 etc.
mv /media/greg/WD-1.8-Ext4/backup.6 /media/greg/WD-1.8-Ext4/backup.7 mv /media/greg/WD-1.8-Ext4/backup.5 /media/greg/WD-1.8-Ext4/backup.6 mv /media/greg/WD-1.8-Ext4/backup.4 /media/greg/WD-1.8-Ext4/backup.5 mv /media/greg/WD-1.8-Ext4/backup.3 /media/greg/WD-1.8-Ext4/backup.4 mv /media/greg/WD-1.8-Ext4/backup.2 /media/greg/WD-1.8-Ext4/backup.3 mv /media/greg/WD-1.8-Ext4/backup.1 /media/greg/WD-1.8-Ext4/backup.2
This is where the magic happens. Take a copy of the original backup preserving everything but hard link all the data instead of copying it. This is the first step in moving it up the ladder.
cp -al /media/greg/WD-1.8-Ext4/backup.0 /media/greg/WD-1.8-Ext4/backup.1
Backup everything in the folder (while taking a log) named data by mirroring an exact copy of it to second folder named backup.0.
rsync -avi --delete --log-file=/media/greg/WD-1.8-Ext4/logs/$(date '+%d-%b-%Y')/"work_$(date +%m-%d-%Y_%H:%m:%S).log" /media/greg/RAID1-500GB-NTFS/data /media/greg/WD-1.8-Ext4/backup.0/
Think of symbolic links as empty placeholders that point to actual files (inodes). We are only backing up the data once and moving it up the ladder in the form of a symbolic link.
The key to understanding this concept
In order for data to be deleted all symbolic links for that data have to be removed.
Changed data is being preserved up the ladder in the form of symbolic links. Those are your incrementals. By the 8th day we are purging all the data in the folder named backup.7
Save your script with a .sh extension and make it executable:
chmod +x script.sh
This script can then be scheduled using cron job.