RLB (Rsync Local Backup). An easy way to backup your system
RLB
This script is the result of wanting a simple backup solution.
For this I needed an efficient way of running rsync for local backups.
My idea was to keep it simple, hence using only bash & rsync.
But also to log useful information so I could easily verify success.
Usage
Create the directories:
/backup/dr/Backups
/backup/dr/myscripts/
Down load the script (rlbackup1script.sh) and exclusion file (rlbackup1exclude.txt).
put both in /backup/dr/myscripts/
Configure backup1script.sh see "Set-up" below
At a terminal enter the command to get a root session:
$ sudo -i
Then run the script:
# /backup/dr/myscripts/rlbackup1script.sh
First time this will take a while, after that not so so long
Set-up
To change the defaults just open the script(rlbackup1script.sh) in gedit and update any of the 3 configuration values - that's it
Details and Help follow:Code:SOURCE=/ DESTPATH=/backup/dr/Backups EXCLUDEFILE=/backup/dr/myscripts/rlbackup1exclude.txt
1. The source path (SOURCE)
e.g
/
what do you want to backup?
This example would backup the whole system.
2. The destination path (DESTPATH)
e.g
/media/medion1/Backups
Where do you want the backup to be saved to?
This example would backup to my external USB hard drive.
3. The exclude file path (EXCLUDEFILE)
e.g
/media/medion1/Backups/rlbackup1exclude.txt
A file listing directories I don't want backed up (1 line per item).
/home/*/.gvfs /home/*/.mozilla/firefox/*/Cache home/*/.thumbnails
/home2 /home3 /backup /backup2 /windows /lib/modules/*/volatile/*
/lost+found /media /mnt /proc /sys /tmp/ /var/run /var/lock
__________________________________________________
This script produces a log to help you confirm what has been backed-up etc..
Here is an example RLB Log
Deleting Backups:Code:/backup/dr/Backups /backup/dr/Backups rsync -ahvv --log-file=/backup/dr/Backups/back-2008-09-06_18-18-50-LOGFILE.txt --link-dest=/backup/dr/Backups/current-0 --exclude-from=/backup/dr/myscripts/rlbackup1exclude.txt --stats / /backup/dr/Backups/back-2008-09-06_18-18-50 LINK: -File: `/backup/dr/Backups/current-0' -> `back-2008-09-06_18-12-34' EXTRACT0: -/backup/dr/Backups/back-2008-09-06_18-12-34 LINK: -File: `/backup/dr/Backups/current-1' -> `back-2008-09-06_15-19-38' EXTRACT1: -/backup/dr/Backups/back-2008-09-06_15-19-38 LINK: -File: `/backup/dr/Backups/current-2' -> `/backup/dr/Backups/back-2008-09-05_23-43-57' EXTRACT2: -/backup/dr/Backups/back-2008-09-05_23-43-57 ============================================================ Free Space on /backup/dr/Backups Filesystem Size Used Avail Use% Mounted on /dev/sdb1 459G 176G 260G 41% /backup ============================================================ Backup started: Sat, 06 Sep 2008 18:18:50 +0100 Backup finished: Sat, 06 Sep 2008 18:21:32 +0100 ============================================================ Free Space on /backup/dr/Backups Filesystem Size Used Avail Use% Mounted on /dev/sdb1 459G 176G 260G 41% /backup ============================================================ Statistics Number of files: 541664 Number of files transferred: 9 Total file size: 36.32G bytes Total transferred file size: 5.81M bytes Literal data: 5.81M bytes Matched data: 0 bytes File list size: 13255164 File list generation time: 113.077 seconds File list transfer time: 0.000 seconds Total bytes sent: 22.65M Total bytes received: 3.58M sent 22.65M bytes received 3.58M bytes 161.42K bytes/sec total size is 36.32G speedup is 1384.68 Number of files transferred:9 2008/09/06 18:20:46 [31266] >f..t...... home/sww/.glipper/history 2008/09/06 18:20:46 [31266] >f..t...... home/sww/.wapi/shared_data-silver-Linux-x86_64-328-11-0 2008/09/06 18:20:46 [31266] >f..t...... home/sww/.wapi/shared_fileshare-silver-Linux-x86_64-40-11-0 2008/09/06 18:21:12 [31266] >f.st...... root/.recently-used.xbel 2008/09/06 18:21:12 [31266] >f..t...... root/.gconf/apps/gedit-2/preferences/ui/statusbar/%gconf.xml 2008/09/06 18:21:12 [31266] >f..t...... root/.gconfd/saved_state 2008/09/06 18:21:12 [31266] >f.st...... root/.gnome2/gedit-metadata.xml 2008/09/06 18:21:32 [31266] >f.st...... var/log/auth.log 2008/09/06 18:21:32 [31266] >f.st...... var/log/syslog Number of Excludes:38 [sender] hiding directory lost+found because of pattern /lost+found [sender] hiding directory home2 because of pattern /home2 [sender] hiding directory windows because of pattern /windows [sender] hiding directory proc because of pattern /proc [sender] hiding directory home3 because of pattern /home3 [sender] hiding directory backup because of pattern /backup [sender] hiding directory tmp because of pattern /tmp/ [sender] hiding directory mnt because of pattern /mnt [sender] hiding directory sys because of pattern /sys [sender] hiding directory media because of pattern /media [sender] hiding directory backup2 because of pattern /backup2 [sender] hiding directory home/sww/.thumbnails because of pattern home/*/.thumbnails [sender] hiding file home/sww/.gvfs because of pattern /home/*/.gvfs [sender] hiding directory home/sww/.mozilla/firefox/khgde814.Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache [sender] hiding directory home/sww/.mozilla/firefox/khgde814 (4).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache [sender] hiding directory home/sww/.mozilla/firefox/khgde814 (font).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache [sender] hiding directory home/sww/.mozilla/firefox/khgde814 (3).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache [sender] hiding directory home/sww/.mozilla/firefox/khgde814 (config speedups done).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache [sender] hiding directory home/sww/.mozilla/firefox/khgde814 (2noLiveMarkBug).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache [sender] hiding directory home/sww/.mozilla/firefox/khgde814 (1disk.capacity).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache [sender] hiding directory home/sww/.mozilla/firefox/0rg5qtzf.clean/Cache because of pattern /home/*/.mozilla/firefox/*/Cache [sender] hiding directory home/sww/.mozilla/firefox/khgde814 (4 php).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache [sender] hiding directory home/sww/.mozilla/firefox/khgde814 (tabs right).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache [sender] hiding file lib/modules/2.6.24-19-generic/volatile/wl.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/nvidia_new.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/nvidia_legacy.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/nvidia.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/fglrx.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcpci.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslusb2.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslusb.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslslusb.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslsl.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdsl2.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/ath_hal.ko because of pattern /lib/modules/*/volatile/* [sender] hiding file lib/modules/2.6.24-19-generic/volatile/.mounted because of pattern /lib/modules/*/volatile/* [sender] hiding directory var/lock because of pattern /var/lock [sender] hiding directory var/run because of pattern /var/run ============================================================ Sat, 06 Sep 2008 18:21:32 +0100 Actual usage for the last 4 backups: 36G /backup/dr/Backups/back-2008-09-05_23-43-57 556M /backup/dr/Backups/back-2008-09-05_23-43-57 496M /backup/dr/Backups/back-2008-09-06_15-19-38 228M /backup/dr/Backups/back-2008-09-06_18-12-34 196M /backup/dr/Backups/back-2008-09-06_18-18-50 37G total Sat, 06 Sep 2008 18:23:49 +0100 Done. ============================================================ Here are some useful bits of the log highlighted: The actual rsync command the script ran.StatisticsCode:rsync -ahvv --log-file=/backup/dr/Backups/back-2008-09-06_18-18-50-LOGFILE.txt --link-dest=/backup/dr/Backups/current-0 --exclude-from=/backup/dr/myscripts/rlbackup1exclude.txt --stats / /backup/dr/Backups/back-2008-09-06_18-18-50Disk UsageCode:Number of files: 541664 Number of files transferred: 9 Total file size: 36.32G bytes Total transferred file size: 5.81M bytes Literal data: 5.81M bytes Matched data: 0 bytes File list size: 13255164 File list generation time: 113.077 seconds File list transfer time: 0.000 seconds Total bytes sent: 22.65M Total bytes received: 3.58M sent 22.65M bytes received 3.58M bytes 161.42K bytes/sec total size is 36.32G speedup is 1384.68__________________________________________________Code:Sat, 06 Sep 2008 18:21:32 +0100 Actual usage for the last 4 backups: 36G /backup/dr/Backups/back-2008-09-05_23-43-57 556M /backup/dr/Backups/back-2008-09-05_23-43-57 496M /backup/dr/Backups/back-2008-09-06_15-19-38 228M /backup/dr/Backups/back-2008-09-06_18-12-34 196M /backup/dr/Backups/back-2008-09-06_18-18-50 37G total Sat, 06 Sep 2008 18:23:49 +0100 ============================================================
You can delete any backup without affecting others.
If you delete the newest backup you will need to:
Create a symbolic link to it called "current-0"
If this isn't done rsync does a full backup - no hard-linking.
Troubleshooting:
Check the start of the log this gives the rsync command you ran.
Investigate and see if the options and arguments are correct.
Check how many files were transferred & which directories hidden.
Check how much Free Space is detailed in the log.
Run from a terminal to see any error messages (I run from a root terminal).
Background
While researching rsync and hard-linking I came across this article:
"Time Machine for every Unix out there"
http://blog.interlinked.org/tutorial...e_machine.html
Like so many brilliant ideas it was deceptively simple, just 4 lines of code!
The key insight was the way a link was created to point to the new backup.
This link saved the previous directory name between runs of the script.
This is essential to make backups efficiently by hard-linking.
Also interesting is this article by mike rubel:
"Easy Automated Snapshot-Style Backups with Linux and Rsync"
http://www.mikerubel.org/computers/rsync_snapshots/
Also see rsnapshot http://www.rsnapshot.org/
A business strength snapshot solution (depends on perl, logrotate & rsync).
The bash script
An Example Exclude FileCode:#!/bin/bash # rlbackup # (Rsync Local Backup or RLB) # Based on 4 lines of code from "Time Machine for every Unix out there" # <http://blog.interlinked.org/tutorials/rsync_time_machine.html> # # Just fill in the next 3 configuration values - that's it :) SOURCE=/ DESTPATH=/backup/dr/Backups EXCLUDEFILE=/backup/dr/myscripts/rlbackup1exclude.txt # === Usage === # # 0. Do not use spaces in any of the configuration values! # e.g # This is OK: DESTPATH=/media/medion1/disasterrecovery/Backups # # This is NOT: DESTPATH=/media/medion1/disaster recovery/Back ups # ^ ^ # This is NOT: DESTPATH="/media/medion1/disasterrecovery/Back ups" # ^ # 1. The source path (SOURCE) # e.g # / # what do you want to backup? # This example would backup the whole system. # # 2. The destination path (DESTPATH) # e.g # /media/medion1/Backups # Where do you want the backup to be saved to? # This example would backup to my external USB hard drive. # # 3. The exclude file path (EXCLUDEFILE) # e.g # /media/medion1/Backups/rlbackup1exclude.txt # A file listing directories I don't want backed up (1 line per item). # /home/*/.gvfs /home/*/.mozilla/firefox/*/Cache home/*/.thumbnails # /home2 /home3 /backup /backup2 /windows /lib/modules/*/volatile/* # /lost+found /media /mnt /proc /sys /tmp/ /var/run /var/lock # === Preferences === # You can leave these set to the defaults. # # If set to "0" this will list Apparent size of backup and actual usage. # If set to "4" this will list Actual usage for the last 4 backups. # If set to "9" this will list Actual usage for all backups. PREF_DISKUSAGE=4 # If set to "1" this will list the size of each directory. # Does not add much time but adds a long list to the RLBLOG. PREF_DIRLIST=0 # If set to "1" then open log in editor # Open Editor once backup complete, but continue processing the log. # NOTE: If you are using this script with crontab you may wish to set both to "0". PREF_POPUP1=0 PREF_POPUP2=1 # Set your text editor PREF_EDIT="gedit" # === Notes === # This script is the result of wanting a simple backup solution. # For this I needed an efficient way of running rsync for local backups. # # My idea was to keep it simple, hence using only bash & rsync. # But also to log useful information so I could easily verify success. # # Logs: # Most of the useful information is written to RLBLOG which is quite small. # LOGFILE is rsync's own built in log (10x smaller than OUTPUTLOG). # OUTPUTLOG is just that. # rlberror.log is only created if there are validation errors. # # Deleting Backups: # You can delete any backup without affecting others. # If you delete the newest backup you will need to: # Create a symbolic link to it called "current-0" # If this isn't done rsync does a full backup - no hard-linking. # # Troubleshooting: # Check the start of the log this gives the rsync command you ran. # Investigate and see if the options and arguments are correct. # Check how many files were transferred & which directories hidden. # Check how much Free Space is detailed in the log. # Run from a terminal to see any error messages (I run from a root terminal). # # While researching rsync and hard-linking I came across this article: # "Time Machine for every Unix out there" # <http://blog.interlinked.org/tutorials/rsync_time_machine.html> # Like so many brilliant ideas it was deceptively simple, just 4 lines of code! # The key insight was the way a link was created to point to the new backup. # This link saved the previous directory name between runs of the script. # This is essential to make backups efficiently by hard-linking. # # Also interesting is this article by mike rubel: # "Easy Automated Snapshot-Style Backups with Linux and Rsync" # <http://www.mikerubel.org/computers/rsync_snapshots/> # # Also see rsnapshot <http://www.rsnapshot.org/> # A business strength snapshot solution (depends on perl, logrotate & rsync). # ===> No need to alter anything below this point <=== # User input validation # Check if string is zero length and that directory or file exits. if [ -z $DESTPATH ] || [ ! -d $DESTPATH ] ; then echo -n "DESTPATH ERROR - " echo "Backup destination does not exist or you don't have permission!" date -R >> rlberror.log echo -n "DESTPATH ERROR - " echo "Backup destination does not exist or you don't have permission!" >> rlberror.log echo "Exiting..." exit 2 fi if [ -z $SOURCE ] || [ ! -d $SOURCE ]; then echo -n "SOURCE ERROR - " echo "Backup source does not exist or you don't have permission!" date -R >> rlberror.log echo -n "SOURCE ERROR - " echo "Backup source does not exist or you don't have permission!" >> rlberror.log echo "Exiting..." exit 2 fi if [ -z $EXCLUDEFILE ] || [ ! -r $EXCLUDEFILE ]; then echo -n "EXCLUDEFILE ERROR - " echo "Backup exclude file does not exist or you don't have permission!" date -R >> rlberror.log echo -n "EXCLUDEFILE ERROR - " echo "Backup exclude file does not exist or you don't have permission!" >> rlberror.log echo "Exiting..." exit 2 fi # Build Backup Strings DATE=`date "+%Y-%m-%d_%H-%M-%S"` BACKUPNAME=$DESTPATH/back-$DATE OUTPUTLOG=$BACKUPNAME-OUTPUT.txt RLBLOG=$BACKUPNAME-RLBLOG.txt LOGFILE=$BACKUPNAME-LOGFILE.txt LINE="============================================================" # Build Options String. OPT_FLAGS=-ahvv OPT_LOGFILE=--log-file=$LOGFILE #OPT_DEL=--delete OPT_LINKDEST=--link-dest=$DESTPATH/current-0 OPT_EXCLUDE=--exclude-from=$EXCLUDEFILE OPT_STATS=--stats OPT="$OPT_FLAGS $OPT_LOGFILE $OPT_DEL $OPT_LINKDEST $OPT_EXCLUDE $OPT_STATS" # Functions. print_dat() { echo >> $RLBLOG date -R >> $RLBLOG echo >> $RLBLOG } print_s() { info="$1" echo $info >> $RLBLOG } print_sn() { info="$1" echo -n $info >> $RLBLOG } getlinkinfo() { linfo="$1" LINKINFO="Nothing" TEMP_EXTRACT="Nothing" # could use readlink LINKINFO=$(stat $DESTPATH/$linfo | grep "$linfo") print_sn "LINK: -" print_s "$LINKINFO" TEMP_EXTRACT=`expr match "$LINKINFO" '.*\(back-....-..-.._..-..-..\)'` } # cd to main directory (mainly redundant). print_s $DESTPATH cd $DESTPATH pwd >> $RLBLOG # Save to RLBLOG the arguments and options you are running. print_s print_s "rsync $OPT $SOURCE $BACKUPNAME" print_s # Find last backup directory fron link. getlinkinfo "current-0" EXTRACT0=$DESTPATH/$TEMP_EXTRACT print_sn "EXTRACT0: -" print_s $EXTRACT0 getlinkinfo "current-1" EXTRACT1=$DESTPATH/$TEMP_EXTRACT print_sn "EXTRACT1: -" print_s $EXTRACT1 getlinkinfo "current-2" EXTRACT2=$DESTPATH/$TEMP_EXTRACT print_sn "EXTRACT2: -" print_s $EXTRACT2 print_s $LINE # Save to RLBLOG the disk space before the backup. print_s print_s "Free Space on $DESTPATH" df -h $DESTPATH >> $RLBLOG print_s print_s $LINE print_s print_s "Backup started:" print_dat # The main rsync command rsync $OPT $SOURCE $BACKUPNAME > $OUTPUTLOG rm $DESTPATH/current-2 mv $DESTPATH/current-1 $DESTPATH/current-2 mv $DESTPATH/current-0 $DESTPATH/current-1 ln -s back-$DATE $DESTPATH/current-0 # Save to RLBLOG the time and disk space after the backup. print_s "Backup finished:" print_dat print_s $LINE print_s print_s "Free Space on $DESTPATH" df -h $DESTPATH >> $RLBLOG print_s print_s $LINE # Save to RLBLOG the tail of OUTPUTLOG & LOGFILE. print_s print_s "Statistics" tail -n15 $OUTPUTLOG >> $RLBLOG print_s #print_s #tail -v -n50 $LOGFILE >> $RLBLOG #print_s # Save to RLBLOG a grep of LOGFILE showing the files transferred. print_s print_sn "Number of files transferred:" grep -c '] >f' $LOGFILE >> $RLBLOG print_s grep '] >f' $LOGFILE >> $RLBLOG print_s # Save to RLBLOG a grep of OUTPUTLOG showing hidden directories. print_s print_sn "Number of Excludes:" grep -c '] hiding' $OUTPUTLOG >> $RLBLOG print_s grep '] hiding' $OUTPUTLOG >> $RLBLOG print_s print_s $LINE print_dat # View backup results but continue processing the rest of the log. if [ $PREF_POPUP1 -eq 1 ]; then print_s print_s "<======> backup has completed so view the results <======>" print_s "<======> ......continuing to process the log..... <======>" print_s $PREF_EDIT $RLBLOG & fi # Save to RLBLOG the Total size of 4 backups. if [ $PREF_DISKUSAGE -eq 4 ]; then print_s "Actual usage for the last 4 backups:" du -shc $EXTRACT2 $EXTRACT2 $EXTRACT1 $EXTRACT0 $BACKUPNAME >> $RLBLOG print_dat # Save to RLBLOG the Total size of all backups. elif [ $PREF_DISKUSAGE -eq 9 ]; then print_s "Actual usage for all backups:" du -shc back-* >> $RLBLOG print_dat else # Save to RLBLOG the Total size of this backup. print_s print_s "Apparent size of backup and actual usage:" du -shc $BACKUPNAME $BACKUPNAME >> $RLBLOG print_dat fi print_s $LINE # Save to RLBLOG the usage for each directory. if [ $PREF_DIRLIST -eq 1 ]; then print_dat print_s "Directory List" du -h $BACKUPNAME >> $RLBLOG print_dat fi # gzip the logs and move them into the backup folder (to keep things tidy). gzip -c $OUTPUTLOG > $OUTPUTLOG.gz gzip -c $LOGFILE > $LOGFILE.gz mv $OUTPUTLOG.gz $BACKUPNAME mv $LOGFILE.gz $BACKUPNAME # delete the original, uncompressed logs rm $OUTPUTLOG rm $LOGFILE print_s print_s "Done." mv $RLBLOG $BACKUPNAME # View backup results. if [ $PREF_POPUP2 -eq 1 ]; then $PREF_EDIT $BACKUPNAME/back-$DATE-RLBLOG.txt & fi
Tested on Ubuntu 8.04/8.10/9.04/9.10/10.04 64 bit.Code:/home/*/.gvfs /home/*/.mozilla/firefox/*/Cache home/*/.thumbnails /home2 /home3 /backup /backup2 /windows /lib/modules/*/volatile/* /lost+found /media /mnt /proc /sys /tmp/ /var/run /var/lock
Reply With Quote
Bookmarks