SilverWave
September 6th, 2008, 06:50 PM
RLB
This script is the result of wanting a simple backup solution.
For this I needed an efficient way of running rsync for local backups.
My idea was to keep it simple, hence using only bash & rsync.
But also to log useful information so I could easily verify success.
Usage
Create the directories:
/backup/dr/Backups
/backup/dr/myscripts/
Down load the script (rlbackup1script.sh) and exclusion file (rlbackup1exclude.txt).
put both in /backup/dr/myscripts/
Configure backup1script.sh see "Set-up" below
At a terminal enter the command to get a root session:
$ sudo -i
Then run the script:
# /backup/dr/myscripts/rlbackup1script.sh
First time this will take a while, after that not so so long :)
Set-up
To change the defaults just open the script(rlbackup1script.sh) in gedit and update any of the 3 configuration values - that's it :)
SOURCE=/
DESTPATH=/backup/dr/Backups
EXCLUDEFILE=/backup/dr/myscripts/rlbackup1exclude.txtDetails and Help follow:
1. The source path (SOURCE)
e.g
/
what do you want to backup?
This example would backup the whole system.
2. The destination path (DESTPATH)
e.g
/media/medion1/Backups
Where do you want the backup to be saved to?
This example would backup to my external USB hard drive.
3. The exclude file path (EXCLUDEFILE)
e.g
/media/medion1/Backups/rlbackup1exclude.txt
A file listing directories I don't want backed up (1 line per item).
/home/*/.gvfs /home/*/.mozilla/firefox/*/Cache home/*/.thumbnails
/home2 /home3 /backup /backup2 /windows /lib/modules/*/volatile/*
/lost+found /media /mnt /proc /sys /tmp/ /var/run /var/lock
__________________________________________________
This script produces a log to help you confirm what has been backed-up etc..
Here is an example RLB Log
/backup/dr/Backups
/backup/dr/Backups
rsync -ahvv --log-file=/backup/dr/Backups/back-2008-09-06_18-18-50-LOGFILE.txt --link-dest=/backup/dr/Backups/current-0 --exclude-from=/backup/dr/myscripts/rlbackup1exclude.txt --stats / /backup/dr/Backups/back-2008-09-06_18-18-50
LINK: -File: `/backup/dr/Backups/current-0' -> `back-2008-09-06_18-12-34'
EXTRACT0: -/backup/dr/Backups/back-2008-09-06_18-12-34
LINK: -File: `/backup/dr/Backups/current-1' -> `back-2008-09-06_15-19-38'
EXTRACT1: -/backup/dr/Backups/back-2008-09-06_15-19-38
LINK: -File: `/backup/dr/Backups/current-2' -> `/backup/dr/Backups/back-2008-09-05_23-43-57'
EXTRACT2: -/backup/dr/Backups/back-2008-09-05_23-43-57
================================================== ==========
Free Space on /backup/dr/Backups
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 459G 176G 260G 41% /backup
================================================== ==========
Backup started:
Sat, 06 Sep 2008 18:18:50 +0100
Backup finished:
Sat, 06 Sep 2008 18:21:32 +0100
================================================== ==========
Free Space on /backup/dr/Backups
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 459G 176G 260G 41% /backup
================================================== ==========
Statistics
Number of files: 541664
Number of files transferred: 9
Total file size: 36.32G bytes
Total transferred file size: 5.81M bytes
Literal data: 5.81M bytes
Matched data: 0 bytes
File list size: 13255164
File list generation time: 113.077 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 22.65M
Total bytes received: 3.58M
sent 22.65M bytes received 3.58M bytes 161.42K bytes/sec
total size is 36.32G speedup is 1384.68
Number of files transferred:9
2008/09/06 18:20:46 [31266] >f..t...... home/sww/.glipper/history
2008/09/06 18:20:46 [31266] >f..t...... home/sww/.wapi/shared_data-silver-Linux-x86_64-328-11-0
2008/09/06 18:20:46 [31266] >f..t...... home/sww/.wapi/shared_fileshare-silver-Linux-x86_64-40-11-0
2008/09/06 18:21:12 [31266] >f.st...... root/.recently-used.xbel
2008/09/06 18:21:12 [31266] >f..t...... root/.gconf/apps/gedit-2/preferences/ui/statusbar/%gconf.xml
2008/09/06 18:21:12 [31266] >f..t...... root/.gconfd/saved_state
2008/09/06 18:21:12 [31266] >f.st...... root/.gnome2/gedit-metadata.xml
2008/09/06 18:21:32 [31266] >f.st...... var/log/auth.log
2008/09/06 18:21:32 [31266] >f.st...... var/log/syslog
Number of Excludes:38
[sender] hiding directory lost+found because of pattern /lost+found
[sender] hiding directory home2 because of pattern /home2
[sender] hiding directory windows because of pattern /windows
[sender] hiding directory proc because of pattern /proc
[sender] hiding directory home3 because of pattern /home3
[sender] hiding directory backup because of pattern /backup
[sender] hiding directory tmp because of pattern /tmp/
[sender] hiding directory mnt because of pattern /mnt
[sender] hiding directory sys because of pattern /sys
[sender] hiding directory media because of pattern /media
[sender] hiding directory backup2 because of pattern /backup2
[sender] hiding directory home/sww/.thumbnails because of pattern home/*/.thumbnails
[sender] hiding file home/sww/.gvfs because of pattern /home/*/.gvfs
[sender] hiding directory home/sww/.mozilla/firefox/khgde814.Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (4).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (font).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (3).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (config speedups done).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (2noLiveMarkBug).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (1disk.capacity).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/0rg5qtzf.clean/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (4 php).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (tabs right).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/wl.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/nvidia_new.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/nvidia_legacy.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/nvidia.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fglrx.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcpci.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslusb2.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslusb.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslslusb.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslsl.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdsl2.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/ath_hal.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/.mounted because of pattern /lib/modules/*/volatile/*
[sender] hiding directory var/lock because of pattern /var/lock
[sender] hiding directory var/run because of pattern /var/run
================================================== ==========
Sat, 06 Sep 2008 18:21:32 +0100
Actual usage for the last 4 backups:
36G /backup/dr/Backups/back-2008-09-05_23-43-57
556M /backup/dr/Backups/back-2008-09-05_23-43-57
496M /backup/dr/Backups/back-2008-09-06_15-19-38
228M /backup/dr/Backups/back-2008-09-06_18-12-34
196M /backup/dr/Backups/back-2008-09-06_18-18-50
37G total
Sat, 06 Sep 2008 18:23:49 +0100
Done.
================================================== ==========
Here are some useful bits of the log highlighted:
The actual rsync command the script ran.
rsync -ahvv --log-file=/backup/dr/Backups/back-2008-09-06_18-18-50-LOGFILE.txt --link-dest=/backup/dr/Backups/current-0 --exclude-from=/backup/dr/myscripts/rlbackup1exclude.txt --stats / /backup/dr/Backups/back-2008-09-06_18-18-50
Statistics
Number of files: 541664
Number of files transferred: 9
Total file size: 36.32G bytes
Total transferred file size: 5.81M bytes
Literal data: 5.81M bytes
Matched data: 0 bytes
File list size: 13255164
File list generation time: 113.077 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 22.65M
Total bytes received: 3.58M
sent 22.65M bytes received 3.58M bytes 161.42K bytes/sec
total size is 36.32G speedup is 1384.68
Disk Usage
Sat, 06 Sep 2008 18:21:32 +0100
Actual usage for the last 4 backups:
36G /backup/dr/Backups/back-2008-09-05_23-43-57
556M /backup/dr/Backups/back-2008-09-05_23-43-57
496M /backup/dr/Backups/back-2008-09-06_15-19-38
228M /backup/dr/Backups/back-2008-09-06_18-12-34
196M /backup/dr/Backups/back-2008-09-06_18-18-50
37G total
Sat, 06 Sep 2008 18:23:49 +0100
================================================== ==========
__________________________________________________
Deleting Backups:
You can delete any backup without affecting others.
If you delete the newest backup you will need to:
Create a symbolic link to it called "current-0"
If this isn't done rsync does a full backup - no hard-linking.
Troubleshooting:
Check the start of the log this gives the rsync command you ran.
Investigate and see if the options and arguments are correct.
Check how many files were transferred & which directories hidden.
Check how much Free Space is detailed in the log.
Run from a terminal to see any error messages (I run from a root terminal).
Background
While researching rsync and hard-linking I came across this article:
"Time Machine for every Unix out there"
http://blog.interlinked.org/tutorials/rsync_time_machine.html
Like so many brilliant ideas it was deceptively simple, just 4 lines of code!
The key insight was the way a link was created to point to the new backup.
This link saved the previous directory name between runs of the script.
This is essential to make backups efficiently by hard-linking.
Also interesting is this article by mike rubel:
"Easy Automated Snapshot-Style Backups with Linux and Rsync"
http://www.mikerubel.org/computers/rsync_snapshots/
Also see rsnapshot http://www.rsnapshot.org/
A business strength snapshot solution (depends on perl, logrotate & rsync).
The bash script
#!/bin/bash
# rlbackup
# (Rsync Local Backup or RLB)
# Based on 4 lines of code from "Time Machine for every Unix out there"
# <http://blog.interlinked.org/tutorials/rsync_time_machine.html>
#
# Just fill in the next 3 configuration values - that's it :)
SOURCE=/
DESTPATH=/backup/dr/Backups
EXCLUDEFILE=/backup/dr/myscripts/rlbackup1exclude.txt
# === Usage ===
#
# 0. Do not use spaces in any of the configuration values!
# e.g
# This is OK: DESTPATH=/media/medion1/disasterrecovery/Backups
#
# This is NOT: DESTPATH=/media/medion1/disaster recovery/Back ups
# ^ ^
# This is NOT: DESTPATH="/media/medion1/disasterrecovery/Back ups"
# ^
# 1. The source path (SOURCE)
# e.g
# /
# what do you want to backup?
# This example would backup the whole system.
#
# 2. The destination path (DESTPATH)
# e.g
# /media/medion1/Backups
# Where do you want the backup to be saved to?
# This example would backup to my external USB hard drive.
#
# 3. The exclude file path (EXCLUDEFILE)
# e.g
# /media/medion1/Backups/rlbackup1exclude.txt
# A file listing directories I don't want backed up (1 line per item).
# /home/*/.gvfs /home/*/.mozilla/firefox/*/Cache home/*/.thumbnails
# /home2 /home3 /backup /backup2 /windows /lib/modules/*/volatile/*
# /lost+found /media /mnt /proc /sys /tmp/ /var/run /var/lock
# === Preferences ===
# You can leave these set to the defaults.
#
# If set to "0" this will list Apparent size of backup and actual usage.
# If set to "4" this will list Actual usage for the last 4 backups.
# If set to "9" this will list Actual usage for all backups.
PREF_DISKUSAGE=4
# If set to "1" this will list the size of each directory.
# Does not add much time but adds a long list to the RLBLOG.
PREF_DIRLIST=0
# If set to "1" then open log in editor
# Open Editor once backup complete, but continue processing the log.
# NOTE: If you are using this script with crontab you may wish to set both to "0".
PREF_POPUP1=0
PREF_POPUP2=1
# Set your text editor
PREF_EDIT="gedit"
# === Notes ===
# This script is the result of wanting a simple backup solution.
# For this I needed an efficient way of running rsync for local backups.
#
# My idea was to keep it simple, hence using only bash & rsync.
# But also to log useful information so I could easily verify success.
#
# Logs:
# Most of the useful information is written to RLBLOG which is quite small.
# LOGFILE is rsync's own built in log (10x smaller than OUTPUTLOG).
# OUTPUTLOG is just that.
# rlberror.log is only created if there are validation errors.
#
# Deleting Backups:
# You can delete any backup without affecting others.
# If you delete the newest backup you will need to:
# Create a symbolic link to it called "current-0"
# If this isn't done rsync does a full backup - no hard-linking.
#
# Troubleshooting:
# Check the start of the log this gives the rsync command you ran.
# Investigate and see if the options and arguments are correct.
# Check how many files were transferred & which directories hidden.
# Check how much Free Space is detailed in the log.
# Run from a terminal to see any error messages (I run from a root terminal).
#
# While researching rsync and hard-linking I came across this article:
# "Time Machine for every Unix out there"
# <http://blog.interlinked.org/tutorials/rsync_time_machine.html>
# Like so many brilliant ideas it was deceptively simple, just 4 lines of code!
# The key insight was the way a link was created to point to the new backup.
# This link saved the previous directory name between runs of the script.
# This is essential to make backups efficiently by hard-linking.
#
# Also interesting is this article by mike rubel:
# "Easy Automated Snapshot-Style Backups with Linux and Rsync"
# <http://www.mikerubel.org/computers/rsync_snapshots/>
#
# Also see rsnapshot <http://www.rsnapshot.org/>
# A business strength snapshot solution (depends on perl, logrotate & rsync).
# ===> No need to alter anything below this point <===
# User input validation
# Check if string is zero length and that directory or file exits.
if [ -z $DESTPATH ] || [ ! -d $DESTPATH ] ; then
echo -n "DESTPATH ERROR - "
echo "Backup destination does not exist or you don't have permission!"
date -R >> rlberror.log
echo -n "DESTPATH ERROR - "
echo "Backup destination does not exist or you don't have permission!" >> rlberror.log
echo "Exiting..."
exit 2
fi
if [ -z $SOURCE ] || [ ! -d $SOURCE ]; then
echo -n "SOURCE ERROR - "
echo "Backup source does not exist or you don't have permission!"
date -R >> rlberror.log
echo -n "SOURCE ERROR - "
echo "Backup source does not exist or you don't have permission!" >> rlberror.log
echo "Exiting..."
exit 2
fi
if [ -z $EXCLUDEFILE ] || [ ! -r $EXCLUDEFILE ]; then
echo -n "EXCLUDEFILE ERROR - "
echo "Backup exclude file does not exist or you don't have permission!"
date -R >> rlberror.log
echo -n "EXCLUDEFILE ERROR - "
echo "Backup exclude file does not exist or you don't have permission!" >> rlberror.log
echo "Exiting..."
exit 2
fi
# Build Backup Strings
DATE=`date "+%Y-%m-%d_%H-%M-%S"`
BACKUPNAME=$DESTPATH/back-$DATE
OUTPUTLOG=$BACKUPNAME-OUTPUT.txt
RLBLOG=$BACKUPNAME-RLBLOG.txt
LOGFILE=$BACKUPNAME-LOGFILE.txt
LINE="================================================== =========="
# Build Options String.
OPT_FLAGS=-ahvv
OPT_LOGFILE=--log-file=$LOGFILE
#OPT_DEL=--delete
OPT_LINKDEST=--link-dest=$DESTPATH/current-0
OPT_EXCLUDE=--exclude-from=$EXCLUDEFILE
OPT_STATS=--stats
OPT="$OPT_FLAGS $OPT_LOGFILE $OPT_DEL $OPT_LINKDEST $OPT_EXCLUDE $OPT_STATS"
# Functions.
print_dat()
{
echo >> $RLBLOG
date -R >> $RLBLOG
echo >> $RLBLOG
}
print_s()
{
info="$1"
echo $info >> $RLBLOG
}
print_sn()
{
info="$1"
echo -n $info >> $RLBLOG
}
getlinkinfo()
{
linfo="$1"
LINKINFO="Nothing"
TEMP_EXTRACT="Nothing"
# could use readlink
LINKINFO=$(stat $DESTPATH/$linfo | grep "$linfo")
print_sn "LINK: -"
print_s "$LINKINFO"
TEMP_EXTRACT=`expr match "$LINKINFO" '.*\(back-....-..-.._..-..-..\)'`
}
# cd to main directory (mainly redundant).
print_s $DESTPATH
cd $DESTPATH
pwd >> $RLBLOG
# Save to RLBLOG the arguments and options you are running.
print_s
print_s "rsync $OPT $SOURCE $BACKUPNAME"
print_s
# Find last backup directory fron link.
getlinkinfo "current-0"
EXTRACT0=$DESTPATH/$TEMP_EXTRACT
print_sn "EXTRACT0: -"
print_s $EXTRACT0
getlinkinfo "current-1"
EXTRACT1=$DESTPATH/$TEMP_EXTRACT
print_sn "EXTRACT1: -"
print_s $EXTRACT1
getlinkinfo "current-2"
EXTRACT2=$DESTPATH/$TEMP_EXTRACT
print_sn "EXTRACT2: -"
print_s $EXTRACT2
print_s $LINE
# Save to RLBLOG the disk space before the backup.
print_s
print_s "Free Space on $DESTPATH"
df -h $DESTPATH >> $RLBLOG
print_s
print_s $LINE
print_s
print_s "Backup started:"
print_dat
# The main rsync command
rsync $OPT $SOURCE $BACKUPNAME > $OUTPUTLOG
rm $DESTPATH/current-2
mv $DESTPATH/current-1 $DESTPATH/current-2
mv $DESTPATH/current-0 $DESTPATH/current-1
ln -s back-$DATE $DESTPATH/current-0
# Save to RLBLOG the time and disk space after the backup.
print_s "Backup finished:"
print_dat
print_s $LINE
print_s
print_s "Free Space on $DESTPATH"
df -h $DESTPATH >> $RLBLOG
print_s
print_s $LINE
# Save to RLBLOG the tail of OUTPUTLOG & LOGFILE.
print_s
print_s "Statistics"
tail -n15 $OUTPUTLOG >> $RLBLOG
print_s
#print_s
#tail -v -n50 $LOGFILE >> $RLBLOG
#print_s
# Save to RLBLOG a grep of LOGFILE showing the files transferred.
print_s
print_sn "Number of files transferred:"
grep -c '] >f' $LOGFILE >> $RLBLOG
print_s
grep '] >f' $LOGFILE >> $RLBLOG
print_s
# Save to RLBLOG a grep of OUTPUTLOG showing hidden directories.
print_s
print_sn "Number of Excludes:"
grep -c '] hiding' $OUTPUTLOG >> $RLBLOG
print_s
grep '] hiding' $OUTPUTLOG >> $RLBLOG
print_s
print_s $LINE
print_dat
# View backup results but continue processing the rest of the log.
if [ $PREF_POPUP1 -eq 1 ]; then
print_s
print_s "<======> backup has completed so view the results <======>"
print_s "<======> ......continuing to process the log..... <======>"
print_s
$PREF_EDIT $RLBLOG &
fi
# Save to RLBLOG the Total size of 4 backups.
if [ $PREF_DISKUSAGE -eq 4 ]; then
print_s "Actual usage for the last 4 backups:"
du -shc $EXTRACT2 $EXTRACT2 $EXTRACT1 $EXTRACT0 $BACKUPNAME >> $RLBLOG
print_dat
# Save to RLBLOG the Total size of all backups.
elif [ $PREF_DISKUSAGE -eq 9 ]; then
print_s "Actual usage for all backups:"
du -shc back-* >> $RLBLOG
print_dat
else
# Save to RLBLOG the Total size of this backup.
print_s
print_s "Apparent size of backup and actual usage:"
du -shc $BACKUPNAME $BACKUPNAME >> $RLBLOG
print_dat
fi
print_s $LINE
# Save to RLBLOG the usage for each directory.
if [ $PREF_DIRLIST -eq 1 ]; then
print_dat
print_s "Directory List"
du -h $BACKUPNAME >> $RLBLOG
print_dat
fi
# gzip the logs and move them into the backup folder (to keep things tidy).
gzip -c $OUTPUTLOG > $OUTPUTLOG.gz
gzip -c $LOGFILE > $LOGFILE.gz
mv $OUTPUTLOG.gz $BACKUPNAME
mv $LOGFILE.gz $BACKUPNAME
# delete the original, uncompressed logs
rm $OUTPUTLOG
rm $LOGFILE
print_s
print_s "Done."
mv $RLBLOG $BACKUPNAME
# View backup results.
if [ $PREF_POPUP2 -eq 1 ]; then
$PREF_EDIT $BACKUPNAME/back-$DATE-RLBLOG.txt &
fi
An Example Exclude File
/home/*/.gvfs
/home/*/.mozilla/firefox/*/Cache
home/*/.thumbnails
/home2
/home3
/backup
/backup2
/windows
/lib/modules/*/volatile/*
/lost+found
/media
/mnt
/proc
/sys
/tmp/
/var/run
/var/lock
Tested on Ubuntu 8.04/8.10/9.04/9.10/10.04 64 bit.
This script is the result of wanting a simple backup solution.
For this I needed an efficient way of running rsync for local backups.
My idea was to keep it simple, hence using only bash & rsync.
But also to log useful information so I could easily verify success.
Usage
Create the directories:
/backup/dr/Backups
/backup/dr/myscripts/
Down load the script (rlbackup1script.sh) and exclusion file (rlbackup1exclude.txt).
put both in /backup/dr/myscripts/
Configure backup1script.sh see "Set-up" below
At a terminal enter the command to get a root session:
$ sudo -i
Then run the script:
# /backup/dr/myscripts/rlbackup1script.sh
First time this will take a while, after that not so so long :)
Set-up
To change the defaults just open the script(rlbackup1script.sh) in gedit and update any of the 3 configuration values - that's it :)
SOURCE=/
DESTPATH=/backup/dr/Backups
EXCLUDEFILE=/backup/dr/myscripts/rlbackup1exclude.txtDetails and Help follow:
1. The source path (SOURCE)
e.g
/
what do you want to backup?
This example would backup the whole system.
2. The destination path (DESTPATH)
e.g
/media/medion1/Backups
Where do you want the backup to be saved to?
This example would backup to my external USB hard drive.
3. The exclude file path (EXCLUDEFILE)
e.g
/media/medion1/Backups/rlbackup1exclude.txt
A file listing directories I don't want backed up (1 line per item).
/home/*/.gvfs /home/*/.mozilla/firefox/*/Cache home/*/.thumbnails
/home2 /home3 /backup /backup2 /windows /lib/modules/*/volatile/*
/lost+found /media /mnt /proc /sys /tmp/ /var/run /var/lock
__________________________________________________
This script produces a log to help you confirm what has been backed-up etc..
Here is an example RLB Log
/backup/dr/Backups
/backup/dr/Backups
rsync -ahvv --log-file=/backup/dr/Backups/back-2008-09-06_18-18-50-LOGFILE.txt --link-dest=/backup/dr/Backups/current-0 --exclude-from=/backup/dr/myscripts/rlbackup1exclude.txt --stats / /backup/dr/Backups/back-2008-09-06_18-18-50
LINK: -File: `/backup/dr/Backups/current-0' -> `back-2008-09-06_18-12-34'
EXTRACT0: -/backup/dr/Backups/back-2008-09-06_18-12-34
LINK: -File: `/backup/dr/Backups/current-1' -> `back-2008-09-06_15-19-38'
EXTRACT1: -/backup/dr/Backups/back-2008-09-06_15-19-38
LINK: -File: `/backup/dr/Backups/current-2' -> `/backup/dr/Backups/back-2008-09-05_23-43-57'
EXTRACT2: -/backup/dr/Backups/back-2008-09-05_23-43-57
================================================== ==========
Free Space on /backup/dr/Backups
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 459G 176G 260G 41% /backup
================================================== ==========
Backup started:
Sat, 06 Sep 2008 18:18:50 +0100
Backup finished:
Sat, 06 Sep 2008 18:21:32 +0100
================================================== ==========
Free Space on /backup/dr/Backups
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 459G 176G 260G 41% /backup
================================================== ==========
Statistics
Number of files: 541664
Number of files transferred: 9
Total file size: 36.32G bytes
Total transferred file size: 5.81M bytes
Literal data: 5.81M bytes
Matched data: 0 bytes
File list size: 13255164
File list generation time: 113.077 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 22.65M
Total bytes received: 3.58M
sent 22.65M bytes received 3.58M bytes 161.42K bytes/sec
total size is 36.32G speedup is 1384.68
Number of files transferred:9
2008/09/06 18:20:46 [31266] >f..t...... home/sww/.glipper/history
2008/09/06 18:20:46 [31266] >f..t...... home/sww/.wapi/shared_data-silver-Linux-x86_64-328-11-0
2008/09/06 18:20:46 [31266] >f..t...... home/sww/.wapi/shared_fileshare-silver-Linux-x86_64-40-11-0
2008/09/06 18:21:12 [31266] >f.st...... root/.recently-used.xbel
2008/09/06 18:21:12 [31266] >f..t...... root/.gconf/apps/gedit-2/preferences/ui/statusbar/%gconf.xml
2008/09/06 18:21:12 [31266] >f..t...... root/.gconfd/saved_state
2008/09/06 18:21:12 [31266] >f.st...... root/.gnome2/gedit-metadata.xml
2008/09/06 18:21:32 [31266] >f.st...... var/log/auth.log
2008/09/06 18:21:32 [31266] >f.st...... var/log/syslog
Number of Excludes:38
[sender] hiding directory lost+found because of pattern /lost+found
[sender] hiding directory home2 because of pattern /home2
[sender] hiding directory windows because of pattern /windows
[sender] hiding directory proc because of pattern /proc
[sender] hiding directory home3 because of pattern /home3
[sender] hiding directory backup because of pattern /backup
[sender] hiding directory tmp because of pattern /tmp/
[sender] hiding directory mnt because of pattern /mnt
[sender] hiding directory sys because of pattern /sys
[sender] hiding directory media because of pattern /media
[sender] hiding directory backup2 because of pattern /backup2
[sender] hiding directory home/sww/.thumbnails because of pattern home/*/.thumbnails
[sender] hiding file home/sww/.gvfs because of pattern /home/*/.gvfs
[sender] hiding directory home/sww/.mozilla/firefox/khgde814.Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (4).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (font).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (3).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (config speedups done).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (2noLiveMarkBug).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (1disk.capacity).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/0rg5qtzf.clean/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (4 php).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding directory home/sww/.mozilla/firefox/khgde814 (tabs right).Default User/Cache because of pattern /home/*/.mozilla/firefox/*/Cache
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/wl.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/nvidia_new.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/nvidia_legacy.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/nvidia.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fglrx.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcpci.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslusb2.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslusb.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslslusb.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdslsl.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/fcdsl2.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/ath_hal.ko because of pattern /lib/modules/*/volatile/*
[sender] hiding file lib/modules/2.6.24-19-generic/volatile/.mounted because of pattern /lib/modules/*/volatile/*
[sender] hiding directory var/lock because of pattern /var/lock
[sender] hiding directory var/run because of pattern /var/run
================================================== ==========
Sat, 06 Sep 2008 18:21:32 +0100
Actual usage for the last 4 backups:
36G /backup/dr/Backups/back-2008-09-05_23-43-57
556M /backup/dr/Backups/back-2008-09-05_23-43-57
496M /backup/dr/Backups/back-2008-09-06_15-19-38
228M /backup/dr/Backups/back-2008-09-06_18-12-34
196M /backup/dr/Backups/back-2008-09-06_18-18-50
37G total
Sat, 06 Sep 2008 18:23:49 +0100
Done.
================================================== ==========
Here are some useful bits of the log highlighted:
The actual rsync command the script ran.
rsync -ahvv --log-file=/backup/dr/Backups/back-2008-09-06_18-18-50-LOGFILE.txt --link-dest=/backup/dr/Backups/current-0 --exclude-from=/backup/dr/myscripts/rlbackup1exclude.txt --stats / /backup/dr/Backups/back-2008-09-06_18-18-50
Statistics
Number of files: 541664
Number of files transferred: 9
Total file size: 36.32G bytes
Total transferred file size: 5.81M bytes
Literal data: 5.81M bytes
Matched data: 0 bytes
File list size: 13255164
File list generation time: 113.077 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 22.65M
Total bytes received: 3.58M
sent 22.65M bytes received 3.58M bytes 161.42K bytes/sec
total size is 36.32G speedup is 1384.68
Disk Usage
Sat, 06 Sep 2008 18:21:32 +0100
Actual usage for the last 4 backups:
36G /backup/dr/Backups/back-2008-09-05_23-43-57
556M /backup/dr/Backups/back-2008-09-05_23-43-57
496M /backup/dr/Backups/back-2008-09-06_15-19-38
228M /backup/dr/Backups/back-2008-09-06_18-12-34
196M /backup/dr/Backups/back-2008-09-06_18-18-50
37G total
Sat, 06 Sep 2008 18:23:49 +0100
================================================== ==========
__________________________________________________
Deleting Backups:
You can delete any backup without affecting others.
If you delete the newest backup you will need to:
Create a symbolic link to it called "current-0"
If this isn't done rsync does a full backup - no hard-linking.
Troubleshooting:
Check the start of the log this gives the rsync command you ran.
Investigate and see if the options and arguments are correct.
Check how many files were transferred & which directories hidden.
Check how much Free Space is detailed in the log.
Run from a terminal to see any error messages (I run from a root terminal).
Background
While researching rsync and hard-linking I came across this article:
"Time Machine for every Unix out there"
http://blog.interlinked.org/tutorials/rsync_time_machine.html
Like so many brilliant ideas it was deceptively simple, just 4 lines of code!
The key insight was the way a link was created to point to the new backup.
This link saved the previous directory name between runs of the script.
This is essential to make backups efficiently by hard-linking.
Also interesting is this article by mike rubel:
"Easy Automated Snapshot-Style Backups with Linux and Rsync"
http://www.mikerubel.org/computers/rsync_snapshots/
Also see rsnapshot http://www.rsnapshot.org/
A business strength snapshot solution (depends on perl, logrotate & rsync).
The bash script
#!/bin/bash
# rlbackup
# (Rsync Local Backup or RLB)
# Based on 4 lines of code from "Time Machine for every Unix out there"
# <http://blog.interlinked.org/tutorials/rsync_time_machine.html>
#
# Just fill in the next 3 configuration values - that's it :)
SOURCE=/
DESTPATH=/backup/dr/Backups
EXCLUDEFILE=/backup/dr/myscripts/rlbackup1exclude.txt
# === Usage ===
#
# 0. Do not use spaces in any of the configuration values!
# e.g
# This is OK: DESTPATH=/media/medion1/disasterrecovery/Backups
#
# This is NOT: DESTPATH=/media/medion1/disaster recovery/Back ups
# ^ ^
# This is NOT: DESTPATH="/media/medion1/disasterrecovery/Back ups"
# ^
# 1. The source path (SOURCE)
# e.g
# /
# what do you want to backup?
# This example would backup the whole system.
#
# 2. The destination path (DESTPATH)
# e.g
# /media/medion1/Backups
# Where do you want the backup to be saved to?
# This example would backup to my external USB hard drive.
#
# 3. The exclude file path (EXCLUDEFILE)
# e.g
# /media/medion1/Backups/rlbackup1exclude.txt
# A file listing directories I don't want backed up (1 line per item).
# /home/*/.gvfs /home/*/.mozilla/firefox/*/Cache home/*/.thumbnails
# /home2 /home3 /backup /backup2 /windows /lib/modules/*/volatile/*
# /lost+found /media /mnt /proc /sys /tmp/ /var/run /var/lock
# === Preferences ===
# You can leave these set to the defaults.
#
# If set to "0" this will list Apparent size of backup and actual usage.
# If set to "4" this will list Actual usage for the last 4 backups.
# If set to "9" this will list Actual usage for all backups.
PREF_DISKUSAGE=4
# If set to "1" this will list the size of each directory.
# Does not add much time but adds a long list to the RLBLOG.
PREF_DIRLIST=0
# If set to "1" then open log in editor
# Open Editor once backup complete, but continue processing the log.
# NOTE: If you are using this script with crontab you may wish to set both to "0".
PREF_POPUP1=0
PREF_POPUP2=1
# Set your text editor
PREF_EDIT="gedit"
# === Notes ===
# This script is the result of wanting a simple backup solution.
# For this I needed an efficient way of running rsync for local backups.
#
# My idea was to keep it simple, hence using only bash & rsync.
# But also to log useful information so I could easily verify success.
#
# Logs:
# Most of the useful information is written to RLBLOG which is quite small.
# LOGFILE is rsync's own built in log (10x smaller than OUTPUTLOG).
# OUTPUTLOG is just that.
# rlberror.log is only created if there are validation errors.
#
# Deleting Backups:
# You can delete any backup without affecting others.
# If you delete the newest backup you will need to:
# Create a symbolic link to it called "current-0"
# If this isn't done rsync does a full backup - no hard-linking.
#
# Troubleshooting:
# Check the start of the log this gives the rsync command you ran.
# Investigate and see if the options and arguments are correct.
# Check how many files were transferred & which directories hidden.
# Check how much Free Space is detailed in the log.
# Run from a terminal to see any error messages (I run from a root terminal).
#
# While researching rsync and hard-linking I came across this article:
# "Time Machine for every Unix out there"
# <http://blog.interlinked.org/tutorials/rsync_time_machine.html>
# Like so many brilliant ideas it was deceptively simple, just 4 lines of code!
# The key insight was the way a link was created to point to the new backup.
# This link saved the previous directory name between runs of the script.
# This is essential to make backups efficiently by hard-linking.
#
# Also interesting is this article by mike rubel:
# "Easy Automated Snapshot-Style Backups with Linux and Rsync"
# <http://www.mikerubel.org/computers/rsync_snapshots/>
#
# Also see rsnapshot <http://www.rsnapshot.org/>
# A business strength snapshot solution (depends on perl, logrotate & rsync).
# ===> No need to alter anything below this point <===
# User input validation
# Check if string is zero length and that directory or file exits.
if [ -z $DESTPATH ] || [ ! -d $DESTPATH ] ; then
echo -n "DESTPATH ERROR - "
echo "Backup destination does not exist or you don't have permission!"
date -R >> rlberror.log
echo -n "DESTPATH ERROR - "
echo "Backup destination does not exist or you don't have permission!" >> rlberror.log
echo "Exiting..."
exit 2
fi
if [ -z $SOURCE ] || [ ! -d $SOURCE ]; then
echo -n "SOURCE ERROR - "
echo "Backup source does not exist or you don't have permission!"
date -R >> rlberror.log
echo -n "SOURCE ERROR - "
echo "Backup source does not exist or you don't have permission!" >> rlberror.log
echo "Exiting..."
exit 2
fi
if [ -z $EXCLUDEFILE ] || [ ! -r $EXCLUDEFILE ]; then
echo -n "EXCLUDEFILE ERROR - "
echo "Backup exclude file does not exist or you don't have permission!"
date -R >> rlberror.log
echo -n "EXCLUDEFILE ERROR - "
echo "Backup exclude file does not exist or you don't have permission!" >> rlberror.log
echo "Exiting..."
exit 2
fi
# Build Backup Strings
DATE=`date "+%Y-%m-%d_%H-%M-%S"`
BACKUPNAME=$DESTPATH/back-$DATE
OUTPUTLOG=$BACKUPNAME-OUTPUT.txt
RLBLOG=$BACKUPNAME-RLBLOG.txt
LOGFILE=$BACKUPNAME-LOGFILE.txt
LINE="================================================== =========="
# Build Options String.
OPT_FLAGS=-ahvv
OPT_LOGFILE=--log-file=$LOGFILE
#OPT_DEL=--delete
OPT_LINKDEST=--link-dest=$DESTPATH/current-0
OPT_EXCLUDE=--exclude-from=$EXCLUDEFILE
OPT_STATS=--stats
OPT="$OPT_FLAGS $OPT_LOGFILE $OPT_DEL $OPT_LINKDEST $OPT_EXCLUDE $OPT_STATS"
# Functions.
print_dat()
{
echo >> $RLBLOG
date -R >> $RLBLOG
echo >> $RLBLOG
}
print_s()
{
info="$1"
echo $info >> $RLBLOG
}
print_sn()
{
info="$1"
echo -n $info >> $RLBLOG
}
getlinkinfo()
{
linfo="$1"
LINKINFO="Nothing"
TEMP_EXTRACT="Nothing"
# could use readlink
LINKINFO=$(stat $DESTPATH/$linfo | grep "$linfo")
print_sn "LINK: -"
print_s "$LINKINFO"
TEMP_EXTRACT=`expr match "$LINKINFO" '.*\(back-....-..-.._..-..-..\)'`
}
# cd to main directory (mainly redundant).
print_s $DESTPATH
cd $DESTPATH
pwd >> $RLBLOG
# Save to RLBLOG the arguments and options you are running.
print_s
print_s "rsync $OPT $SOURCE $BACKUPNAME"
print_s
# Find last backup directory fron link.
getlinkinfo "current-0"
EXTRACT0=$DESTPATH/$TEMP_EXTRACT
print_sn "EXTRACT0: -"
print_s $EXTRACT0
getlinkinfo "current-1"
EXTRACT1=$DESTPATH/$TEMP_EXTRACT
print_sn "EXTRACT1: -"
print_s $EXTRACT1
getlinkinfo "current-2"
EXTRACT2=$DESTPATH/$TEMP_EXTRACT
print_sn "EXTRACT2: -"
print_s $EXTRACT2
print_s $LINE
# Save to RLBLOG the disk space before the backup.
print_s
print_s "Free Space on $DESTPATH"
df -h $DESTPATH >> $RLBLOG
print_s
print_s $LINE
print_s
print_s "Backup started:"
print_dat
# The main rsync command
rsync $OPT $SOURCE $BACKUPNAME > $OUTPUTLOG
rm $DESTPATH/current-2
mv $DESTPATH/current-1 $DESTPATH/current-2
mv $DESTPATH/current-0 $DESTPATH/current-1
ln -s back-$DATE $DESTPATH/current-0
# Save to RLBLOG the time and disk space after the backup.
print_s "Backup finished:"
print_dat
print_s $LINE
print_s
print_s "Free Space on $DESTPATH"
df -h $DESTPATH >> $RLBLOG
print_s
print_s $LINE
# Save to RLBLOG the tail of OUTPUTLOG & LOGFILE.
print_s
print_s "Statistics"
tail -n15 $OUTPUTLOG >> $RLBLOG
print_s
#print_s
#tail -v -n50 $LOGFILE >> $RLBLOG
#print_s
# Save to RLBLOG a grep of LOGFILE showing the files transferred.
print_s
print_sn "Number of files transferred:"
grep -c '] >f' $LOGFILE >> $RLBLOG
print_s
grep '] >f' $LOGFILE >> $RLBLOG
print_s
# Save to RLBLOG a grep of OUTPUTLOG showing hidden directories.
print_s
print_sn "Number of Excludes:"
grep -c '] hiding' $OUTPUTLOG >> $RLBLOG
print_s
grep '] hiding' $OUTPUTLOG >> $RLBLOG
print_s
print_s $LINE
print_dat
# View backup results but continue processing the rest of the log.
if [ $PREF_POPUP1 -eq 1 ]; then
print_s
print_s "<======> backup has completed so view the results <======>"
print_s "<======> ......continuing to process the log..... <======>"
print_s
$PREF_EDIT $RLBLOG &
fi
# Save to RLBLOG the Total size of 4 backups.
if [ $PREF_DISKUSAGE -eq 4 ]; then
print_s "Actual usage for the last 4 backups:"
du -shc $EXTRACT2 $EXTRACT2 $EXTRACT1 $EXTRACT0 $BACKUPNAME >> $RLBLOG
print_dat
# Save to RLBLOG the Total size of all backups.
elif [ $PREF_DISKUSAGE -eq 9 ]; then
print_s "Actual usage for all backups:"
du -shc back-* >> $RLBLOG
print_dat
else
# Save to RLBLOG the Total size of this backup.
print_s
print_s "Apparent size of backup and actual usage:"
du -shc $BACKUPNAME $BACKUPNAME >> $RLBLOG
print_dat
fi
print_s $LINE
# Save to RLBLOG the usage for each directory.
if [ $PREF_DIRLIST -eq 1 ]; then
print_dat
print_s "Directory List"
du -h $BACKUPNAME >> $RLBLOG
print_dat
fi
# gzip the logs and move them into the backup folder (to keep things tidy).
gzip -c $OUTPUTLOG > $OUTPUTLOG.gz
gzip -c $LOGFILE > $LOGFILE.gz
mv $OUTPUTLOG.gz $BACKUPNAME
mv $LOGFILE.gz $BACKUPNAME
# delete the original, uncompressed logs
rm $OUTPUTLOG
rm $LOGFILE
print_s
print_s "Done."
mv $RLBLOG $BACKUPNAME
# View backup results.
if [ $PREF_POPUP2 -eq 1 ]; then
$PREF_EDIT $BACKUPNAME/back-$DATE-RLBLOG.txt &
fi
An Example Exclude File
/home/*/.gvfs
/home/*/.mozilla/firefox/*/Cache
home/*/.thumbnails
/home2
/home3
/backup
/backup2
/windows
/lib/modules/*/volatile/*
/lost+found
/media
/mnt
/proc
/sys
/tmp/
/var/run
/var/lock
Tested on Ubuntu 8.04/8.10/9.04/9.10/10.04 64 bit.