using rsync for backups

[david.macquigg]

How do I make a hard link to a directory?

$ ln /home/david
ln: hard link not allowed for directory

[CharlesA]

I don't think it hard links the directories, since they are directories, but it does it for all the files. Then it rsyncs the files. *shrugs* It's worked fairly well for me so far.

The script I use is pretty crappy, but it gets the job done:

#!/bin/bash

###
### monthly.sh
### Script for monthly document backup
### Created by Charles
### Tested 10/03/2010
### Updated 10/03/2010
###

BIN=/bin/
USR=/usr/bin/
ARRAYPATH=/array/
MONTHLYPATH=/monthly/
HOME=/home/charles/
ERR=${HOME}logs/monthly_backup_error.log
DATE=$(${BIN}date +%D)
FOLDER=$(${BIN}date +%m%d%Y)

if ${BIN}mountpoint -q $ARRAYPATH
then
  { ${BIN}mount -t ext4 UUID=f00c8f27-7a11-42b0-96f9-0e04a261d682 $MONTHLYPATH; } 2> $ERR || { echo "Mounting of backup drive failed"; exit; }
else
  { echo "$ARRAY not mounted!"; exit; }
fi

OLDMONTHLYNAME=$(${BIN}cat ${MONTHLYPATH}latest)
NEWMONTHLYDIR=$MONTHLYPATH$FOLDER
OLDMONTHLYDIR=$MONTHLYPATH$OLDMONTHLYNAME
MONTHLYLOG=${MONTHLYPATH}log/${FOLDER}.txt

# rsync
if ${BIN}mountpoint -q $MONTHLYPATH
then
  { ${BIN}cp --archive --link $OLDMONTHLYDIR $NEWMONTHLYDIR && echo $FOLDER > ${MONTHLYPATH}latest && ${USR}rsync --archive --itemize-changes --delete --log-file $MONTHLYLOG $ARRAYPATH --exclude dvds --exclude lost+found $NEWMONTHLYDIR && ${BIN}umount $MONTHLYPATH; } 2> $ERR || { echo "Monthly backup failed."; exit; }
fi
# eof

[david.macquigg]

OK, there are no links to directories, therefore, every backup must include a copy of every directory in the tree that is backed up. Try backing up a large tree like my earlier example, after changing just one small file. Run df to see how much disk is actually used before and after the backup. I think you will get the same results as I got.

rsync is just not able to do "Time Machine" style backups, given the limitations of hard links to directories. I've tried this with both ext4 and NTFS filesystems in Ubuntu.

Note: I can make hard links to directories in Mac OSX, so this may be a fundamental difference between these systems.

[CharlesA]

I suppose it would add up after a while, but check this:

charles@thor:~$ du -h --max-depth=1 /cdrom/
1.4T    /cdrom/05312010
15G     /cdrom/07192010
53G     /cdrom/09042010
1.6M    /cdrom/08182010
1.6M    /cdrom/07152010
1.6M    /cdrom/09182010
1.6M    /cdrom/09142010
1.6M    /cdrom/05252010
1.6M    /cdrom/10182010
1.6M    /cdrom/06072010
1.6M    /cdrom/06162010
1.6M    /cdrom/07052010

So, yeah it takes some space, but not all that much if the files haven't changed.

[Vegan]

I use tar and gzip to make backups. The rsync tool is not as effective in my opinion for backup.

I see rsync as more of a replication tool for server roll over.

What I do is use a sump of MySQL and then tar up everything else I want to backup.

Now I have a DVD burner to write too, or I could cp them to another machine. If you have a take drive, that is another option.

tar is a better backup tool compared to other solutions I have seen.

I have looked in this area a lot. And every alternative has fell short compared to the way I do it now.

[david.macquigg]

Vegan, gzipped tar files are really inconvenient for browsing and recovering a single file from an old backup. rsync was designed as an alternative. Each of us has different expectations, but for my small office, the features of the Time Machine are really ideal. I just need to figure out how to make rsync do that more efficiently.

Charles, your cp --archive --link command is indeed making hard links to subdirectories in my destination directory!! This is surprising since not even ln will do that. This also tells me that the prohibition against hard linking to directories is not a limitation of Linux or of the file system, but something specific to rysnc and ln. I'll have to experiment with this and get back to you.

[CharlesA]

Glad you are able to figure it out.

I don't use tar because it would be a major pain in the butt to grab a specific file from within the tar archive without using something like 7zip to open the archive.

[david.macquigg]

I've got some results comparing Charles' method to that in RLBackup. Using cp -al to set up the link tree *before* the rsync command is orders of magnitude faster and smaller than using the --link-dest option in rsync.

In both tests, I backed up my home directory (2125 files, 1.34GB). In both tests, I started with a backup already in place, and did an incremental backup to a new directory. 26 files were updated, with a total data transfer of 90MB.

For TestA I ran RLBackup with no modification other than setting the directories for source, destination, and exclude files.
http://ubuntuforums.org/showthread.p...t=rsync+backup

For Test B I left out the --link-dest option on the rsync command, and inserted a cp -al command just ahead of rsync. The hacked script then has these lines:

LASTBAK=back-2010-11-09_11-32-35       ### need to automate this
cp -al $DESTPATH/$LASTBAK $BACKUPNAME     ### make links with a local copy
rsync $OPT $SOURCE $BACKUPNAME > $OUTPUTLOG   # no link-dest in $OPT

                        Test A     Test B
 Time (seconds):            59          3
 Disk usage (MB):         1300         87

[CharlesA]

Wow. Thanks for posting the results. I didn't know it would make that much of a difference.

You need to automate creating the folder for the new backup, right?

My script does that based on the output of date, so maybe that'll be of some use to you.

[senor_smile]

OK, there are no links to directories, therefore, every backup must include a copy of every directory in the tree that is backed up. Try backing up a large tree like my earlier example, after changing just one small file. Run df to see how much disk is actually used before and after the backup. I think you will get the same results as I got.

rsync is just not able to do "Time Machine" style backups, given the limitations of hard links to directories. I've tried this with both ext4 and NTFS filesystems in Ubuntu.

Note: I can make hard links to directories in Mac OSX, so this may be a fundamental difference between these systems.

I would recommend trying out backintime if you want a comparable solution to the mac's time machine. I use it for backing up my home directory while at work. It requires a gui (i.e. gnome or kde). It uses rsync, diff and hard links to do exactly what it sounds like you want to do. It might not work for exactly what you need, but I wanted to throw it out there as no one else had.