HOWTO: Backup multimedia with rsync

[djgrandmarquis]

The Ubuntu forums contain several great posts about fully backing up a Linux system. I use this one regularly: http://ubuntuforums.org/showthread.php?t=35087

But this method doesn't work very well for large, numerous multimedia files. The above process yields a single tarball, which is not ideal for accessing music or movies. As a digital DJ, I need to carry around all my music on an external hard drive even though my collection is stored on my desktop.

My woes began with Windoze, where trying to backup a large number of files was painful at best. My needs were pretty straightforward: I wanted a one-way mirror of all my multimedia, with the ability to not only copy files from Kubuntu to the external disk, but also with the ability to remove files that I deleted on my desktop machine. I also needed incremental backup capability, since I don't want to copy the same gigabytes of data over and over.

Linux offers several tools for backing up/mirroring files. Some include Unison (http://www.cis.upenn.edu/~bcpierce/unison/), fullSync(http://fullsync.sourceforge.net/), and rsync(http://samba.anu.edu.au/rsync/). (Unison is available from the Ubuntu repositories, fullSync is a downloadable Java app, and rsync is a command-line tool that comes with Linux.) This howto documents my success with rsync.

At first I formatted my external hard drive with FAT32(http://en.wikipedia.org/wiki/Fat32), so that my XP laptop and my Kubuntu desktop could read and write to it. However, FAT32 lacks several keys features that make it difficult to use with synchronization tools:
-case-insensitive file names
-no user permissions
-goofy time stamps (they vary by significant amounts)
-it's slow (it's a very old file system)

The result is that automated backups of files are nearly impossible with FAT32. So my solution was to reformat the drive as ext3(http://en.wikipedia.org/wiki/Ext3). I found a nice driver to read and write ext3 in Windows, ext2IFS(http://www.fs-driver.org/). (related Ubuntu forum: http://ubuntuforums.org/showthread.php?t=115717)

With everything in place, it was time to start backing up! Unison is designed for two-way synchronization, which I'm not interested in; if I accidentally delete a file on my external drive, I surely don't want it to get deleted on my desktop. FullSync never installed properly (it didn't like my pango settings, I assume because of 64-bit incompatibilities), so rsync emerged as my tool of choice.

Here's my backup script, to automate the process:

#!/bin/sh

# rsync.sh
#
# - this script will back up my 4 dirs that need to be synchronized with 
# (mirrored to) my external ext3 usb disk:
# 
# /home/<username>/docs (small)
# /home/<username>/pics (small)
# /home/<username>/thor (large)
# /home/<username>/music (very large)
# 
# - can be executed by user (no need for sudo/root access)
# - other dirs (video, tgz, etc.) must be updated by hand
# 

echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"
echo "(1) Backing up docs..."
echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"

ls /home/<username>/docs
rsync -vurt --progress --delete /home/<username>/docs/ /mnt/usbdevice/docs/

echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"
echo "(2) Backing up pics..."
echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"

ls /home/<username>/pics
rsync -vurt --progress --delete /home/<username>/pics/ /mnt/usbdevice/pics/

echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"
echo "(3) Backing up thor..."
echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"

ls /home/<username>/thor
rsync -vurt --progress --delete /home/<username>/thor/ /mnt/usbdevice/thor/

echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"
echo "(4) Backing up music..."
echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"

ls /home/<username>/music
rsync -vurt --progress --delete /home/<username>/music/ /mnt/usbdevice/music/

echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"
echo "All done!"

I run this script from ~/ but you can run it from wherever you like. Make sure the script is executable:

chmod 755 rsync.sh

Simply run, and everything will be mirrored to the USB drive.

./rsync.sh

Some explanation of the various rsync flags (info from rsync's man page):

-v: verbose - this flag makes rsync tell you what files it's moving and gives a summary at the end
-u: update - this forces rsync to skip any files for which the destination file already exists and has a date later than the source file
-r: recursive - rsync will mirror the directories you specified and all the directories inside them.
-t: times - rsync will copy al the timestamps of the source files, so that the files on the external drive are exact replicas.
--progress: rsync will tell you how many files need to be copied before it's finished. this is useful if you want to see what's going on.
--delete: this tells rsync to delete any files on the receiving side that aren't on the sending side.

Remember, backup often! Linux makes it incredibly easy to do, unlike Microsoft products. And don't worry about files being 'in use,' that's a Windows ailment. rsync is a very powerful tool, and it's also useful for synchronizing to a network drive. It can exclude files if you choose, and the '-n' flag will let you preview changes before they are actually made. Read about rsync in the man page:

man rsync