HOWTO: Backup multimedia with rsync

[djgrandmarquis]

The Ubuntu forums contain several great posts about fully backing up a Linux system. I use this one regularly: http://ubuntuforums.org/showthread.php?t=35087

But this method doesn't work very well for large, numerous multimedia files. The above process yields a single tarball, which is not ideal for accessing music or movies. As a digital DJ, I need to carry around all my music on an external hard drive even though my collection is stored on my desktop.

My woes began with *******, where trying to backup a large number of files was painful at best. My needs were pretty straightforward: I wanted a one-way mirror of all my multimedia, with the ability to not only copy files from Kubuntu to the external disk, but also with the ability to remove files that I deleted on my desktop machine. I also needed incremental backup capability, since I don't want to copy the same gigabytes of data over and over.

Linux offers several tools for backing up/mirroring files. Some include Unison (http://www.cis.upenn.edu/~bcpierce/unison/), fullSync(http://fullsync.sourceforge.net/), and rsync(http://samba.anu.edu.au/rsync/). (Unison is available from the Ubuntu repositories, fullSync is a downloadable Java app, and rsync is a command-line tool that comes with Linux.) This howto documents my success with rsync.

At first I formatted my external hard drive with FAT32(http://en.wikipedia.org/wiki/Fat32), so that my XP laptop and my Kubuntu desktop could read and write to it. However, FAT32 lacks several keys features that make it difficult to use with synchronization tools:
-case-insensitive file names
-no user permissions
-goofy time stamps (they vary by significant amounts)
-it's slow (it's a very old file system)

The result is that automated backups of files are nearly impossible with FAT32. So my solution was to reformat the drive as ext3(http://en.wikipedia.org/wiki/Ext3). I found a nice driver to read and write ext3 in Windows, ext2IFS(http://www.fs-driver.org/). (related Ubuntu forum: http://ubuntuforums.org/showthread.php?t=115717)

With everything in place, it was time to start backing up! Unison is designed for two-way synchronization, which I'm not interested in; if I accidentally delete a file on my external drive, I surely don't want it to get deleted on my desktop. FullSync never installed properly (it didn't like my pango settings, I assume because of 64-bit incompatibilities), so rsync emerged as my tool of choice.

Here's my backup script, to automate the process:

#!/bin/sh

# rsync.sh
#
# - this script will back up my 4 dirs that need to be synchronized with 
# (mirrored to) my external ext3 usb disk:
# 
# /home/<username>/docs (small)
# /home/<username>/pics (small)
# /home/<username>/thor (large)
# /home/<username>/music (very large)
# 
# - can be executed by user (no need for sudo/root access)
# - other dirs (video, tgz, etc.) must be updated by hand
# 

echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"
echo "(1) Backing up docs..."
echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"

ls /home/<username>/docs
rsync -vurt --progress --delete /home/<username>/docs/ /mnt/usbdevice/docs/

echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"
echo "(2) Backing up pics..."
echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"

ls /home/<username>/pics
rsync -vurt --progress --delete /home/<username>/pics/ /mnt/usbdevice/pics/

echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"
echo "(3) Backing up thor..."
echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"

ls /home/<username>/thor
rsync -vurt --progress --delete /home/<username>/thor/ /mnt/usbdevice/thor/

echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"
echo "(4) Backing up music..."
echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"

ls /home/<username>/music
rsync -vurt --progress --delete /home/<username>/music/ /mnt/usbdevice/music/

echo ""
echo "~*~*~*~*~*~*~*~*~*~*~*~*~*"
echo "All done!"

I run this script from ~/ but you can run it from wherever you like. Make sure the script is executable:

chmod 755 rsync.sh

Simply run, and everything will be mirrored to the USB drive.

./rsync.sh

Some explanation of the various rsync flags (info from rsync's man page):

-v: verbose - this flag makes rsync tell you what files it's moving and gives a summary at the end
-u: update - this forces rsync to skip any files for which the destination file already exists and has a date later than the source file
-r: recursive - rsync will mirror the directories you specified and all the directories inside them.
-t: times - rsync will copy al the timestamps of the source files, so that the files on the external drive are exact replicas.
--progress: rsync will tell you how many files need to be copied before it's finished. this is useful if you want to see what's going on.
--delete: this tells rsync to delete any files on the receiving side that aren't on the sending side.

Remember, backup often! Linux makes it incredibly easy to do, unlike Microsoft products. And don't worry about files being 'in use,' that's a Windows ailment. rsync is a very powerful tool, and it's also useful for synchronizing to a network drive. It can exclude files if you choose, and the '-n' flag will let you preview changes before they are actually made. Read about rsync in the man page:

man rsync

[Lunar_Lamp]

Thanks for this. I see nobody else has responded, but this is pretty much what I was looking for. I too needed a simple backup solution, that was NOT automated (i.e. I just did it when I wanted to). I knew rsync was what I needed to use, but this script has meant that I barelyhave to use any effort in reading the rsync man-pages to write my own!

[senorJ]

Thanks that's really useful and just what I needed right now.

As a linux noobie that was my first attempt at using a bash script and I can feel myself swelling with pride and self satisfaction

Cheers
Jay

[xander848]

You mentioned that the external drive had to be formatted at ext3. Could the partition that I am backing up from be FAT32 though?

[djgrandmarquis]

You mentioned that the external drive had to be formatted at ext3. Could the partition that I am backing up from be FAT32 though?

Unfortunately, no. rsync does not play well with FAT32. I think there were issues with case sensitivity and time stamping. I couldn't get rsync to properly update the files incrementally; it just copied all the files. I've never tried it with FAT32 as the source drive, but I imagine the problems would be the same.

[kefkakiller]

Great guide, I think this is exactly what I need to sync my music on my laptop and my media server. I have one question on the -u flag, though. Am I correct in assuming that it will only update files with changed timestamps ie it will backup an mp3 whose ID3 tag has been changed, even though the filename is the same?

[guymauve]

Hi,

Any body get that problem... I am keeping lost connection with unison...

I reset ssh conf and keys, remove firewall (firestarter) set nat translating between both computer TCP UDP on, reset unison conf...

Nothing

Keep lost connection

The gui gtk interface ask for passwd and everything and when the authentification start, it pop lost connection...

Where could it coming from?

[guymauve]

I get by using unison in command line the error from bash :

bash: unison : commande introuvable

That mean that that it steel has some correction to do to the package since the package and invoking command change for :

unison-2.13.16

There should be a remaining unison command straigh somewhere that conflict with the ssh protocol...

After the answering of the user identification by openssh-server code somewhere....

???

Thanks for the correction if possible...

And sorry for the post in the midle of nowhere, I was no able to publish a new thread cause I just join the forum...

Guy

[djgrandmarquis]

Great guide, I think this is exactly what I need to sync my music on my laptop and my media server. I have one question on the -u flag, though. Am I correct in assuming that it will only update files with changed timestamps ie it will backup an mp3 whose ID3 tag has been changed, even though the filename is the same?

Yes, I think you are correct. If you change an ID3 tag on your media server, the -u flag will overwrite the file on your laptop to reflect the changes. rsync compares the timestamps of files and updates them if the server's time stamp is newer than the client's. The -u flag ignores client files that have matching names and newer (or same) timestamps.

[guymauve]

Bug report here : https://launchpad.net/distros/ubuntu...son/+bug/58134

Sorry for the missleading of this one...

Guy