TUTORIAL: Automated backups with rsync & ssh

[Sir_Yaro]

Exactly Gourgi.
I think it's very secure as long you keep your private key safe.

[durus]

Here's a quick little script I wrote for backing up a mac laptop to a linux system. Then modified to back up an ubuntu system to another box.

basically the structure will look similar to this
(output of tree -L 3 with some cutting)
.
`-- backup
|-- now
| |-- torrents
| `-- workspace
`-- old
|-- 2006-01-05-at-23.47.51
|-- 2006-01-05-at-23.49.20
|-- 2006-01-06-at-00.12.29
|-- 2006-01-06-at-00.12.42
|-- 2006-01-06-at-00.13.19
|-- 2006-01-06-at-00.13.54
`-- 2006-01-06-at-19.37.40

Where the stuff in old contains files/directories that were changed or deleted at that time.

Excludes may be useful if you have a large amount of data that you really could care less about if you lose it, say you akready have a backup, or the originals of the data.

Uncomment and recomment the parts about $tobebackedup if you just want certain directories.

#!/usr/bin/perl
# Kyle Byerly
# 20.December.2005
# Backs up everything to another system...don't know (or care) if it backs up
# the resource fork since the system I'm backing up to is linux.
# Perhaps at some later date I will worry about this.
# Backup to a mac from a mac using rsyncX.
$user = kyle;
$base = "/home/$user";
#if no leading slash then starts in home directory of remote login user
#pay attention to what will happen with --backup-dir
$backupdir = "backup";
#@dirs = < Documents Library Pictures perl amesfug apocalyptica backups classes english software #starbuck wireless lastbackup>;
#foreach (@dirs) {
#    $tobebackedup = "$tobebackedup $base/$_";
#}
$tobebackedup = "$base/";
#path from backup directory if starting with /
#if it ends with a / it is a directory, else it is a file
#if no leading / then it will exclude if it finds the word anywhere
#be careful here
@excludes =< /DVD /Music/ /source/ /archive.org/ Cache/ /.gnome-system-monitor.kyle >;
foreach (@excludes) {
    $excluding = "$excluding --exclude=\"$_\"";
}

#the below two lines will make it so I know when the last (complete) backup was
#as lastbackup is the last dir
system "rm $base/lastbackup/*";
system "touch $base/lastbackup/\"`date +%Y-%m-%d-at-%H.%M.%S`\"";
#notice \@ to comment out the @
system "rsync -rltb -e \"ssh -p 24\" $excluding --delete --backup-dir=~/$backupdir/old/`date +%Y-%m-%d-at-%H.%M.%S`/ $tobebackedup kyle2800\@192.168.2.13:$backupdir/now";

you can "execute" the script with either

perl nameofscript

or

chmod +x nameofscript 
./nameofscript

have fun

Thank you for this excellent script. Say that I have two backups
2008-06-26-at-00.05.00
2008-06-26-at-00.10.28
Then it is easy to go back to 2008-06-26-at-00.05.00, but how do I go back to 2008-06-26-at-00.10.28 ?

Thanks
Durus

[bayvista]

i have followed your Howto, but still get asked for a password. This is what I get:

david@david-desktop:~$ ssh david-desktop
The authenticity of host 'david-desktop (127.0.1.1)' can't be established.
RSA key fingerprint is xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (zapped by me)
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'david-desktop' (RSA) to the list of known hosts.
david@david-desktop's password:
Linux david-desktop 2.6.24-23-generic #1 SMP Thu Nov 27 18:44:42 UTC 2008 i686

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Also, I need to execute the "exportfs" command after every reboot. Is there any way of saving this info?

The only strange response was to the ssh-copy-id command. In the end, I copied the id_dsa.pub to the destination machine to /home/.ssh.

Any suggestions welcome.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To access official Ubuntu documentation, please visit:
http://help.ubuntu.com/
Last login: Thu Jan 22 17:49:23 2009 from hp-laptop
david@david-desktop:~$

[bayvista]

Well, forget all that! I got it working at last by doing a keygen on the other machine. Now I can SSH to my laptop without a password. Next step is to setup Crontab.

I believe that it is important to secure the .ssh directory. I suggest that you add some instructions for this.

Thanks for an excellent Howto.

[bayvista]

Well, I spoke too soon. I can now connect OK to my laptop, but get the following errors when I RSYNC:

david@david-desktop:~$ rsync -varuzP hp-laptop:/home/pam/ /media/xpmedia/backups/hp-home-pam/>>/home/david/cronlog.txt
rsync: opendir "/home/pam/.sane" failed: Permission denied (13)
rsync: opendir "/home/pam/.nautilus/metafiles" failed: Permission denied (13)
rsync: opendir "/home/pam/.config/tracker" failed: Permission denied (13)
rsync: opendir "/home/pam/.config/gtk-2.0" failed: Permission denied (13)
rsync: opendir "/home/pam/.config/totem" failed: Permission denied (13)
rsync: opendir "/home/pam/.gnome2_private" failed: Permission denied (13)
rsync: opendir "/home/pam/.evolution/calendar/local" failed: Permission denied (13)
rsync: opendir "/home/pam/.evolution/tasks/local" failed: Permission denied (13)
rsync: opendir "/home/pam/.evolution/memos/local" failed: Permission denied (13)
rsync: opendir "/home/pam/.mozilla" failed: Permission denied (13)
rsync: opendir "/home/pam/.gnupg" failed: Permission denied (13)
rsync: opendir "/home/pam/.macromedia" failed: Permission denied (13)
rsync: opendir "/home/pam/.gconfd" failed: Permission denied (13)
rsync: opendir "/home/pam/.openoffice.org2" failed: Permission denied (13)
rsync: opendir "/home/pam/.thumbnails" failed: Permission denied (13)
rsync: opendir "/home/pam/.adobe" failed: Permission denied (13)
rsync: opendir "/home/pam/.update-notifier" failed: Permission denied (13)
rsync: opendir "/home/pam/.grsync" failed: Permission denied (13)
rsync: opendir "/home/pam/.metacity" failed: Permission denied (13)
rsync: opendir "/home/pam/.ssh" failed: Permission denied (13)
rsync: opendir "/home/pam/.gnome2" failed: Permission denied (13)
rsync: opendir "/home/pam/.gvfs" failed: Permission denied (13)
rsync: opendir "/home/pam/.local/share/Trash" failed: Permission denied (13)
rsync: opendir "/home/pam/.qt" failed: Permission denied (13)
rsync: opendir "/home/pam/.mozilla-thunderbird" failed: Permission denied (13)
rsync: opendir "/home/pam/.hplip" failed: Permission denied (13)
rsync: opendir "/home/pam/.gconf" failed: Permission denied (13)
rsync error: some files could not be transferred (code 23) at main.c(1385) [generator=2.6.9]
david@david-desktop:~$

Well, of course. My wife had not closed the applications. Once closed, all went fine

[bayvista]

A question for the Rsync gurus. It seems that you cannot archive an Rsync backup to a DVD. I keep getting error messages saying that I have too many directory levels. I think 6 is the maximum allowed.

To solve this problem, I am using TAR to archive my backups to DVD. Is there a cleaner way?

[bayvista]

Many thanks for this Rosslaird.

My wife's laptop died on Wed last. Blank screen. Using this Tutorial I had successfully backed up her Home directory every night to the Desktop. It was very easy to just restore the /home directory and logon as her. Bingo, everything just worked, including Email.

A great Tutorial

[rosslaird]

Glad you found it useful. This tutorial sure has had a long life. Now, I will also suggest another thing: backup your data online, using a getdropbox.com free account (2 gigs) and rsync with a local folder. Works like a charm.

Ross

[RgnKjnVA]

I was running this configuration like a champ for quite some time backing up my Hardy PC to my Intrepid laptop and vice versa but suddenly last night it stopped working. I get ssh timeouts from Intrepid laptop to Hardy PC but I can ssh the PC to the laptop although I get prompted for a password. I can also ping each machine from the other without a problem and nslookup works for each from the other. I also tried all of the above using IP instead of hostname to no avail. WTH?! Do those keys expire or something? To my knowledge I haven't done anything that would have affected networking nor connectivity.

laptop:~$ ssh -vv hardypc.home
OpenSSH_5.1p1 Debian-3ubuntu1, OpenSSL 0.9.8g 19 Oct 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to hardypc.home [192.168.1.8] port 22.

[RgnKjnVA]

[SOLVED] Wife restarted the router and my laptop was assigned a different IP than it had previously. I'm off to learn how to make IPs static.