Page 1 of 4 123 ... LastLast
Results 1 to 10 of 32

Thread: TUTORIAL: Automated backups with rsync & ssh

  1. #1
    Join Date
    Feb 2005
    Location
    Vancouver, BC
    Beans
    329

    TUTORIAL: Automated backups with rsync & ssh

    Several times I've had to configure automated backups for my machines, and each time I do, a long enough period has elapsed since the previous time that I forget all the important details. So this little tutorial is as much to help me remember (and to keep a record and to trace the steps) as for other users.

    1. What You Need

    a) ssh (should be installed by default) and rsync ("sudo apt-get install rsync" if you don't already have it).

    b) Users on both machines (the source machine and the destination machine), ideally with the same username (so: if user "ross" exists on source machine, user "ross" should also exist on destination machine).

    c) ssh and rsync installed on both machines.


    2. Testing ssh (also see step 4)

    Start on the source machine. As user ("ross" in my case -- not root) try to login to the destination machine using ssh, specifying either an ip address or a hostname (if the hostname appears in /etc/hosts). You'll be asked for a password. This is the password for the user on the destination machine, not the local password (in my case, they're the same). For me, the destination machine is called "kids":

    Code:
    ross@ross:~$ssh kids
    (I will show my full prompt several times in this tutorial, but obviously you just enter the last part. For above, it would be "ssh kids", or whatever your hostname/ip is.)

    Type yes to the authentication message.
    Enter the password. (If you are logging on as another user, you can type the username before the host, i.e.: ssh ross_other_user@kids).
    All good so far. ssh works. If it doesn't use google to search for the error message.
    Type "exit" to return to the source machine.

    3. Setup public/private key pair

    Code:
    ross@ross:~$ ssh-keygen -t dsa
    Follow the prompts, just hitting enter for the passphrase. This will yield the id_dsa.pub and id_dsa files (the public and private key pair):

    Code:
    ...Generating public/private dsa key pair. [Enter]
    ...Enter file in which to save the key (/home/ross/.ssh/id_dsa): [Enter]
    ...Created directory '/home/ross/.ssh'. [Enter: you might not see this message]
    ...Enter passphrase (empty for no passphrase): [Enter]
    ...Enter same passphrase again: [Enter]
    ...Your identification has been saved in /home/ross/.ssh/id_dsa.
    ...Your public key has been saved in /home/rick/.ssh/id_dsa.pub.
    4. Copy the public key to the destination machine:

    Code:
    ross@ross:~$ ssh-copy-id -i ~/.ssh/id_dsa.pub ross@kids [or enter ip address instead of hostname, e.g."kids"]
    If you already tested ssh as in step 2 above, you'll simply be asked for the passphrase. If not, you'll get this:

    Code:
    The authenticity of host '...[host ip or name]...' can't be established.
    RSA key fingerprint is ...
    Are you sure you want to continue connecting (yes/no)? [type "yes"]
    Warning: Permanently added '...' (RSA) to the list of known hosts.
    ross@[kids]'s password: [Enter the password]
    You should be logged in to the remote machine, and you should (might) see a test message about confirming the setup.

    5. Logout of the destination machine and try getting back in without the password:

    Code:
    ssh kids
    This (with your correct username and hostname/ip address) should get you in to the destination machine. If it works, it will work every time for all ssh connections for that user. No password required (this is actually a small security risk, but it has a big payoff -- see next step.

    6. Make a backup script (mine's in my home directory, but it could be anywhere):

    Code:
    nano backup.txt
    Enter the rsync options and paths into the new file. The following line will login (with user "ross", for which we just setup password-less ssh logins) and rsync the home directory on my source machine with a backup directory on the destination machine ("kids"):

    Code:
    rsync -e ssh -varuzP /home/ross/ kids:/home/ross/backup/
    The closing slashes matter. Don't leave them out. The varuzP options mean "verbose, archive, recursive, update, compress, partial". Check "rsync -h" from a command prompt to see what this all means. Basically, it will create directories if they don't exist, won't overwrite newer files on the destination, and won't delete any files on the destination that have been deleted from the source machine (for that, you use "--delete").

    (If you did not setup password-less logins, the script would halt and ask for a password. And this would prevent the procedure from being automated.)

    Save and close backup.txt
    From the directory in which backup.txt was saved, type:

    Code:
    ./backup.txt
    If it balks (i.e. permissions), change the permissions. I would use "chmod 775 backup.txt", though this is a fairly lax permission scheme. (I have a personal network, which only I and my kids use, and I'm behind a hardware firewall. Your setup may have different requirements. At least make the script exectuable ("chmod +x backup.txt", I think, but permissions are not my specialty).

    Run the backup and see how it goes. If it works and you like what it's doing, go to step 7. Otherwise, adjust it.

    7. Automate the process.

    make a new file called crons.cron (I think you can call it anything, but I'not sure):

    Code:
    nano crons.cron
    Put some cron variables in there. Mine says this:

    Code:
    #Mins   Hours   Days    Months  Day of Week     Run program
       0        23        *          *            *                       /home/ross/backup.txt
    The titles are just reminders for me. They are commented out, as you can see. This script runs my backup.txt script every day at 11pm.

    Save and exit crons.cron
    Add it to your crontab:

    Code:
    crontab crons.cron
    List your crontab, to check that it's in there properly:

    Code:
    crontab -l
    (that's an L, as in "list")

    You should get a nice little message showing your cron settings.

    Done.

    A Final Word:

    Don't just follow this tutorial willy-nilly. Your system may have a configuration that makes parts of what I'm outlining dangerous or foolish. Try to understand (at least minimally) what you are doing. This setup does one thing only: it syncronizes files from a directory on a local machine (the source) to a directory on a destination machine. It does not make multiple copies of anything (as a true backup scheme would) and it will overwrite matching files on the destination unless they are newer. Be cautious. Rsync is an amazingly powerful tool, but with great power...

    ___________

    Other users will likely have better or more efficient ways of doing this. Please do post them so folks can see what options there are. And, of course, I may have made a host of mistakes that I don't see at the moment. Please help me correct them.

    I hope this helps to keep someone's data safe.
    Last edited by rosslaird; February 12th, 2005 at 01:14 AM.
    Ideas matter.
    Ross A. Laird

  2. #2
    Join Date
    Dec 2004
    Beans
    40

    Re: TUTORIAL: Automated backups with rsync & ssh

    If the goal is to synchronize two computers (e.g. a laptop and a desktop), I can recommend Unison (works also great for backups):
    http://www.cis.upenn.edu/~bcpierce/unison/
    apt-get install unison unison-gtk

  3. #3
    Join Date
    Oct 2005
    Beans
    6

    Re: TUTORIAL: Automated backups with rsync & ssh

    Here's a quick little script I wrote for backing up a mac laptop to a linux system. Then modified to back up an ubuntu system to another box.

    basically the structure will look similar to this
    (output of tree -L 3 with some cutting)
    .
    `-- backup
    |-- now
    | |-- torrents
    | `-- workspace
    `-- old
    |-- 2006-01-05-at-23.47.51
    |-- 2006-01-05-at-23.49.20
    |-- 2006-01-06-at-00.12.29
    |-- 2006-01-06-at-00.12.42
    |-- 2006-01-06-at-00.13.19
    |-- 2006-01-06-at-00.13.54
    `-- 2006-01-06-at-19.37.40

    Where the stuff in old contains files/directories that were changed or deleted at that time.

    Excludes may be useful if you have a large amount of data that you really could care less about if you lose it, say you akready have a backup, or the originals of the data.

    Uncomment and recomment the parts about $tobebackedup if you just want certain directories.

    Code:
    #!/usr/bin/perl
    # Kyle Byerly
    # 20.December.2005
    # Backs up everything to another system...don't know (or care) if it backs up
    # the resource fork since the system I'm backing up to is linux.
    # Perhaps at some later date I will worry about this.
    # Backup to a mac from a mac using rsyncX.
    $user = kyle;
    $base = "/home/$user";
    #if no leading slash then starts in home directory of remote login user
    #pay attention to what will happen with --backup-dir
    $backupdir = "backup";
    #@dirs = < Documents Library Pictures perl amesfug apocalyptica backups classes english software #starbuck wireless lastbackup>;
    #foreach (@dirs) {
    #    $tobebackedup = "$tobebackedup $base/$_";
    #}
    $tobebackedup = "$base/";
    #path from backup directory if starting with /
    #if it ends with a / it is a directory, else it is a file
    #if no leading / then it will exclude if it finds the word anywhere
    #be careful here
    @excludes =< /DVD /Music/ /source/ /archive.org/ Cache/ /.gnome-system-monitor.kyle >;
    foreach (@excludes) {
        $excluding = "$excluding --exclude=\"$_\"";
    }
    
    #the below two lines will make it so I know when the last (complete) backup was
    #as lastbackup is the last dir
    system "rm $base/lastbackup/*";
    system "touch $base/lastbackup/\"`date +%Y-%m-%d-at-%H.%M.%S`\"";
    #notice \@ to comment out the @
    system "rsync -rltb -e \"ssh -p 24\" $excluding --delete --backup-dir=~/$backupdir/old/`date +%Y-%m-%d-at-%H.%M.%S`/ $tobebackedup kyle2800\@192.168.2.13:$backupdir/now";
    you can "execute" the script with either
    Code:
    perl nameofscript
    or
    Code:
    chmod +x nameofscript 
    ./nameofscript
    have fun

  4. #4

    Re: TUTORIAL: Automated backups with rsync & ssh

    If you are running SSH on a port other than 22 (let's assume 4000 for this example). You must alter the following steps:

    Step 2 - When testing SSH:
    Code:
    ssh -p 4000 kids
    Step 4 - When copying the key:
    Code:
    ssh-copy-id -i ~/.ssh/id_dsa.pub "-p 4000 ross@kids"
    Step 5 - When testing SSH again:
    Code:
    ssh -p 4000 kids
    Step 6 - When creating an rsync command:
    Code:
    rsync -e "ssh -p 4000" -varuzP /home/ross/ kids:/home/ross/backup/
    Also be sure not to run any of these commands as root (eg. sudo ssh ...) or you will end up with keys files and keys all over the place. If for some reason you get to the end and your rsync is not running. Make sure to run chown on the remote folder in question. Obviously this is not a tutorial on permissions but this is a common problem that people overlook.

  5. #5
    Join Date
    May 2007
    Beans
    3

    Re: TUTORIAL: Automated backups with rsync & ssh

    Awesome! Can someone explain the switches?

    I am also getting
    rsync: failed to set permissions on "......": Operation not permitted (1)

  6. #6
    Join Date
    Oct 2007
    Beans
    10

    Re: TUTORIAL: Automated backups with rsync & ssh

    Absolutely excellent tutorial - there probably IS a tool oout there that will do all this for me automatically, but then I wouldn't have learnt all the incidental stuff that I picked up along the way. I used this tutorial to back up an opensuse 10.2 laptop to an older laptop running Ubuntu edgy and it worked perfectly. Thanks for sharing your knowledge and in language I was able to understand!!!

  7. #7
    Join Date
    Apr 2008
    Beans
    29

    Re: TUTORIAL: Automated backups with rsync & ssh

    Thank you so much for the a tutorial! I tried all day long to try and make this happen between two Windows servers, and it was a such a headache...

    Now I will be using to ubuntu virtual machines to do the job...

    Just wanted to say thanks and great job!

  8. #8
    Join Date
    Apr 2008
    Beans
    29

    Re: TUTORIAL: Automated backups with rsync & ssh

    Thank you dk01 for adding the part about alternative ports, that was very helpful!!

  9. #9
    Join Date
    Sep 2005
    Location
    Virginia
    Beans
    23
    Distro
    Ubuntu 9.04 Jaunty Jackalope

    Re: TUTORIAL: Automated backups with rsync & ssh

    Just FYI

    Using passphrase less keys IS NOT SECURE at all... if you are doing this on a local network (ie inside your house) its "semi-ok" but not for backing up across the internet!!!

    REMEMBER EMPTY PASSPHRASES ARE THE DEVIL!!!!
    Last edited by Toky; June 21st, 2008 at 07:57 AM.

  10. #10
    Join Date
    Sep 2007
    Location
    Pteleos Greece
    Beans
    408
    Distro
    Ubuntu Development Release

    Re: TUTORIAL: Automated backups with rsync & ssh

    Quote Originally Posted by Toky View Post
    Using passphrase less keys IS NOT SECURE at all...
    REMEMBER EMPTY PASSPHRASES ARE THE DEVIL!!!!
    why is that ?
    i 'm supposed to be secure as long as i own my private key.
    isn't that correct?
    "Computers are like air conditioners, when you open WINDOW$ they stop working."
    Όσο ζώ μαθαίνω ...
    If Microsoft ever does applications for Linux it means I've won.
    Linus Torvalds

Page 1 of 4 123 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •