How to auto rsync with ssh passwordless

[patsissons]

This HOWTO is designed to help those who are looking to backup a directory from their local computer to a remote computer using the power of rsync, the security of ssh, and the ease of automation. Your local computer will make use of a cron job that will execute the backup as often as you want, behind the scene so that you don't have to worry about your data's saftey.

There will be a quick-start rundown of the steps at the end of this howto.

First we start off by making a public key on the local machine

ssh-keygen -t rsa

use -t rsa unless you plan on accessing a older machine, or rather a machine whose version of openSSH is older. In most cases you will not have a problem with -t rsa. However, if you do have a problem you can try leaving it out, this will create a DSA key instead. ssh-keygen will ask you first where to store the public key. The default location is usually fine, that is unless you want to manage multiple public keys. Unless multiple public keys is what you need, simply hit enter to accept the default location. Next you will be asked for a password. You must hit enter twice without typing in a password or else every time you use this public key you will have to enter that password instead. This should result in the creation of the key and the public key pair. If you chose the default values, ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub.

Now we need to copy our public key to the remote machine so the remote machine can add it to its list of authorized keys.

ssh-copy-id -i ~/.ssh/id_rsa.pub username@remote_host

Of course, you should use your custom public key if you did not use the default name in the previous step. Also as you would expect, username and remote_host should be replaced by their respective values. After executing this, it will ask you for your password, this is just the ssh password to the remote machine for the username that you used. Upon completion, there should be a file on the remote machine ~/.ssh/authorized_keys that contains the public key that you just generated.

You can test out if you were successful now by ssh'ing to the remote machine, you should no longer be asked for a password. If this is the case, you are in the clear, otherwise, something has gone wrong.

To setup rsync you will need to make a script that looks something like this.

#!/bin/sh
rsync -e 'ssh -p 22' -avzp /some/dir remote_host:/var/backups/some_host

The -e 'ssh -p 22' is not completely necessary (in fact its very redundant), however, if you are connecting to ssh on a non standard port, you will need to change the 22. Otherwise, you can just use -e ssh instead. Again, remember to change the remote_host to the actual hostname of the remote server. Now depending on how often you want to run this backup, you can either setup a specific entry in the /etc/crontab file (this is a little more complicated), or you can just use the pre-built cron directories in ubuntu (cron.hourly, cron.daily, cron.weekly, cron.monthly). I recommend the second option, since it makes for a very easy setup. Simply save the script you created inside the desired cron directory, then chmod +x the filename that you save it as. If you wish to test it out, simply execute the script from the console.

Finally, make sure that the directory that you are sending your backups to on the remote server actually exists, otherwise rsync will error out and you will not backup anything.


Quick-Start
On the local machine:

ssh-keygen -t rsa
# hit return three times

ssh-copy-id -i ~/.ssh/id_rsa.pub username@remote_host
# enter your password for username on remote_host

cat > /etc/cron.daily/remote_backup
#!/bin/sh
rsync -e 'ssh -p 22' -avzp /some/dir remote_host:/var/backups/some_host
^D

chmod +x /etc/cron.daily/remote_backup

ssh username@remote_host mkdir /var/backups/some_host

Final Thoughts
I am no rsync expert, nor ssh or cron. Actually, I learnt most of this stuff in the last couple of hours. But since there was no tutorial on ubuntu forums, I decided to make my own. This means that there may very well be a better way to do this. I wouldn't doubt it for a second. Also, there may be better flags to include in the rsync script. If anyone knows anything better to add, please do so as I think this is a somewhat important topic to understand. To everyone else, I hope this how to is helpful in making your backup automations a breeze


--
Pat

[JonRohan]

Just what im after. Thanks

[claypole]

Thanks, this is exactly what I needed too!

[carlossousa]

hello,

I noticed your nice rsync how-to but I wonder if you could help me out with something. I have 3 servers, pc1, pc2 and pc3.
pc1 has the data, pc2 has a rsync of pc1 done at 11pm every night and I wanted pc3 to only have a "update" of what happens to be different from pc1 and pc2 every night.
What rsync option do I have to activate on pc3 to allow me to store, on a directory name basis of the type "home[dayN-monthM-yearO]" so I would have the likes of a "incremental" daily backup of pc1?

Can you help?
Thanks,

Carlos Sousa

[plusbryan]

My remote server is running on another port, and ssh-copy-id uses a somewhat unintuitive format for specifying ports:

# ssh-copy-id -i ~/.ssh/id_rsa.pub "-p 222 root@server"

[novakyu]

BTW, I just wanted to say that this setup is potentially very insecure, especially if you created a passwordless private key.

With a passwordless private key, if anyone ever gets a hold of the private key, he will have complete root access to your other server as well.

There's no way to make a passwordless backup completely secure, but at the very least, you should limit the key on the remote server end to allow connection from only the IPs that need to use the key and limit it to only the command it needs to use (rsync). Check the ssh manpage for authorized_keys for more information.

[CodeAlias]

Great post,

I would like to point out to this howto :

Rsync configuration for password-less mirroring over ssh

that also describes server side setup.

wrt security issues, I agree with novakyu. An alternative and a bit more secure way to perform password-less rsync would be through kerberized ssh connections.
Instead of using public keys, kerberos allows the rsync client to use renewable tickets to authenticate its self to the rsync server. If an attacker steals the ticket, he will be able to use it only for a short period of time (depending on your configuration), also he will not be able to use it from another machine.

[dbrine]

great guide but I'm getting error (password prompts actually). I followed the guide and created keys on the local and copied the pub key to the server. this is the error I get. Where am I going wrong???

dbrine@VM-UServer2:~/.ssh$ sudo ssh-copy-id -i ~/.ssh/id_rsa.pub dbrine@192.168.1.107
dbrine@192.168.1.107's password:
Now try logging into the machine, with "ssh 'dbrine@192.168.1.107'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

dbrine@VM-UServer2:~/.ssh$ ssh dbrine@192.168.1.107
Enter passphrase for key '/home/dbrine/.ssh/id_rsa':
dbrine@192.168.1.107's password:

[Antronin]

For me it seems, the private key file has to be named .ssh/identity instead of .ssh/id_rsa to work on Ubuntu.

[baddnady23]

I have actually recently been tinkering with rsync and cron to back up to a remote computer on the local network and have gotten it to successfully run at 2:30 AM every night. Just some other stuff to throw out there for others:

I am using the

--delete

option, which is nice depending on what your trying to accomplish. It will automatically delete anything in the "backup directory" that has been deleted in the directory that your backing up. Some users may like this, others may not.

Also, an FYI to others, for some reason I had to use the

--exclude'/.gvfs/'

option when backing up /home. Otherwise rsync seemed to be caught in an endless loop.

I also like having the output saved into a log file in designated directory so that if I want at a later time, I can review what was actually copied or removed.

My actual command which I have saved to a .sh script is:

rsync -av --delete --exclude='/.gvfs/' --log-file=/home/andrew/Logs/rsync/$(date +%Y%m%d)_rsync.log /home/andrew/ andrew@192.168.1.70:/media/data/backups/ubuntu_home_backup/home_laptop

You'll notice the --log-file argument. It will output it to a file named today'sdate_rsync.log where the today'sdate is in the format 20100311 for easy cataloging.

I also don't have to specify it to use ssh -e because, if I am not mistaken, by telling rsync to copy to a remote location, it defaults to ssh with port 22. I may be wrong but this is what works for me.

Also be aware that rsync pays particular attention to the / at the end of a path. If it is there, it will copy only the contents of that directory into the desired location. If it is not there, it will copy the the actual directory and its contents. For example:

rsync -av /home/andrew/ andrew@192.168.1.70:/media/data/backups/ubuntu_home_backup/home_laptop

copies the contents of my /home to the desired location. Whereas

rsync -av /home/andrew andrew@192.168.1.70:/media/data/backups/ubuntu_home_backup/home_laptop

copies the actual /andrew folder and its contents to the desired location.

To quickly show my crontab file, I would enter

crontab -e

which opens my crontab file, looking something like this:

# m h  dom mon dow   command

.
Explaing the use of CRON is beyond the scope of this thread but for those who are curious about mine, it is:

# m h  dom mon dow   command
30 02 * * * /home/andrew/Scripts/backup.sh

which allows me to run the backup script above at 2:30 AM everyday when I am soundly sleeping and the computer isn't doing anything else.

Great tutorial and I hope my two cents will help someone else down the road in return for all the help that I have gotten over the years!