Page 1 of 5 123 ... LastLast
Results 1 to 10 of 43

Thread: HOWTO: Backup nightly via rsync

  1. #1
    Join Date
    Mar 2006
    Location
    Canada
    Beans
    1,313
    Distro
    Ubuntu 10.10 Maverick Meerkat

    HOWTO: Backup nightly via rsync

    Preamble
    As a Linux System Administrator, I've had to keep two servers in synchronization, either nightly or every couple of hours. The term is 'mirror' and can be seen in use when downloading Ubuntu .iso files from localized servers, say from a Cambodian server when the main server is in Britain. (You may not know this, but that's how it works, using pretty much the same script you see below).

    Why should I use this script?
    You don't have to be a Linux System Administrator to enjoy the beautiful interoperability benefits of Linux, you can put it to use right at home!

    Use this script to backup your entire family's data, every night, by using a central server and making the server 'pull' everyone's documents to it.

    How do I use this script?
    How? Well if you purchase a 500GB server, then you could potentially backup five family member's computers if they each had a 100GB hard drive.

    Simply use this script with the 'install' argument five times on the server and it will create five keys, one for each of your family members.

    You need to setup either DNS names for the computers or static IP addresses for each computer.

    I won't go into great detail on how to do this in every situation, but for a home network, you can change the IP address of one of your computers to a static IP address like this:
    Code:
    gksu gedit /etc/network/interfaces
    Before:
    # This file describes the network interfaces available on your system
    # and how to activate them. For more information, see interfaces(5).

    # The loopback network interface
    auto lo
    iface lo inet loopback

    # The primary network interface
    auto eth0
    iface eth0 inet dhcp
    After:
    # This file describes the network interfaces available on your system
    # and how to activate them. For more information, see interfaces(5).

    # The loopback network interface
    auto lo
    iface lo inet loopback

    # The primary network interface
    auto eth0
    #iface eth0 inet dhcp

    iface eth0 inet static
    address 192.168.1.10 # computer1 = 192.168.1.10, computer2 = 192.168.1.11, computer3 = 192.168.1.12, etc.
    netmask 255.255.255.0
    gateway 192.168.1.1
    As for the Linux System Administrators, I don't need to tell you how useful this script is

    Any alternatives?
    Of course! I'm not the first person to need file-synchronization or mirroring, but I feel most comfortable writing my own scripts so I know what's going on and can control the encryption strength.

    GUI (Graphical User Interface)
    unison: a file synchronization program used to sync files between two directories, either on one computer, or between a computer and another storage device. (I've never used it, but I heard its reliable)
    Code:
    sudo aptitude install unison unison-gtk
    CLI (Command Line)
    rsnapshot: a filesystem snapshot utility for making backups of local and remote systems. The disk space required is just a little more than the space of one full backup, plus incrementals. (My program does not use incrementals).
    Code:
    sudo aptitude install rsnapshot
    Vocabulary
    • ssh: a network protocol that allows data to be exchanged using a secure channel (encryption) between two computers
    • rsync: a software application for Linux which synchronizes files and directories from one location to another using minimal bandwidth (only transfers files (or parts of files) that don't exist)
    • mirror: an exact copy of a data set
    • push: to send (or give) data from one computer to another
    • pull: to receive (or ask) for data from one computer to another


    Let's get started
    It has several flags and options, but some are hard-coded, so you may have to edit the script by hand.

    Options
    General usage:
    ./rsync.sh ['uninstall' | 'install' | 'run'] ['push' | 'pull'] [local_dir] [remote_user] [remote_host] [remote_dir] [remote_ssh_port]
    Note: '[remote_ssh_port]' is usually 22 unless you change it from the default. If you don't know how to change the ssh port, its more than likely 22.

    install
    Use this option if you want to setup an automated, nightly backup between one computer and another.
    Arguments:
    ./rsync.sh install [push | pull] [local_dir] [remote_user] [remote_host] [remote_dir] [remote_ssh_port]
    Example:
    ./rsync.sh install pull /home/brett/ brett 192.168.1.2 /home/brett 2222
    • Creates the following directories: $HOME/bin, $HOME/cron, $HOME/logs
    • Creates 4096-bit RSA encryption key (illegal in the USA I think)
    • Installs file to directories above


    uninstall
    Use this option if you want to remove a previously installed setup.
    Arguments:
    ./rsync.sh uninstall [remote_user] [remote_host]
    Example:
    ./rsync.sh uninstall brett 192.168.1.2
    • Removes cron-job
    • Removes RSA key


    run
    Use this option if you want to run the program without installing anything. Good to use as a test and on a single-needs basis.
    Arguments:
    ./rsync.sh run [push | pull] [local_dir] [remote_user] [remote_host] [remote_dir] [remote_ssh_port]
    Example:
    ./rsync.sh run pull /home/brett/ brett 192.168.1.2 /home/brett 2222
    • Runs rsync with your parameters


    rsync.sh
    http://github.com/brettalton/rsync-o...aster/rsync.sh
    Last edited by altonbr; September 10th, 2010 at 05:01 PM. Reason: reactive --log-file=$LOCAL_LOG_FILE for Ubuntu users

  2. #2
    Join Date
    Apr 2006
    Location
    Michigan
    Beans
    55
    Distro
    Ubuntu 8.10 Intrepid Ibex

    Re: HOWTO: Backup nightly via rsync

    This was very useful!
    I didn't actually use your script in conjunction with cron, but it let me come up with my own backup scheme. Thanks

  3. #3
    Join Date
    Mar 2006
    Location
    Canada
    Beans
    1,313
    Distro
    Ubuntu 10.10 Maverick Meerkat

    Re: HOWTO: Backup nightly via rsync

    Quote Originally Posted by grim4593 View Post
    This was very useful!
    I didn't actually use your script in conjunction with cron, but it let me come up with my own backup scheme. Thanks
    Great! I was worried that this tutorial might be to complex, when really it's only a couple steps.

  4. #4
    Join Date
    Apr 2006
    Beans
    Hidden!

    Re: HOWTO: Backup nightly via rsync

    I haven't tried this yet (just trying to figure out what's happening in your script) so I could be wrong, but don't you need the $BACKUPDIR variabe in your rsync-line?

    It looks like you only define a destination and no source directory.

  5. #5
    Join Date
    Mar 2006
    Location
    Canada
    Beans
    1,313
    Distro
    Ubuntu 10.10 Maverick Meerkat

    Re: HOWTO: Backup nightly via rsync

    Quote Originally Posted by madk View Post
    I haven't tried this yet (just trying to figure out what's happening in your script) so I could be wrong, but don't you need the $BACKUPDIR variabe in your rsync-line?

    It looks like you only define a destination and no source directory.
    Brutal. You're correct. Thank you and my apologizes.
    Code:
    rsync -avrz --delete --delete-excluded --exclude-from=$EXCLUDEFILE --log-file=$LOGFILE --rsh="ssh -p $SSHPORT -i /home/$USER/.ssh/backup" $BACKUPDIR $USER@$IPADDRESS:/home/$USER/backup/

  6. #6
    Join Date
    Jan 2007
    Beans
    23

    Re: HOWTO: Backup nightly via rsync

    The ssh-copy-id command is useful for copying the public key.

    ssh-copy-id -i <path to key> user@host

  7. #7
    Join Date
    Mar 2006
    Location
    Canada
    Beans
    1,313
    Distro
    Ubuntu 10.10 Maverick Meerkat

    Re: HOWTO: Backup nightly via rsync

    Yeah, but I needed to do logic checks and create folders as well.

    I'm planning on re-writing this soon actually as it seems a bit scattered... It could really be made into one script.
    Last edited by altonbr; May 18th, 2008 at 11:10 PM.

  8. #8
    Join Date
    Aug 2007
    Location
    Olavarria, Argentina
    Beans
    208
    Distro
    Ubuntu 11.10 Oneiric Ocelot

    Re: HOWTO: Backup nightly via rsync

    Thank you, very usefull!!

  9. #9
    Join Date
    Sep 2005
    Location
    Virginia
    Beans
    23
    Distro
    Ubuntu 9.04 Jaunty Jackalope

    Re: HOWTO: Backup nightly via rsync

    This is a pretty good contribution, but WHY would you recommend a passphrase-less key??????

    I would just create a proper key with passphrase and have a ssh-agent running. This would make it much much much secure/complete solution.

    Passphrase-less Keys ARE THE DEVIL!!!

    Of course I would do that for any communication across the cloud, if your backup is inside your house's LAN then its not that bad (although not the best practice).

    Cheers!

  10. #10
    Join Date
    Aug 2006
    Beans
    841

    Re: HOWTO: Backup nightly via rsync

    Quote Originally Posted by Toky View Post
    This is a pretty good contribution, but WHY would you recommend a passphrase-less key??????

    I would just create a proper key with passphrase and have a ssh-agent running. This would make it much much much secure/complete solution.

    Passphrase-less Keys ARE THE DEVIL!!!

    Of course I would do that for any communication across the cloud, if your backup is inside your house's LAN then its not that bad (although not the best practice).

    Cheers!
    if you need it to be automatic, you cannot have it requiring user input.

    getting that off my chest. why on earth reinvent the wheel? there are dozen scripts already doing this. my favorite due to simplicity being rsnapshot.

    good luck

Page 1 of 5 123 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •