Page 4 of 6 FirstFirst ... 23456 LastLast
Results 31 to 40 of 56

Thread: using rsync for backups

  1. #31
    Join Date
    Sep 2010
    Beans
    37

    Re: using rsync for backups

    How do I make a hard link to a directory?
    Code:
    $ ln /home/david
    ln: hard link not allowed for directory

  2. #32
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: using rsync for backups

    I don't think it hard links the directories, since they are directories, but it does it for all the files. Then it rsyncs the files. *shrugs* It's worked fairly well for me so far.

    The script I use is pretty crappy, but it gets the job done:

    Code:
    #!/bin/bash
    
    ###
    ### monthly.sh
    ### Script for monthly document backup
    ### Created by Charles
    ### Tested 10/03/2010
    ### Updated 10/03/2010
    ###
    
    BIN=/bin/
    USR=/usr/bin/
    ARRAYPATH=/array/
    MONTHLYPATH=/monthly/
    HOME=/home/charles/
    ERR=${HOME}logs/monthly_backup_error.log
    DATE=$(${BIN}date +%D)
    FOLDER=$(${BIN}date +%m%d%Y)
    
    if ${BIN}mountpoint -q $ARRAYPATH
    then
      { ${BIN}mount -t ext4 UUID=f00c8f27-7a11-42b0-96f9-0e04a261d682 $MONTHLYPATH; } 2> $ERR || { echo "Mounting of backup drive failed"; exit; }
    else
      { echo "$ARRAY not mounted!"; exit; }
    fi
    
    OLDMONTHLYNAME=$(${BIN}cat ${MONTHLYPATH}latest)
    NEWMONTHLYDIR=$MONTHLYPATH$FOLDER
    OLDMONTHLYDIR=$MONTHLYPATH$OLDMONTHLYNAME
    MONTHLYLOG=${MONTHLYPATH}log/${FOLDER}.txt
    
    # rsync
    if ${BIN}mountpoint -q $MONTHLYPATH
    then
      { ${BIN}cp --archive --link $OLDMONTHLYDIR $NEWMONTHLYDIR && echo $FOLDER > ${MONTHLYPATH}latest && ${USR}rsync --archive --itemize-changes --delete --log-file $MONTHLYLOG $ARRAYPATH --exclude dvds --exclude lost+found $NEWMONTHLYDIR && ${BIN}umount $MONTHLYPATH; } 2> $ERR || { echo "Monthly backup failed."; exit; }
    fi
    # eof
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  3. #33
    Join Date
    Sep 2010
    Beans
    37

    Re: using rsync for backups

    OK, there are no links to directories, therefore, every backup must include a copy of every directory in the tree that is backed up. Try backing up a large tree like my earlier example, after changing just one small file. Run df to see how much disk is actually used before and after the backup. I think you will get the same results as I got.

    rsync is just not able to do "Time Machine" style backups, given the limitations of hard links to directories. I've tried this with both ext4 and NTFS filesystems in Ubuntu.

    Note: I can make hard links to directories in Mac OSX, so this may be a fundamental difference between these systems.

  4. #34
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: using rsync for backups

    I suppose it would add up after a while, but check this:

    Code:
    charles@thor:~$ du -h --max-depth=1 /cdrom/
    1.4T    /cdrom/05312010
    15G     /cdrom/07192010
    53G     /cdrom/09042010
    1.6M    /cdrom/08182010
    1.6M    /cdrom/07152010
    1.6M    /cdrom/09182010
    1.6M    /cdrom/09142010
    1.6M    /cdrom/05252010
    1.6M    /cdrom/10182010
    1.6M    /cdrom/06072010
    1.6M    /cdrom/06162010
    1.6M    /cdrom/07052010
    So, yeah it takes some space, but not all that much if the files haven't changed.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  5. #35
    Join Date
    Aug 2008
    Location
    Victoria, BC Canada
    Beans
    1,594

    Re: using rsync for backups

    I use tar and gzip to make backups. The rsync tool is not as effective in my opinion for backup.

    I see rsync as more of a replication tool for server roll over.

    What I do is use a sump of MySQL and then tar up everything else I want to backup.

    Now I have a DVD burner to write too, or I could cp them to another machine. If you have a take drive, that is another option.

    tar is a better backup tool compared to other solutions I have seen.

    I have looked in this area a lot. And every alternative has fell short compared to the way I do it now.
    SERVER: Azure datacenters, Hyper-V

  6. #36
    Join Date
    Sep 2010
    Beans
    37

    Re: using rsync for backups

    Vegan, gzipped tar files are really inconvenient for browsing and recovering a single file from an old backup. rsync was designed as an alternative. Each of us has different expectations, but for my small office, the features of the Time Machine are really ideal. I just need to figure out how to make rsync do that more efficiently.

    Charles, your cp --archive --link command is indeed making hard links to subdirectories in my destination directory!! This is surprising since not even ln will do that. This also tells me that the prohibition against hard linking to directories is not a limitation of Linux or of the file system, but something specific to rysnc and ln. I'll have to experiment with this and get back to you.
    Last edited by david.macquigg; November 10th, 2010 at 08:36 PM. Reason: expand on linking

  7. #37
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: using rsync for backups

    Glad you are able to figure it out.

    I don't use tar because it would be a major pain in the butt to grab a specific file from within the tar archive without using something like 7zip to open the archive.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  8. #38
    Join Date
    Sep 2010
    Beans
    37

    Re: using rsync for backups

    I've got some results comparing Charles' method to that in RLBackup. Using cp -al to set up the link tree *before* the rsync command is orders of magnitude faster and smaller than using the --link-dest option in rsync.

    In both tests, I backed up my home directory (2125 files, 1.34GB). In both tests, I started with a backup already in place, and did an incremental backup to a new directory. 26 files were updated, with a total data transfer of 90MB.

    For TestA I ran RLBackup with no modification other than setting the directories for source, destination, and exclude files.
    http://ubuntuforums.org/showthread.p...t=rsync+backup

    For Test B I left out the --link-dest option on the rsync command, and inserted a cp -al command just ahead of rsync. The hacked script then has these lines:
    Code:
    LASTBAK=back-2010-11-09_11-32-35       ### need to automate this
    cp -al $DESTPATH/$LASTBAK $BACKUPNAME     ### make links with a local copy
    rsync $OPT $SOURCE $BACKUPNAME > $OUTPUTLOG   # no link-dest in $OPT
    Code:
                            Test A     Test B
     Time (seconds):            59          3
     Disk usage (MB):         1300         87

  9. #39
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: using rsync for backups

    Wow. Thanks for posting the results. I didn't know it would make that much of a difference.

    You need to automate creating the folder for the new backup, right?

    My script does that based on the output of date, so maybe that'll be of some use to you.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  10. #40
    Join Date
    Feb 2007
    Location
    Seattle, WA
    Beans
    146
    Distro
    Ubuntu 10.10 Maverick Meerkat

    Re: using rsync for backups

    Quote Originally Posted by david.macquigg View Post
    OK, there are no links to directories, therefore, every backup must include a copy of every directory in the tree that is backed up. Try backing up a large tree like my earlier example, after changing just one small file. Run df to see how much disk is actually used before and after the backup. I think you will get the same results as I got.

    rsync is just not able to do "Time Machine" style backups, given the limitations of hard links to directories. I've tried this with both ext4 and NTFS filesystems in Ubuntu.

    Note: I can make hard links to directories in Mac OSX, so this may be a fundamental difference between these systems.
    I would recommend trying out backintime if you want a comparable solution to the mac's time machine. I use it for backing up my home directory while at work. It requires a gui (i.e. gnome or kde). It uses rsync, diff and hard links to do exactly what it sounds like you want to do. It might not work for exactly what you need, but I wanted to throw it out there as no one else had.

Page 4 of 6 FirstFirst ... 23456 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •