Page 2 of 4 FirstFirst 1234 LastLast
Results 11 to 20 of 38

Thread: RLBackup (Rsync Local Backup). An easy way to backup your system

  1. #11
    Join Date
    Dec 2006
    Beans
    717
    Distro
    Xubuntu 13.04 Raring Ringtail

    Arrow Re: RLBackup (Rsync Local Backup). An easy way to backup your system

    Quote Originally Posted by scooter28 View Post
    Old thread I know, but I was wondering how you restore using this backup script. i.e. if I back up from / but only need to restore my boot partition or /home, how would I do that? I probably should have asked before I used the script and set up the cron job, but I figured better late then never.
    Well the quick answer is any way you like, as they are just files that can be copied as normal.

    Or you can have a look at my restoration plan below...
    __________________

    My plan if I need to copy more than a few files is:

    1. Install Ubuntu via LiveCD (So I am immediately back in business).
    2. Copy /home (bits of etc and anything else I need) from my backup USB Drive.
    3. Reinstall all my old applications with one click. Install Packages from a list

    For 2. I will probably just use the RLB script with altered source and destinations info (and no link destination).

    __________________

    Other ideas...

    This sounds good as well, but I haven't tried it yet.

    Quote Originally Posted by paulisdead View Post
    When you do the restore, the /proc and /sys folders need to exist, but don't need anything in them. They're not actually folders on the disk, but a representation of the running kernel in memory.

    If I've got a full backup including the binaries, or just want to transfer an existing install to another drive, I wouldn't do a full reinstall. In the past, I took a livecd, partition the drive, mount all the partitions into the running livecd's os to somewhere like /mnt/ubuntu. Then I copy all the files back into place. It's very important to make sure your backups preserve permissions, and when you restore that you keep the original permissions. A 'cp -Rp' will do it (note the capital R, so it picks up hidden files/folders), as will 'rsync -ar' work. Once the files are copied in place, you need to fix up the fstab. Use blkid to determine the new UUIDs of the partitions, and then put them into the new fstab. You then need to mount /proc and /dev from the livecd's folders into the copied folder for the next step. You can do it like this 'mount -o bind /proc /mnt/ubuntu/proc'. The final step is to reinstall the bootloader by cd'ing into the new install and chroot into it, 'chroot /mnt/ubuntu /bin/bash' then a grub-setup && grub-install should do it, and you can exit and unmount everything and reboot.
    __________________

    A Note about copying and permissions:

    Quote Originally Posted by nilleso
    cp -p and/or scp -p will probably work for you

    ...from cp man page
    -p Preserve. cp duplicates not only the contents of
    source_file, but also preserves the owner and group
    id, permission modes, modification and access time,
    ACLs, and extended attributes, if applicable. Notice
    that the command may fail if ACLs are copied to a file
    system without appropriate support. The command will
    not fail if unable to preserve extended attributes,
    modification and access time, or permission modes. If
    unable to preserve owner and group id, cp will not
    fail, and it will clear S_ISUID and S_ISGID bits in
    the target. cp will print a diagnostic message to
    stderr and return a non-zero exit status if unable to
    clear these bits.

    In order to preserve the owner and group id, permis-
    sion modes, and modification and access times, users
    must have the appropriate file access permissions.
    This includes being superuser or the same owner id as
    the destination file.
    http://www.linuxquestions.org/questi...ssions-480735/
    Last edited by SilverWave; November 19th, 2010 at 11:58 PM. Reason: My Plan...

  2. #12
    Join Date
    Sep 2010
    Beans
    37

    Re: RLBackup (Rsync Local Backup). An easy way to backup your system

    My last backup took an hour and 50 minutes, transferring 2476 files, total 160MB. Then it took an additional 51 minutes to check the disk usage. du reports 206M, so I assume 46M is in the directory tree.
    Code:
    $ sudo du -shc $BNAME $BNAME
    5.0G    back-2010-11-19_02-05-01
    206M    back-2010-11-19_02-05-01
    5.2G    total
    Here are some typical backups from the last few days:
    Code:
    Date     # Files   MB Transferred     Time(hr:min)  du -shc
    11-10      167       239               0:33           0:14
    11-14       33        50               0:56           0:31
    11-16      451       220               1:38           0:51
    11-19     2476       160               1:50           0:51
    The times seem to be un-related to the number and size of the files transferred. Perhaps the times are increasing as the backup files get more spread out on the destination disk.

    Question: Where can I find an explanation of what "du" is doing when the same name is repeated, as above? I don't see any discussion of this in the man page.

  3. #13
    Join Date
    Dec 2006
    Beans
    717
    Distro
    Xubuntu 13.04 Raring Ringtail

    Arrow Re: RLBackup (Rsync Local Backup). An easy way to backup your system

    Quote Originally Posted by david.macquigg View Post
    My last backup took an hour and 50 minutes, transferring 2476 files, total 160MB. Then it took an additional 51 minutes to check the disk usage. du reports 206M, so I assume 46M is in the directory tree.
    [...]
    The times seem to be un-related to the number and size of the files transferred. Perhaps the times are increasing as the backup files get more spread out on the destination disk.
    __________________

    Answer: I think you have a problem related to your drive

    e.g. its very slow for some reason.

    __________________

    My backup time was 6 mins for 946,163 files (checked) 48,703 new/changed files (Copied) (Size of copy 3.10Gb).
    Note that I am using a hard disk for storing the backup (not a USB disk).

    __________________

    Code:
    /backup/dr/Backups
    /backup/dr/Backups
    
    rsync -ahvv --log-file=/backup/dr/Backups/back-2010-11-19_18-49-26-LOGFILE.txt --link-dest=/backup/dr/Backups/current-0 --exclude-from=/backup/dr/myscripts/rlbackup1exclude.txt --stats / /backup/dr/Backups/back-2010-11-19_18-49-26
    
    LINK: -File: `/backup/dr/Backups/current-0' -> `back-2010-09-26_12-26-33'
    EXTRACT0: -/backup/dr/Backups/back-2010-09-26_12-26-33
    LINK: -File: `/backup/dr/Backups/current-1' -> `/backup/dr/Backups/back-2009-11-06_22-10-06'
    EXTRACT1: -/backup/dr/Backups/back-2009-11-06_22-10-06
    LINK: -File: `/backup/dr/Backups/current-2' -> `/backup/dr/Backups/back-2009-10-27_17-59-45'
    EXTRACT2: -/backup/dr/Backups/back-2009-10-27_17-59-45
    ============================================================
    
    Free Space on /backup/dr/Backups
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sdb1             459G  396G   40G  91% /backup
    
    ============================================================
    
    Backup started:
    
    Fri, 19 Nov 2010 18:49:26 +0000
    
    Backup finished:
    
    Fri, 19 Nov 2010 18:55:32 +0000
    
    ============================================================
    
    Free Space on /backup/dr/Backups
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sdb1             459G  399G   37G  92% /backup
    
    ============================================================
    
    Statistics
    
    Number of files: 946163
    Number of files transferred: 48703
    Total file size: 26.20G bytes
    Total transferred file size: 3.10G bytes
    Literal data: 3.10G bytes
    Matched data: 0 bytes
    File list size: 22.07M
    File list generation time: 0.001 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 3.12G
    Total bytes received: 1.04M
    
    sent 3.12G bytes  received 1.04M bytes  8.52M bytes/sec
    total size is 26.20G  speedup is 8.39
    Code:
    ============================================================
    
    Fri, 19 Nov 2010 18:55:37 +0000
    
    Actual usage for the last 4 backups:
    37G    /backup/dr/Backups/back-2009-10-27_17-59-45
    947M    /backup/dr/Backups/back-2009-10-27_17-59-45
    3.8G    /backup/dr/Backups/back-2009-11-06_22-10-06
    15G    /backup/dr/Backups/back-2010-09-26_12-26-33
    3.3G    /backup/dr/Backups/back-2010-11-19_18-49-26
    60G    total
    
    Fri, 19 Nov 2010 18:58:06 +0000
    
    ============================================================
    
    Done.
    3.3 -3.1 = 200M overhead for the hard links.
    Last edited by SilverWave; November 20th, 2010 at 09:51 AM. Reason: note re overhead for the hard links

  4. #14
    Join Date
    Dec 2006
    Beans
    717
    Distro
    Xubuntu 13.04 Raring Ringtail

    Arrow Re: RLBackup (Rsync Local Backup). An easy way to backup your system

    Quote Originally Posted by david.macquigg View Post
    Question: Where can I find an explanation of what "du" is doing when the same name is repeated, as above? I don't see any discussion of this in the man page.
    TBH I don't think I ever did see an explanation for this

    I saw in testing RLB that adding the folder name twice, to the argument list sent to du, gave me 2 output sizes so I added it to to report

    I think of it as being related to the directory having both a total disk usage including hard-linked files and a total disk usage excluding hard-linked files...

    __________________

    BTW I think you have a problem related to your drive

    e.g. its very slow for some reason.
    Last edited by SilverWave; November 20th, 2010 at 09:53 AM.

  5. #15
    Join Date
    Sep 2010
    Beans
    37

    Re: RLBackup (Rsync Local Backup). An easy way to backup your system

    Note: Skip this post. I think I found the problem. My backup partition is of type NTFS. I tried an Ext4 partition, and the original script seems to work. I'll do some more experiments and confirm later.

    My backup time was 6 mins for 946,163 files (checked) 48,703 new/changed files (Copied) (Size of copy 3.10Gb).
    5 MB/sec average throughput. That's more like what I would expect. I switched back to your original script for another try, but ran into the same problem with an inability to create hard links on the destination disk. It works only if the destination is on the same disk (/dev/sda1). If I make the destination my second internal disk (/dev/sdb1), not only are there no hardlinks, but the permissions and ownership are changed.

    Here are some typical files and directories in the backup copy on sda1. The permissions and ownership are preserved, and the number of hardlinks is correct, since there were two backups prior to this listing.

    Code:
    /tmp/testback/current-0/david/ASTR501$ ll
    ...
    drwx------  2 david david 4096 2010-10-21 04:14 images/
    -rwxr-xr-x  2 david david 4096 2010-09-23 17:52 ._images*
    -rw-r--r--  2 david david 1160 2010-11-01 18:00 ipython_notes.txt
    drwxr-xr-x  2 david david 4096 2010-11-01 17:58 Matplotlib/
    ...
    And here are the same files after changing only the destination.

    Code:
    /media/sdb/RLBackups/test/current-0/david/ASTR501$ ll
    ...
    drwxrwxrwx 1 root root  4096 2010-10-21 04:14 images/
    -rwxrwxrwx 1 root root  4096 2010-09-23 17:52 ._images*
    -rwxrwxrwx 1 root root  1160 2010-11-01 18:00 ipython_notes.txt*
    drwxrwxrwx 1 root root  4096 2010-11-01 17:58 Matplotlib/
    ...
    I just noticed that my modified script (using cp -al) has more serious problems than being slow. It has the same problems as the original in not preserving ownership and permissions. At least the number of links is correct.

    Code:
    /media/sdb/RLBackups/home/david/current-0/ASTR501$ ll
    ...
    drwxrwxrwx  1 root root  4096 2010-10-21 04:14 images/
    -rwxrwxrwx 21 root root  4096 2010-09-23 17:52 ._images*
    -rwxrwxrwx 21 root root  1160 2010-11-01 18:00 ipython_notes.txt*
    drwxrwxrwx  1 root root  4096 2010-11-01 17:58 Matplotlib/
    ...
    What do you see from the ll command in one of your backups?
    Last edited by david.macquigg; November 20th, 2010 at 05:18 PM. Reason: Found the problem!!

  6. #16
    Join Date
    Sep 2010
    Beans
    37

    Re: RLBackup (Rsync Local Backup). An easy way to backup your system

    I think of it as being related to the directory having both a total disk usage including hard-linked files and a total disk usage excluding hard-linked files...
    Interesting observation. However, I don't think du can figure out which directory entries are for original files, and which are links. All it sees is a link pointing to a file, and a field with the number of links to that file in all directories. In other words, adding a hard link in directory B to a file originally saved in directory A, makes it impossible to discover which directory had the first link. All links are equivalent. At least that is my understanding of how hard links work.

    Maybe what is happening here is that when du sees a directory listed twice, instead of just repeating the same information, it totals only files which have a single link. These would be the newly-created files on the last backup copy.

  7. #17
    Join Date
    Sep 2010
    Beans
    37

    Re: RLBackup (Rsync Local Backup). An easy way to backup your system

    It looks like the du command is doing something more sophisticated than just looking at the link count. Here are some tests I just ran on a sequence of four backups using the original script. The file ._images is linked in all four backups, but it gets counted in only one of the targets of the du command, the target where the file is first seen when all targets are processed in the order specified by the du command. The summaries at the end of the RLBlog files now make sense.

    Code:
    david@david-desktop:/media/SDB5/testback$ du -s *
    1217380	back-2010-11-20_09-01-43                   # <--
    1656	back-2010-11-20_09-04-37
    1660	back-2010-11-20_09-05-34
    749856	back-2010-11-20_09-22-01
    0	current-0
    0	current-1
    0	current-2
    david@david-desktop:/media/SDB5/testback$ rm back-2010-11-20_09-01-43/david/ASTR501/._images 
    david@david-desktop:/media/SDB5/testback$ du -s *
    1217376	back-2010-11-20_09-01-43
    1660	back-2010-11-20_09-04-37                   # <--
    1660	back-2010-11-20_09-05-34
    749856	back-2010-11-20_09-22-01
    0	current-0
    0	current-1
    0	current-2
    david@david-desktop:/media/SDB5/testback$ rm back-2010-11-20_09-04-37/david/ASTR501/._images 
    david@david-desktop:/media/SDB5/testback$ du -s *
    1217376	back-2010-11-20_09-01-43
    1656	back-2010-11-20_09-04-37
    1664	back-2010-11-20_09-05-34                   # <--
    749856	back-2010-11-20_09-22-01
    0	current-0
    0	current-1
    0	current-2
    david@david-desktop:/media/SDB5/testback$ rm back-2010-11-20_09-05-34/david/ASTR501/._images 
    david@david-desktop:/media/SDB5/testback$ du -s *
    1217376	back-2010-11-20_09-01-43
    1656	back-2010-11-20_09-04-37
    1660	back-2010-11-20_09-05-34
    749860	back-2010-11-20_09-22-01                   # <--
    0	current-0
    0	current-1
    0	current-2
    david@david-desktop:/media/SDB5/testback$ rm back-2010-11-20_09-22-01/david/ASTR501/._images 
    david@david-desktop:/media/SDB5/testback$ du -s *
    1217376	back-2010-11-20_09-01-43
    1656	back-2010-11-20_09-04-37
    1660	back-2010-11-20_09-05-34
    749856	back-2010-11-20_09-22-01
    0	current-0
    0	current-1
    0	current-2

  8. #18
    Join Date
    Dec 2006
    Beans
    717
    Distro
    Xubuntu 13.04 Raring Ringtail

    Re: RLBackup (Rsync Local Backup). An easy way to backup your system

    Quote Originally Posted by david.macquigg View Post
    Note: Skip this post. I think I found the problem. My backup partition is of type NTFS. I tried an Ext4 partition, and the original script seems to work. I'll do some more experiments and confirm later.
    Glad you has identified the root cause of the issue

  9. #19
    Join Date
    Dec 2006
    Beans
    717
    Distro
    Xubuntu 13.04 Raring Ringtail

    Arrow Re: RLBackup (Rsync Local Backup). An easy way to backup your system

    Quote Originally Posted by david.macquigg View Post
    [Files are] counted in only one of the targets of the du command, the target where the file is first seen when all targets are processed in the order specified by the du command.
    Good explanation for: du -s *

    But what of the actual RLB code:

    Code:
    # Save to RLBLOG the Total size of 4 backups.
     if [ $PREF_DISKUSAGE -eq 4 ]; then
         print_s "Actual usage for the last 4 backups:"
         du -shc $EXTRACT2 $EXTRACT2 $EXTRACT1 $EXTRACT0 $BACKUPNAME >> $RLBLOG
         print_dat
    So

    $ du -shc $EXTRACT2 $EXTRACT2 $EXTRACT1 $EXTRACT0 $BACKUPNAME

    Or expanded:

    $ du -shc /backup/dr/Backups/back-2009-10-27_17-59-45 /backup/dr/Backups/back-2009-10-27_17-59-45 /backup/dr/Backups/back-2009-11-06_22-10-06 /backup/dr/Backups/back-2010-09-26_12-26-33 /backup/dr/Backups/back-2010-11-19_18-49-26


    Code:
    ============================================================
    
    Fri, 19 Nov 2010 18:55:37 +0000
    
    Actual usage for the last 4 backups:
    37G   /backup/dr/Backups/back-2009-10-27_17-59-45
    947M  /backup/dr/Backups/back-2009-10-27_17-59-45
    3.8G  /backup/dr/Backups/back-2009-11-06_22-10-06
    15G   /backup/dr/Backups/back-2010-09-26_12-26-33
    3.3G  /backup/dr/Backups/back-2010-11-19_18-49-26
    60G   total
    
    Fri, 19 Nov 2010 18:58:06 +0000
    
    ============================================================
    37G and 947M
    I am struggling to find a good way to describe this but how about:
    The 37G could be considered the "old usage" and the 947M the "new usage"

    [update]

    After more testing I think "Total Usage" and "New Usage" are more accurate descriptions.

    So:

    The 37G is the "Total Usage" size, 947M the "New Usage" size.

    This in turn means that the grand total is over by the "New Usage" size.
    (The total Disk Usage is 60G-947M=59.1G)
    Last edited by SilverWave; November 21st, 2010 at 12:56 PM. Reason: old new

  10. #20
    Join Date
    Sep 2010
    Beans
    37

    Re: RLBackup (Rsync Local Backup). An easy way to backup your system

    The 37G could be considered the "old usage" and the 947M the "new usage"
    I agree that's a good brief explanation for regular users. For those of us who like to understand what's going on behind the scenes, it won't help. I'm still puzzling over the details of how these totals are counted. When I run the exact same "du -shc" command (using real names instead of variables from the script) I get results that are off by a few percent, compared to the output of the script. Very strange!!

Page 2 of 4 FirstFirst 1234 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •