using rsync for backups

[djanie78]

Wise men. Very helpful people

[djanie78]

I can boot an rsync'ed root partition in a VM in minutes. What exactly is horrible about that?

rsync over to a new directory every time and have fslint dedup with hard links. throw in a copy of sfdisk's output and you have what you need to put it back down.
(BackupPC would work better though)

This sounds interesting. Point us in the right direction please. Looks like something i will definately look at as a solution to other stuff

[LightningCrash]

This sounds interesting. Point us in the right direction please. Looks like something i will definately look at as a solution to other stuff

I would start with something like LINUXexplo
http://www.unix-consultants.co.uk/ex...inux-explorer/

Have it run in cron and copy to NFS or email the results somewhere. The script needs to be modified to exclude process trees in /proc, though.

Then I would just run BackupPC.

To restore you will need the explorer output

1. lvcreate a volume the same size as the original
2. use your sfdisk output to partition it
3. use kpartx to map the partitions
4. mkfs, mkswap on the partitions
5. mount the partitions in /mnt/restore/ as needed
6. have BackupPC lay the filesystem back down relative to /mnt/restore
7. pull the UUIDs of the kpartx partitions, then go into /mnt/restore/etc/ and modify the fstab to reflect the new UUIDs. (or if you're using xen you can use xvda1, xvdb1, etc)
8. check the networking information in the networking directories
9. umount the kpartx mapped partitions
10. use kpartx to unmap the partitions
11. make a xen config file or make a vmdk file for virtualbox,vmware
12. boot the VM

I have scripts that do most of this on my behalf. It's actually very similar to how I provision VMs when I don't use virt-install

[LightningCrash]

I will add that one advantage to using image files (like dd files or qcow2, etc) in production over direct LVM assignment is that you don't have to screw with rsync and kpartx and sfdisk, etc. You just copy the image file and you're pretty good to go.
You get a lot of duplicated blocks that way, though.

[paulisdead]

This is the script I use for home backups, I have this machine and a couple others back up to /var/backup/ on my home file server for quick access. Once it's done with that, it mounts a USB drive, and then dumps /var over to the USB drive, since I want some other stuff in /var dumped over to it. Since I don't want to backup all the binaries, just the configs, I dump out the list of installed packages, so I can easily feed that list back into apt-get and have it reinstall everything, and since I have /etc backed up, I can then easily restore the configs for anything I need.

#!/bin/bash
/usr/bin/dpkg --get-selections > /etc/installed-software
/usr/bin/rsync -avrt --exclude 'mtab' /etc /var/backups/hal9000/
/bin/mount /mnt/backup
/usr/bin/rsync -avrt --exclude 'cache/apt' --exclude 'cache/apt-cacher-ng/' --exclude 'cache/man'  /var/ /mnt/backup/var/
/usr/bin/rsync -avrt --exclude 'smb/videos/' --exclude 'smb/downloads' --exclude 'smb/backup/Linux/' --exclude 'smb/backup/images/' /home/ /mnt/backup/
/bin/umount /mnt/backup

When I lost the hard disk in my home DNS/SSH/DHCP server, it took me just over an hour to replace the disk, reinstall debian (I use apt-cacher-ng as an apt proxy, so I had a copy of all packages sitting on my server), import the package list, bring over the config files, and get it up and running again.

Unless you use the --delete flag with rsync, it shouldn't be deleting files in the destination folder that no longer exist in the source folder.

If anyone is looking for a real managed backup solution, that can fully backup windows and linux clients, plus the OS files, I recommend bacula. I use it at work to backup our servers, but it's way overkill for a few home machines.

[SeijiSensei]

I think the idea is that rsync can protect you from hardware failure, since you can restore to your most recent rsync, however if you delete a file and run rsync you still don't have a backup of the file.

Unless you use the one of the various --delete options, rsync won't remove files from the backup directory just because they don't exist on the source.

Sorry, I just noticed that paulisdead made this point as well.

[amac777]

Unless you use the one of the various --delete options, rsync won't remove files from the backup directory just because they don't exist on the source.

What about if you inadvertently change a file (or it gets corrupted/malware etc) and you need to go back to an earlier version?

"rsnapshot" is another good option that will efficiently keep snapshots using rsync and hardlinks so you can recover older versions of files.

[djanie78]

This is the script I use for home backups, I have this machine and a couple others back up to /var/backup/ on my home file server for quick access. Once it's done with that, it mounts a USB drive, and then dumps /var over to the USB drive, since I want some other stuff in /var dumped over to it. Since I don't want to backup all the binaries, just the configs, I dump out the list of installed packages, so I can easily feed that list back into apt-get and have it reinstall everything, and since I have /etc backed up, I can then easily restore the configs for anything I need.

#!/bin/bash
/usr/bin/dpkg --get-selections > /etc/installed-software
/usr/bin/rsync -avrt --exclude 'mtab' /etc /var/backups/hal9000/
/bin/mount /mnt/backup
/usr/bin/rsync -avrt --exclude 'cache/apt' --exclude 'cache/apt-cacher-ng/' --exclude 'cache/man'  /var/ /mnt/backup/var/
/usr/bin/rsync -avrt --exclude 'smb/videos/' --exclude 'smb/downloads' --exclude 'smb/backup/Linux/' --exclude 'smb/backup/images/' /home/ /mnt/backup/
/bin/umount /mnt/backup

When I lost the hard disk in my home DNS/SSH/DHCP server, it took me just over an hour to replace the disk, reinstall debian (I use apt-cacher-ng as an apt proxy, so I had a copy of all packages sitting on my server), import the package list, bring over the config files, and get it up and running again.

Unless you use the --delete flag with rsync, it shouldn't be deleting files in the destination folder that no longer exist in the source folder.

If anyone is looking for a real managed backup solution, that can fully backup windows and linux clients, plus the OS files, I recommend bacula. I use it at work to backup our servers, but it's way overkill for a few home machines.

Damn there as some clever people in this joint. You all make the world go round

[david.macquigg]

rsync sure seems like the best way to do backups for anyone who can write a script, or even just modify one that somebody else has written. Certainly no more difficult than figuring out all the settings in a black box like Acronis, and then you have a script you understand thoroughly.

I've got a Python script running as a cron job, keeping a fully-synchronized copy of all my program and data files on an external disk. This script runs every night, and it also makes copies so I can keep snapshots of the system 1, 2, 4, and 8 days ago. Everything is now backed up except the /proc and /sys directories. I'm assuming those can be left as-is when I restore all the old files to a new hard disk with a system built from my original Ubuntu CD. Is that a safe assumption? Is there a better way to do this?

Here is the core of my script. /media is the external drive.

src = '/'
excludes = "--exclude '/media' --exclude '/proc' --exclude '/sys'"
command = "rsync -a --delete %s %s %s"  %  (excludes, src, dest)

[paulisdead]

When you do the restore, the /proc and /sys folders need to exist, but don't need anything in them. They're not actually folders on the disk, but a representation of the running kernel in memory.

If I've got a full backup including the binaries, or just want to transfer an existing install to another drive, I wouldn't do a full reinstall. In the past, I took a livecd, partition the drive, mount all the partitions into the running livecd's os to somewhere like /mnt/ubuntu. Then I copy all the files back into place. It's very important to make sure your backups preserve permissions, and when you restore that you keep the original permissions. A 'cp -Rp' will do it (note the capital R, so it picks up hidden files/folders), as will 'rsync -ar' work. Once the files are copied in place, you need to fix up the fstab. Use blkid to determine the new UUIDs of the partitions, and then put them into the new fstab. You then need to mount /proc and /dev from the livecd's folders into the copied folder for the next step. You can do it like this 'mount -o bind /proc /mnt/ubuntu/proc'. The final step is to reinstall the bootloader by cd'ing into the new install and chroot into it, 'chroot /mnt/ubuntu /bin/bash' then a grub-setup && grub-install should do it, and you can exit and unmount everything and reboot.