Howto: Backup and restore your system!

[t1m]

Thanks for the guide! (again again again again :rolleyes: )

It would be nice to have a program (similar to "keep (http://kde-apps.org/content/show.php?content=32984)") that would automate the whole thing.
Maby you could give it your password too if it needed it for permission stuff.

[kilou]

When excluding folders from the TAR archive, those needs to be recreated manually afterwards. This becomes a problem when booting from a LiveCD and restoring from there because when I recreate a folder from the LiveCD it doesn't get the correct permission. For instance I excluded /tmp and thus had to recreate it from the LiveCD. I did it with mkdir and then rebooted. The restored system booted correctly but right after login, the system freezed because of a permission error on /tmp. To make the system work I had to pass a 777 permission with chmod on /tmp......but this is clearly not optimal!! Also I have no idea what should be the permissions of other recreated folders.

When doing the TAR archive, I tried to find a way to exclude only the content of some folders but not the folders themselves (leaving them as empty folders). This would exclude their content but would keep their permissions and would of course avoid the need to recreate these folders afterwards. They would just be untarred as empty folders with the correct permissions. However I'm unable to find out how to achieve this. One interesting thing is that the option --exclude=/folder/* removes the content of the folder without destroying the folder itself (because the folder appears as being created in the verbose mode of TAR). However opening the TAR archive still doesn't show that folder so it still doesn't work. There must be something in TAR that automatically removes empty folders from the archive probably.

Has anyone faced the same issues? I'm really wanting to find out how to restore the system using TAR from a LiveCD. Also I've read that the TAR archive must be stored on an EXT3 partition because FAT32 will not store the permissions. Is that right??

[phil90]

Hello

Is there a way for me to know the purposes of folders? People reccomend to exclude lots of folder in the backup like /var /opt /usr etc. But I don't know what most these folder do or contains so I don't really know what to exclude.

thanks

[kilou]

Excluding folders depend on what you want to backup. Some people just want backups of their /home and mabe prefer an image of their system for the system backup. For a system backup personally I wouldn't exclude folders such as usr, var or opt in a backup because those are really important and contain your programs. The one you want to exclude however are /tmp, /proc, /sys and /media or /mnt because you don't want to backup temporary files that would be erased on next reboot anyway and you don't want to backup the content of your CD-Rom or other attached devices in /media. You could also unmount these before but I suppose it is more convenient to exclude them simply. I'm unsure if excluding /var/tmp, /var/cache, /var/log and /var/crash would be a good idea. For now I wouldn't exclude them.

However the problem with excluded folders is that you need to recreate those before rebooting on your restored system. If you do this from a LiveCD, you get wrong permissions on those recreated folders and your will not boot properly (mine was stuck after the login screen because of a wrong permission on /tmp as I had to recreate it fom the LiveCD). This is really a problem and I'd like to be able to only exclude the content of excluded folders but leaving the folders intacts so that they are rebuilt with the correct permissions. I don't know how to do this because even the --exclude=/folder/* command removes folder from the archive where it should only just empty it. If anybody can help regarding that issue, please let us know. The TAR method for backing up a system is REALLY convenient because it let you restore a system either a running system without reboot or from a LiveCD. But the latter solution just creates troubles when folders are excluded and must be recreated from the LiveCD.

[Draaku]

Hi, thanks for the guide, I've used it on the 3 systems I admin. I dont claim to be a ubuntu expert, and am looking for advice on this.

I have a SBS Windows 2003 server with a 500GB USB drive attached to it. It is formatted FAT32 shared and only administrator has permissions. I have created 3 directories one each for my ubuntu file/web servers. I am looking for a way to make this all automated. Here is what I have run but manually. Could someone help me with a script to automate this? I work in a volunteer environment and need to make sure the next person is able to backup the servers without to much headache. Thanks in advance.


sudo su
cd /
tar cvpzf backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/sys /
mount -t cifs -o username=Administrator \\\\192.168.10.20\\Hickory /mnt/hickory/
cp /backup.tgz /mnt/hickory

[phil90]

thanks kilou

[kilou]

So nobody is restoring from a LiveCD?????

[kilou]

up

[xyz]

Yes I also have backuped/restored using a Live CD + downloading PartImage and going from there.
But I don't think this thread is about backingup/restoring using a Live CD.
So what do you mean?

[kilou]

This thread is supposed to be about backing up/restore a system with TAR in all circumstances. The first post especially mentions that if your system is screwed up you'll have to restore it from a LiveCD. When doing it from a LiveCD, the problem is that when you exclude folders from the TAR archive using --exclude=/folder, you'll have to recreate these from the LiveCD and then the permissions on these folders are not correct because they've been created by root@ubuntu (LiveCD) and not by the owner that was supposed to own these folders. If for instance you exclude /tmp from the archive (what you definitely want to do) and you restore the system from the LiveCD and thus recreate /tmp by hand from there, the /tmp doesn't get the right permission and your restored system will freeze right after the login screen! You can ""solve"" this problem by giving a 777 permission to /tmp but this is clearly something that you don't want to do generally. This happen to every folder that you'll need to recreate from the LiveCD. Of course you could reset the correct permissions on these folders but this means that every time you exclude a folder you'll have to know what is its permissions.....not really convenient.

I wanted to have the --exclude=/folder command only exclude the content of a folder but not the actual folder itself so as to preserve its permissions and enables it to be untarred as an empty folder with correct permissions when restoring the system. However this doesn't appear to be possible with either --exclude=/folder/* or --exclude=/folder/*.* :(

If we can find a way to solve this that'd be great because to me this TAR method is the best way to backup/restore a system as it can be done on a running system as well. You can exclude things etc what makes it much more convenient than imaging softwares as well. But of course the major problem is this apparent inability to restore the system from a LiveCD if you excluded folders.....and any backup/restore method MUST be able to handle restoring from a totally screwed up install and thus use a LiveCD to initiate the restore process.

I'm new to Ubuntu and for sure don't know all the possibilities of the TAR command. So if anyone can bring a solution on how to exclude the content of a folder WITHOUT removing the folder itself, please advise.

[mannheim]

kilou, one strategy for restoring from a tar backup is the following (and I think this is what the origina poster meant):


Install Ubuntu onto the new hard disk using a LiveCD
Extract the tar archive on top of the fresh install.


The first step may seem like a pain, but it doesn't take long and hopefully you won't be doing this often. Step 1 creates /tmp and /dev and all such things with correct permissions.

There is a problem with the second step: simply extracting the tar archive will not delete files on the hard disk which are not in the backup. So if the LiveCD installs stuff which you had removed from your system, the restore procedure above will not correctly remove that stuff again during the restore. The way to get around that is to use the --listed-incremental option for tar, during both backup and restore.

I have recently switched to using rsync for backups, because of uncertainties with tar and --listed-incremental. It seems like a good choice for backup to a hard drive. See this post (http://www.ubuntuforums.org/showthread.php?t=345146) for an elegant approach to incremental backups with snapshots using rsync; see also
here. (http://www.mikerubel.org/computers/rsync_snapshots/)

[kilou]

I look into rsync more carefully but I think this is something that must be downloaded from the repository when booting from a LiveCD (I'm unsure if it is not installed by default in the LiveCD though. I don't really like that because I use a laptop and would like to be able to restore a system even if I can't connect to Internet, something that is not possible if I have to download a package to be able to restore a system from a LiveCD.

Also I don't like the idea of having to reinstall Ubuntu from the LiveCD before restoring it with TAR. I understand this is not that long but I do a lot of reinstall because I try a lot of things with Ubuntu and having to reinstall Ubuntu from the LiveCD every time is not really appealing. To be honest I already find it a pain to boot the LiveCD, it takes soo long to reach the desktop :D The best would be to build a custom CD that boots to command line and have the backup tool already builtin, something like SystemRescue CD but with rsync or where it is possible to store a script for a TAR command.

Do you know if you have to recreate excluded folders if you backup with rsync???

[mannheim]

Well, returning to your original question, it seems that tar does include the directory itself if you do "--exclude=blah/*". Eg, here we record the directory "one" in the tar archive, but we not its contents:

user@machine2:~$ mkdir tar-src
user@machine2:~$ mkdir tar-src/one
user@machine2:~$ touch tar-src/one/foo
user@machine2:~$ find tar-src # This is the tree we made
tar-src
tar-src/one
tar-src/one/foo
user@machine2:~$ tar -cf test.tar --exclude=one/* tar-src # Create the archive
user@machine2:~$ rm -r tar-src # Delete the tree
user@machine2:~$ tar -xf test.tar # Restore tree from tar archive
user@machine2:~$ find tar-src # This is the tree we restored
tar-src
tar-src/one


which I think is what you wanted. rsync works similarly.

[mannheim]

Another useful point is to use --one-file-system (or whatever it is) as an option to tar (or use -x as an option to rsync), which stops tar or rsync from recursing into mounted filesystems. This stops recursion into /dev /proc and /sys on live filesystems, because when these are live, they are actually just mountpoints.

The result (e.g. with rsync) is that the backup contains empty /dev /proc and /sys directories. This is what you want, at least with Ubuntu edgy. Using udev, edgy boots okay with an empty /dev directory, at least in my experience.

[kilou]

WOW well mannheim you're great!!! Basically I tried to backup /boot with exclude=/boot/grub/* and just looked into the tar archive and there was no grub folder at all so I assumed the folder was not saved. However I didn't try to untar the archive....and when untarring the grub folder was recreated with correct permissions but empty, exactly what I was looking for!!!! Great, thanks for that one :D

So now the right command to choose when excluding a folder is --exclude=/folder/* instead of --exclude=/folder. This will make the restore possible from a LiveCD without hassle and without having to reinstall Ubuntu first!!

Now what is the difference between using --one-file-system and --exclude=/proc/* --exclude=/dev/* --exclude=/sys/* ? Also what is the purpose of the /dev directory? Is it really safe to exclude it from a full system backup?? What about /var/cache as a whole or /var/cache/apt?

[mannheim]

A comment first: in my example with --exclude, it wasn't a good test, because I should have put quotes around "blah/*". If you do--exclude=blah/* then the shell will expand the wild card *. If you do, more properly, --exclude="blah/*" then the argument "blah/*" is passed to tar, and tar interprets the wildcard. In the example above, I think the result is the same.

With the other things, I think there are earlier comments in this long thread. The linux standard (or something) says that the entire contents of /var/cache are disposable, and app will recreate what they need there. But apps might take a while to do so. /var/cache/apt may contain a great deal of deb files downloaded from the repositories, particularly if you have upgraded your ubuntu system. To save space in backups, and because I have a fast internet connection, I exclude /var/cache/apt.

If /dev is just an empty directory, then, as I said, I think it doesn't matter. At boot time, the kernel mounts a temporary file system at /dev anyway, hiding the actual contents. It fills the temporary file system at /dev with whatever is needed. Maybe someone else knows better than me whether the contents of the /dev on disk are ever needed: I don't think so. (This applies to edgy, and maybe dapper, but not to earlier versions of ubuntu).

If you are using tar or rsync to backup the contents of a partition when it is live, then it makes sense to use --one-file-system. Otherwise, tar will start adding the contents of your mounted removable media and other stuff too to the backup.

[kilou]

Thanks Mannheim for the info. I've been now able to do a backup and a restore from a LiveCD without major issue. However a minor glitch is that my other partitions on the same harddisk are no more automatically mounted. I excluded media from the archive and this is probably related but I do not understand why the other partition is not mounted automatically after restore because it is still mentionned in fstab file.

What I did is to backup the system, reboot on the LiveCD, reformat the Linux partition and restore the system from the archive. Does the formatting destroys something that prevent other partitions on the same harddisk to be automatically remounted??? Interestingly my extrenal USB disk is automatically detected and mounted but not my other Ext3 partition on the laptop harddrive which is referred in fstab....

Also the update manager doesn't appear to work properly after restore. It tells me that updates are available but when I click the orange tray icon, it scans the available udates and then dies :) I also excluded var/cache/apt from the archive, is this related?

Any clue?

[mannheim]

What I did is to backup the system, reboot on the LiveCD, reformat the Linux partition and restore the system from the archive. Does the formatting destroys something that prevent other partitions on the same harddisk to be automatically remounted???
No, I wouldn't think so. According to the fstab file, where are the mount-points for these other partitions (the ones which aren't mounting)? Are they under "/media"? Do the mount-points still exist in the restored file system? If not, you just need to recreate them: e.gsudo mkdir /media/hda2
I think that Ubuntu creates mount-points like "/media/usbdisk" for external hard drives on-the-fly; perhaps it needs to be done manually, or at installation time, for the internal partitions.

There is no harm in including /media/* in your backup. You don't need to exclude it. Just use --one-file-system to make sure the other mounted partitions aren't added to the archive.

No idea about the /var/cache/apt problem. Linux is supposed to allow /var/cache/ to be deleted between boots. Doing a backup and restoring isn't quite the same thing, because you are taking a snapshot of a live filesystem.

[kilou]

Since I used --exclude=/media/* the mount points for my other partitions were not recreated during the restore. However as you pointed out if I recreate manually the mount points, the partitions are again accessible on next reboot. I'll try to use --one-file-system instead of --exclude=/media/* to see if this avoids this minor issue.

However the problem with update manager remains: it's no more possible to update Ubuntu after the restore. Probably that I must not exclude the content of /var/cache/apt as well. I can't think of anything else that could have caused this problem.

[mannheim]

Did you exclude "/var/cache/apt" or "/var/cache/apt/*" ?

It shouldn't matter, but if it was the first one, then the directory /var/cache/apt will not exist on the restore. If that is the cause of the problem, then you can just mkdir /var/cache/apt, and see if it fixes it. Then next time only exclude /var/cache/apt/*

My backup scripts exclude /var/cache/apt/*, and I have succesfully restored without any problem from apt or synaptic.

[mannheim]

My backup scripts exclude /var/cache/apt/*, and I have succesfully restored without any problem from apt or synaptic.

Ok. Contrary to what I claimed above, trying it again now, I find that apt needs those directories. The backup needs to include:

/var/cache/apt/
/var/cache/apt/archives/
/var/cache/apt/archives/partial/

but it does not need any of the files in those directories. You can just recreate emtpy directories. For rsync, the appropriate rules are

include /var/cache/apt/archives/
include /var/cache/apt/archives/partial/
exclude /var/cache/apt/**


Edit: It seems someone thinks this is a bug in apt: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=403279

[kilou]

Great! thanks for this mannheim :) However I can't see any --include option for TAR. I'll probably leave /var/cache/apt in the backup. Also the --one-file-system allows the backup to work correctly with /media and mounted partitions are automatically remounted after restore, something that was not possible with --exclude=/media/*

[vicnov]

How to restore Ubuntu without Live CD:

Seems that you can create a bootable CD with GRUB loader, look here:
http://www.gnu.org/software/grub/manual/grub.html#Making-a-GRUB-bootable-CD_002dROM

You can put any tools (like TAR) on this CD. Or maybe this CD already contains TAR?

So, you should boot from this CD, switch to command-line mode, set your root (/) partition, and then you may do what you want - make partitions, untar the archive, install GRUB to the hard drive, etc.
Of course, you should read GRUB documentation first! But it is not so much to read.

I did not test this yet, but I want to try on a test machine ;-)

[FrozenFOXX]

Thanks for the assistance, exactly what I needed and nothing I didn't.

[ounas]

Lot's of help here, but has anyone tried the Super Grub Disk from
http://users.bigpond.net.au/hermanzone/SuperGrubDiskPage.html

[fernando_lopes_jr]

How can I do this backup but instead of using exclude I'd like to only include folders since I only want to backup same personal files. Is is possible to schedule this backup once a week.How can I do that?

[Bias]

Hi,

Had occassion to use this today, needed to increase the partition size and could not do this in the partitioner or from a partitioner on a boot cd.

So did the backup (to a different partition).
Rebooted and removed the ubuntu partition and swap partitions.
Rebooted and reinstalled ubuntu to take up all the free space on the disk.
Rebooted and ubuntu started.
Mounted the partition with the backup on it.
Navigated to the directory with the backup in
typed sudo su
typed tar xvpfz backup.tgz -C /
This was working fine until it got to try to create directory /sys/module/isofs were it failed with an operation not permited error.
After this terminal exited to the prompt.

Result, I have a partly restored system.

Anyone know how to get this to create the isofs directory? Have tried manually and get the same problem.

Failing that, anyone know how to get the tar restore process to continue after it fails?

Thanks,

Nick.

[Bias]

Forget the above, I see the process has been updated since I printed a copy so it now excludes the sys directory.

Ended up just reinstalling the applications I needed, bit of a ball ache but it's running again now.

NIck.

[Pugwash]

Just a question, why are we excluding the /sys folder? Does it get automatically generated each time you boot?

[Gen2ly]

Just a question, why are we excluding the /sys folder? Does it get automatically generated each time you boot?

I'm thinking the same Pug. Had to restore so I formated the drive and did as the tutorial said and rebooted. I COULDN'T load into the desktop. Consoles tty2 - tty9 wouldn't start. I think there is some pretty important detail in that file that CANNOT be erased. I'm not using Ubuntu but that shouldnt make a difference.

Edit May be a issue on my end. A link in /dev for some reason was not tarred.

[herbster]

Nevermind

[cubanresourceful]

Is there a way to make it, so that on start-up, it scans the a folder, lets say Audio, and compares it or something, and if there is a difference, then update backup.tgz? Just wondering. :P

[latebeat]

Hi, and welcome to the Heliode guide to successful backing-up and restoring of a Linux system!
...

EDIT : kvidell suggests below we also exclude the /dev directory. I have other evidence that says it is very unwise to do so though.

...

Just writing to note that I followed a similar guide that said not to include /dev and the system didn't boot. However that was only because u need to recreate the device nodes with a live cd after. see:
http://forums.gentoo.org/viewtopic-t-552411-highlight-.html

if u have similar problems

[aniskop]

Hi, and welcome to the Heliode guide to successful backing-up and restoring of a Linux system!
....
As always, any feedback is appreciated!

Hi

I tried your backup and after recovery and after recovery I cannot switch from X to ttys.

That's my second try - I tried backup and recovery on other box with different configuration.

I tried recreate

mknod tty0 c 4 0

but Ctrl + Alt +F1 doesn't work

Any ideas??

Thanks
Peter

[joel_gil]

Hello everyone, 1st of all, GREAT GUIDE

i got to do the backup file with no probs, as well as resotre it.. kinda

it overwrote al the files as expected and once i rebooted when i hit on the ubuntu OS on the grub it says "error 17: unable to mount partition, select another OS"

i made the backup some weeks ago and durin that time i reformatted my computer and moved the linux partition (it was on sda4, now its on sda3) i guess that could be the reason... but again i have no idea

anyone who knows how to fix it?

thanks in advanced

[kilou]

I'm using Feisty and wanted to use a separate home partition to keep my settings for future clean installs. Previously when I was using home on the root partition I was backing up the system using this TAR command and everything was fine (backup AND restore):

sudo tar -cvpzf backup.tgz --one-file-system $exclude_list /

with exclude_list="
--exclude=/lost+found/*
--exclude=/tmp/*
--exclude=/var/tmp/*
--exclude=/var/log/*
--exclude=/var/cache/apt/archives/*.*
--exclude=/home/user/.Trash/*
--exclude=/home/user/.thumbnails/*
--exclude=/root/.Trash/*
--exclude=/root/.thumbnails/* "

Now that /home is on a separate partition I had to change the TAR command. I decided to create one backup file for root and another separate backup file for /home. The backup file for root filesystem is:

sudo tar -cvpzf backup_root.tgz --one-file-system $exclude_root /

with exclude_root="
--exclude=/lost+found/*
--exclude=/tmp/*
--exclude=/var/tmp/*
--exclude=/var/log/*
--exclude=/var/cache/apt/archives/*.*
--exclude=/root/.Trash/*
--exclude=/root/.thumbnails/* "

Since /home is on another filesystem this backup effectively doesn't include it since I use the option --one-file-system.

I then did some tests, format the root partition (untouching /home that was on a separate partition and working, I didn't delete it!) and then booting from the LiveCD I restored the root file system from the above TAR command. Everything appears to work correctly when untarring but when I reboot the system hangs on the Ubuntu boot image (the bar that represent boot process stay at the beginning. I tried several times and always get the same results. I also tried to remove the --one-file-system option in the TAR command and add --exclude=/home/* to the exclude_root list (see above) but always get the same results: the system is able to boot with Grub, load the Ubuntu boot image but the boot process stops.

I also tried to launch Ubuntu in recovery mode to see where it hangs and the boot stops after loading USB drivers.

Do you know what Iam doing wrong? It seems that it's impossible to backup and restore a root partition if home is on another partition (untouched).... The backup was saved on an EXT3 volume so the permissions are stored.

Any advice would be appreciated. I'd like to solve this problem before going with a proper install of Feisty and tune all parameters. Thanks!

PS: I'm restoring from a LiveCD to the root partition (sda1) with:

#Mount root partition
mkdir /mnt/restored_root
mount /dev/sda1 /mnt/restored_root

#Restore root system
tar xvpfz backup_root.tgz -C /mnt/restored_root

[phidia]

This looks like an excellant guide-unfortunately I have the same situation as kilou-my home is on a separate partition. I just had an unbootable system and couldn't fix it I had a back up of etc,var, boot, and usr/local but it didn't help at all. (i think etc was corrupted already when I did the backup) IMO there needs to be a simple and useable backup program that will successfully perform a restore.

[kilou]

I found the problem: for some reasons the Feisty did change the UUID of my root partition so when untarring the system from the archive, the UUID in fstab and menu.lst didn't match the new UUID.

What I did:
- boot on the LiveCD
- format the root partition
- restore the root partition from the tar archive
- open /dev/disk/by-uuid on the LiveCD filesystem and check the UUID for every partitions (these are the new UUID that need to be copied on the restored partition)
- go on the restored root partition, open /etc/fstab and adjust the UUID with the new ones
- on the restored root partition, open /boot/grub/menu.lst and adjust the UUID of the root partition with the new one
- Reboot and your restored system should be useable!


It doesn't appear to be a problem with TAR or the backup process. It really has something to do with Feisty changing the UUID of partitions. Probably this happens when a partition is formated :confused: However it is strange because it seems if the whole filesystem is on the same partition (no separate home), there is no problem.........

I'd like to find a way to automatically correct the UUID in fstab and menu.lst after the restore process.....

[jaakan]

Has anyone tried http://freshmeat.net/projects/g4l/ ?

[kilou]

I didn't try it but it doesn't appear to be anything different from Partimage. Those ghost programs are certainly nice but they have 3 drawbacks for me:

- you cannot specify which folder you want to backup and which one you don't want. They just ghost a whole partition
- the tool is not part of the LiveCD, you have to download the package everytime or build a specific boot CD for it. I prefer using the LiveCD but if I don't have access to Internet then I can't restore the system...
- I think that you cannot do live backup with ghost programs (not sure though....)

I really like TAR with a specific script a lot better. I can backup exactly what I want and nothing more to keep the files rather "small" and it is immediately recoverable from the LiveCD since TAR is not a fancy command. There is the possibility to do incremental backup and everything can run in the background on a live system.

I just need to figure out this UUID problem with Feisty but I guess this only happened because I formatted the drive with the install util on the LiveCD. If I run sudo mkfs.ext /dev/hda this should work without changing the UUID.

[wiseleader]

Just finished Heliode's backup strategy at very beginning of this post.

100% works !

I wish to add some more cautions :

the exclude path of the backup file must be changed if you're not planning to put the backup file in root

for example, on backup

sudo -i
mount /dev/sda1 /mnt/usbdisk
cd /mnt/usbdisk
tar cvpzf backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/mnt/usbdiskbackup.tgz --exclude=/mnt --exclude=/sys /

I didn't test the restore guide, I was wondering whether to restore the backuped system to a corrupted system works or not. My way is to reinstall ubuntu (any version) so that the original partition and mounting is retained, then restore the backup.tgz file. 100% works !

This way is great if you're going to move whole system including data to a new hdd, but please don't forget to use the same partition and mounting information.

I am a newbie, maybe my method is stupid to EXPERIENCEDbies.

Thanks for Heliode's posting, it's really great and helpful by using the native tools of Linux, we don't need Norton Ghost....so and so here.

Sincerely hope I can do something in Ubuntu world.

p.s. still looking for a good method to backup my alternative WinXp system similar to Heliode's method.

[kilou]

Personally I used --exclude=/folder/* instead of --exclude=/folder to exclude a folder (note the /*). This way only the content of the folder is excluded but not the actual folder itself. This is useful because you don't need to recreate excluded folders after restoring since they are automatically recreated by the archive (they may not apear in the archive through because they are empty). This is important because the excluded folders will be recreated with the correct permission because if you recreate them manually from a LiveCD for instance, they won't get the right permissions and your system might not boot (mine hang after login screen because one of the excluded folder had not the correct permissions after a restore from the LiveCD)

[wiseleader]

Personally I used --exclude=/folder/* instead of --exclude=/folder to exclude a folder (note the /*). This way only the content of the folder is excluded but not the actual folder itself. This is useful because you don't need to recreate excluded folders after restoring since they are automatically recreated by the archive (they may not apear in the archive through because they are empty). This is important because the excluded folders will be recreated with the correct permission because if you recreate them manually from a LiveCD for instance, they won't get the right permissions and your system might not boot (mine hang after login screen because one of the excluded folder had not the correct permissions after a restore from the LiveCD)

Your idea is great, thanks for sharing experience, I'll change my backup script :)

[kilou]

Your idea is great, thanks for sharing experience, I'll change my backup script :)

Well I'm a newbie and this is not my idea, I found it somewhere but just remember where......but glad to help :)

[abcuser]

For Windows you would need proprietary software for which you would have to reboot your machine and boot into a special environment in which you could perform the backing-up/restoring (programs like Norton Ghost).
I have been using Windows for more then 15 year and what I like about Norton Gost is to start the program from DOS mode, because I am 100% sure no program is writing to my disk storage during backup of my whole disk.

How can I be sure that "tar" method in Ubuntu will prevent writing any program during backup (for example database server writing to database)? So how to prevent corrupted backup? Is there any other more reliable way of doing a backup of whole system?

[kilou]

Partimage work just like Ghost and you can also do the backup from a boot disk. You can also do TAR from a LiveCD so that your system is not live while you do the backup. However as far as I know these considerations only comes because we are used to Windows. Linux apparently work differently and there is no problem doing a backup on a live system.....it's even possible to restore a system while working on it!! I just love TAR because it can be customized just to my liking and I can do backup on the fly from a living system. This is much more reliable for me because I don't hesitate doing a backup since it does take any time as you can continue to work. On Windows I had to reboot on a Ghost disc and perform the backup... Recent Ghost version can also backup on a living Windows but I've head horror stories about that and noone recommend doing that with Windows. With Linux it's a completely different story :)

[shizeon]

Great tutorial. Due to disk space constraints, I needed to export the tar to an external drive that is fat32 formated. If this hasn't been posted already, thought I would share how to get around the 4GB size restraint using the split command.

If you take Heliode's command and alter it like this

tar cvpzf - --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/media --exclude=/sys / | split -a 2 -b 2000m - /media/ieee1394disk/backup.tgz


It will create your backup.tar in 2000MB chunks at /media/ieee1394/

Restore like

gzcat /media/ieee1394disk/backup.tgz* | tar xvpfz - -C /


I've done the backup portion using this, but haven't actually tested a restore. So use at your own risk :)

[kilou]

Well I think the problem with a FAT32 partition is that it won't store permission of your files so it's likely you'll get troubles after restore. You should probably make an Ext3 partition to store the backup if you intend to keep the permissions on your files.

[shizeon]

Well I think the problem with a FAT32 partition is that it won't store permission of your files so it's likely you'll get troubles after restore. You should probably make an Ext3 partition to store the backup if you intend to keep the permissions on your files.

The tar file itself stores your permissions. You can place the tar file on any type of file system you want without effecting the contents.

[kilou]

I've been told that only Linux filesystems will store the permissions, even within a tar file. Did you try to restore some archived files from the FAT32 and see if the permission were correct?

[shizeon]

Did you try to restore some archived files from the FAT32 and see if the permission were correct?

Yes, did a test restore tonight and all permissions and ownerships were maintained.

[cypah]

After messing up the xwindow config trying to increase resolution on my ancient 'real 3d starfighter vid card, I got xwindow failure and had a command line. I went looking for how to restore and found this guide...I tried to follow this guide to restore my Badger 5.10 I only sort of understand the partitioner and finally agreed to allow it to format the Root and Swap partitions since it seemed to be in a loop with that, hoping that would leave all data on the #3 & #4 intact.

I did get the error messages complaining it was a fatal error for Grub I asked it to finish and it did. Unmounted the installation cd and booted but then is sitting right now saying "Grub errow, please wait .. error 15.

I am stuck, still hoping I will not have to total install and lose all my mail on local machine, important but not critical.](*,)
thanks Herb

[dtoader]

####################
# NFS Server Install
####################

# Create the backup directory
mkdir /backup

# Install NFS server
sudo apt-get install nfs-kernel-server nfs-common portmap
# When configuring portmap do not bind loopback.
# If you do you can either edit /etc/default/portmap by hand or run:
sudo dpkg-reconfigure portmap
sudo /etc/init.d/portmap restart

# Create a share
sudo echo -e "/backup\t192.168.1.0/24(ro,sync,nohide) ## use tab, not spaces" >/etc/exports

# save this file and then in a terminal type
sudo /etc/init.d/nfs-kernel-server restart

# Also after making changes to /etc/exports in a terminal you must type
sudo exportfs -ra

####################################
# Save system settings on NFS server
####################################

fdisk -l > /backup/`date +%Y%m%d`_`hostname`_partition_table
cp /etc/fstab /backup/`date +%Y%m%d`_`hostname`_fstab
cp /proc/mounts /backup/`date +%Y%m%d`_`hostname`_mounts
cp /etc/mtab /backup/`date +%Y%m%d`_`hostname`_mtab
cp /boot/grub/menu.lst /backup/`date +%Y%m%d`_`hostname`_menu.lst
# Send everything to cpio except the /backup directory and /dev files
(cd / ; find . -xdev -wholename ./backup -prune -o -print | cpio -voc > /backup/`date +%Y%m%d`_`hostname`.cpio.gz)

########
# Client
########
# Boot using the LiveCD.
# Set the root password.
sudo passwd
# Login as root. This is not necessary. I'm lazy and don't like to sudo everything.
su
# Create your partitions manually
# See above line containing /backup/`date +%Y%m%d`_`hostname`_fstab
fdisk /dev/hdc

################################################## ############################
# My server/original 20070512_original-desktop_partition_table file looks like:
Disk /dev/hda: 60.0 GB, 60081831936 bytes
255 heads, 63 sectors/track, 7304 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 7117 57167271 83 Linux
/dev/hda2 7118 7304 1502077+ 5 Extended
/dev/hda5 7118 7304 1502046 82 Linux swap / Solaris
################################################## ############################

# The client system has /dev/hdc as the hard drive label.
# It should look something like below. You need to fill in the x's
# with valid values that will get you in the ballpark for the client
# system. Your units could be different so might need to use a different
# number of blocks also.
################################################## ############################
Device Boot Start End Blocks Id System
/dev/hdc1 * 1 xxxx 57167271 83 Linux
/dev/hdc2 xxxx xxxx 1502077+ 5 Extended
/dev/hdc5 xxxx xxxx 1502046 82 Linux swap / Solaris
################################################## ############################


# After partitions are created, make filesystem
mkswap /dev/hdc5
mkfs.ext3 /dev/hdc1
mkdir /mnt/hdc1 /mnt/nfs
mount /dev/hdc1 /mnt/hdc1
mount 192.168.1.50:/backup /mnt/nfs

# Check to see if the nfs share shows up ok.
cd /mnt/nfs
ls

# Now, you can proceed to unzip the backup into your new system
(cd /mnt/hdc1 ; cpio -vicdm < /mnt/nfs/20070512_original-desktop.cpio.gz)

# Change the hostname and empty existing ip/hw address info.
# This is done in the chroot environment.
chroot /mnt/hdc1
mount -t proc /proc proc

# Change the hostname of the client machines. For example:
sed -i 's/original-desktop/copy-desktop/' /etc/hostname
sed -i 's/192.168.1.50/192.168.1.51/' /etc/network/interfaces
# Change drive assignments.
sed -i 's/hda1/hdc1/' /etc/fstab
sed -i 's/hda5/hdc5/' /etc/fstab
sed -i 's/hdc /hda /' /etc/fstab
sed -i 's/hda1/hdc1/' /boot/grub/menu.lst

# Now, I exit the chroot environment
umount /proc
Ctrl+D or type: exit

# Now we are back to the LiveCD environment.
# Now install grub.
grub

grub> find /sbin/init
grub> root (hd0,0)
grub> setup (hd0)
grub> quit

# Unmount all.
cd /
umount -a

# Restart the system.
# You might need to reconfigure networking.
# After rebooting, look at the network interfaces
ifconfig

# If your system has reassigned your NIC as eth1
# (suppose you expected it to be eth0 as before)
vi /etc/network/interfaces
# Reverse eth0 for eth1 and vice versa.

################################################

# Before
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 192.168.1.51
netmask 255.255.255.0
gateway 192.168.1.1

auto eth1
iface eth1 inet dhcp

auto eth2
iface eth2 inet dhcp

auto ath0
iface ath0 inet dhcp

auto wlan0
iface wlan0 inet dhcp

################################################

# After
auto lo
iface lo inet loopback

auto eth1
iface eth1 inet static
address 192.168.1.51
netmask 255.255.255.0
gateway 192.168.1.1

auto eth0
iface eth0 inet dhcp

auto eth2
iface eth2 inet dhcp

auto ath0
iface ath0 inet dhcp

auto wlan0
iface wlan0 inet dhcp

################################################

[dtoader]

The above instructions worked for me on my home network.
Using Ubuntu 6.06.1

Questions or comments are welcomed.

[jhenager]

The only error message I got was:
tar: /: file changed as we read it
root@ubuntu-desktop:/#

but no worries. I understand that "/" changed, because I wrote to it by creating backup.tgz.

Great writeup.
Highly recommended procedure.:KS

[rsvirani]

Is there no graphical backup program that supports incremental backups and scheduling for automated backups?

[kilou]

Is there no graphical backup program that supports incremental backups and scheduling for automated backups?

sbackup does that AFAIK

http://onlyubuntu.blogspot.com/2007/03/backup-and-restore-ubuntu-system-using.html

[vyoufinder]

This backup method didn't work for me for several reasons. First, I kept getting an error while making the backup saying that profile had been written to. I tried it 3 times before figuring out that I can't do it using sudo, I have to actually logout and login as root. Then it will work. Using sudo only works if your filesystem is very small from what I can tell. Then when I did the restore, everything went smoothly. I installed Ubuntu on the new drive, then unpacked to the new drive. Restarted and the Ubuntu screen loads. That's it, it loads and stays there and doesn't change.

[vyoufinder]

It took me 2.5 days to figure it out but I now have a backup. None of the tools mentioned in this post worked for me. I tried them all I think. I also tried all the methods described here but most of them have missing steps. I kept backing up then tried to boot from the new drive and it would freeze after Grub, as soon as it shows the Ubuntu logo. It would just hang there. Finally I hit a carriage return and the damn thing booted. So.. I could've been done 2 days ago if I had known that part. HAVE TO HIT ENTER TO BOOT (now I need to figure out how to get rid of the enter button needing to be pressed!)


*EDIT* - nope. I just realized I had the original hard drive connected.. I'm still trying to make a freaking backup!

[nami]

I tried this and it didn't work. I have a lovely working system with twinview and nvidia drivers and everything. So I backed it up, and just as a test I removed the nvidia drivers and tried restoring the system to get the nvidia drivers back. It wouldn't let me into gnome after the restore...

[dasbooter]

It took me 2.5 days to figure it out but I now have a backup. None of the tools mentioned in this post worked for me. I tried them all I think. I also tried all the methods described here but most of them have missing steps. I kept backing up then tried to boot from the new drive and it would freeze after Grub, as soon as it shows the Ubuntu logo. It would just hang there. Finally I hit a carriage return and the damn thing booted. So.. I could've been done 2 days ago if I had known that part. HAVE TO HIT ENTER TO BOOT (now I need to figure out how to get rid of the enter button needing to be pressed!)


*EDIT* - nope. I just realized I had the original hard drive connected.. I'm still trying to make a freaking backup!

:D been there, done that I remember building an old system after just finishing exams went to install the OS .... Hmm whats that burning smell ...ack! I forgot to plug the cpu fan. When your tired its best not to attempt rocket science ;) Sorry didnt mean to digress but I feel your pain. I think an easy gui based scheduable bare metal restore package should be priority #1. In order to convert new users we really need this. Learning linux/ubuntu is in someways akin to learning a high wire act and you will probably fall eventually while learning something new and you Just need that safety net to Work .A safety net that is too complex to set up wont get set up or will fail.

On that note did you get sbackup to work or did anybody ever try mondorescue which was mentioned much earlier in this thread?

I have tried mondorescue before but I never could make it work although that was with suse linux...I dunno but too me this is the most important thing ... I don't mind troubleshooting an issue, its always a learning experience but even I give up eventually and sometimes you just need a working system...

[vyoufinder]

Mondo didn't work for me. It looked like it was going to but it couldn't burn to disk. Was looking promising. I'm still trying to fix my backup I made using the tar method described in this thread. I know it works because when I boot using Super Grub Disk it works fine. But that's not going to work for me. I need it to boot from hard disk. I've tried uninstalling and re-installing grub too. No luck. Hangs on the Ubuntu screen when booting as soon as the progress indicator begins.

[dasbooter]

Mondo didn't work for me. It looked like it was going to but it couldn't burn to disk. Was looking promising. I'm still trying to fix my backup I made using the tar method described in this thread. I know it works because when I boot using Super Grub Disk it works fine. But that's not going to work for me. I need it to boot from hard disk. I've tried uninstalling and re-installing grub too. No luck. Hangs on the Ubuntu screen when booting as soon as the progress indicator begins.

Hmmm had a similiar problem with another distro. Try reducing everthing to the simplest configuration eg. one hard drive. Sometimes I think the system or the person may get confused about which hard drive is really the first hard drive. Grub is typically installed to the MBR of the first drive. Sometimes I wonder if people have problems here as they typically add in another hard drive to back up to then they run into problems reinstalling grub or there whole system and next thing you know it is a big mess.

I am not sure how you get to terminal in the boot but in suse during boot up you just hit escape and the nice logo screen disappears giving you a detailed output of what is going on in the boot.Maybe somebody can pipe up and say how this is done in Ubuntu. I think more information as to what is happening is crucial. I had a problem wherein a whole bunch of modules were not being loaded making my system unusable and this was due to somehow my fstab getting screwed up and I think grub was calling the wrong partition. If I booted the system using the suse install disk everything went ok which really frustrated me. I went through my fstab and made sure that grub was calling the correct system partion. You will have to learn the basics of using grub. Sorry if I am telling you stuff you already know. I am curious as to when you boot your system from the CD is everything functioning normally can you read and write from everything. Did you already post your /boot/grub/device.,map , /boot/grub/menu.lst , /etc/grub.conf .

Lastly you may want to scrub your mbr if you have installed a third party boot loader you may have to fdisk /mbr to clean it out.

[slackmaster]

Hello,

Thanks for explaining hot to backup and restore an Ubuntu system. I will try this soon. I have a few questions though. Do you have any experience restoring an windows backup to an Ubuntu OS? I'm not sure what to do with all these BKF files I have on my external HD.
Also, I wonder if you have any experience restoring or importing PST files to Thunderbird 2.0.
Finally, can you think of anyway to search ZIP backups for specific file extensions without extracting 50 ZIP folders? Or do just have to go through each one and extract the ones that have stuff I want?
Thanks ahead of time for any tips.

Regards,
S.

[dannymichel]

I have a 30 gig partition for Ubuntu and 2 320gig HDDS with Vista and OSX86 installed on them with about 25% of those 320gig HDDs filled each. If I use the first method described, how big is this backup going to be?

2 more questions:

Where do I find the backup on the HDD
How do I find out how much space has been filled on any given HDD on Ubuntu?

[CautionaryX]

If your referring to the first post, I'd suggest keeping your windows partitions/drives from being included in the backup. The backup is by default placed in the root directory in Ubuntu. You will need to change permissions on the backup file too.

[xyz]

* How do I find out how much space has been filled on any given HDD on Ubuntu?


You can run the following command in a terminal:
df -a

[dannymichel]

If your referring to the first post, I'd suggest keeping your windows partitions/drives from being included in the backup. The backup is by default placed in the root directory in Ubuntu. You will need to change permissions on the backup file too.

"keeping your windows partitions/drives from being included in the backup"
How do I do that?

[Thorun]

@dannymichel
type in the code as in page one:
--exclude=/mnt --exclude=/media
which should exclude the partitions you have mounted in those folders e.g. windows and other data-partitions.


I have a question on the restore-process though.
I did make a backup with success. Several in fact. I will call the files on my computer today "Y" and the files in the backup-file for "X" files (hehe).

When i did the restore-process as told on the front page - i thought, that it would remove all my Y-files in the root and the folders connected to the root. So to speak - i would end up with a restored system with only X-files left.
BUT!
To my suprise, after the restore, i did both have my Y and X-files. Which made a big mess of new and old files everywhere. Then i cried a lot ;-) and started all over, reinstalling Ubuntu from scratch to a formated partition.


Did i do anything wrong in the backup-process? Or what can i do, to get completly rid of the Y-files when i want to restore a backup containing X-files?

[guybrush1]

In the first post it says:

EDIT2:
At the end of the process you might get a message along the lines of 'tar: Error exit delayed from previous errors' or something, but in most cases you can just ignore that.

---

Can you explain how to know if you can ignore this error ?

---

How can I compare my tar file against the directory itself?
tar --compare -f file.tar /the/directory doesn't seem to work...

[Das Goat]

Ok! I'm going in! Going to back up to a UBS HD, erase some partitions and restore.

I read most of the posts! I feel confident! I BELIEVE it will work!

Mommy! :o

[Das Goat]

Holly Guano Batman!

Well, that was quite a disaster. The commands seemed to work all right. the .tar was created.

The problem came in when I tried to copy the file to my USB external HD. Maybe it was because it was formated under Windows, but I couldn't seem to copy the .tar file there. If it was smaller, it would have fit on a 1 gig thumb drive. Maybe I should have tried to burn it to CD, but I though "hell, it should just go on the ext HD.

Well, I guess I didn't quite know what the correct command was because I over wrote the whole 100 gigs with crap by accident, so Windows nor Ubuntu could read it. So all of my stored Ghost images for all my different laptops went....Poof!

Oh well. At least I don't have to worry about space any more. And my last Win image was almost a year old, so what better time to ditch Windows totally and format the drive pure Ubuntu?

Right now I have to get Virtual Box working (so I can install quicken and get my bank info) but after I do that I will try again.

[Das Goat]

This might not be for this thread, but how do i mount a USB device and write to it?

This may seem like a silly question, but it is how I toasted my external drive by doing it wrong.

If i plug in a USB stick to my Ubuntu, it reads and writes to it with out needing permissions, but I can not just paste to my UBS HD. I assume it is permissions, but how do I set them.

This is germain to the thread because once I make my image, I have to be able to write it some where.

Thanks in advance.

[DrRootabega]

Nice guide.

But I used Norton Ghost which works perfectly for me. Norton Ghost 2003 supports EXT2/EXT3 filesystems. Older versions of Norton Ghost DON'T support EXT3 filesystems or Linux altogether.

Please go to this site (http://service1.symantec.com/SUPPORT/ghost.nsf/docid/1999021909463125?Open&src=&docid=2000033111503625&nsf=ghost.nsf&view=docid&dtype=&prod=&ver=&osv=&osv_lvl=) for a feature compare between the different Norton Ghosts versions: Norton Ghost compatibility with Linux. (http://service1.symantec.com/SUPPORT/ghost.nsf/docid/1999021909463125?Open&src=&docid=2000033111503625&nsf=ghost.nsf&view=docid&dtype=&prod=&ver=&osv=&osv_lvl=)

I ended up with a backup-image that's about 1 GB big, the amount of data on the disk that was backed-up by Norton Ghost was about 2 GB. So as you can see the compression was very good. I went from 2 GB to 1 GB. :grin:

I used the bzip2 version of this guide and it packed my 2.5 gig partition down to 742 meg.

[xfile087]

I know loads of you have probably already posted your backup scripts that you've created but I thought I would also post mine seen as I couldn't of done without this thread and it was good practice. It's far from perfect but someone else might find it useful.

It's a simple bash script that uses zenity to display dialogs with questions (what you want to call your backup, where, compression etc).

You can just install the .deb and use it straight away (System->Administration->Matt Backup) or you can also download the attached archive which includes the source so you can modify it. Like I say, i'm only learning so it's far from perfect and might not even work for you (though it's working completely fine for me on Ubuntu 7.04 and even on Arch Linux).

[Das Goat]

well, after much disappointment, I think I will have to abandon this approach. The backing up seems to work, i know because I can "unpack" files I stuffed into the archive (I restored my music).
But the restoration part doesn't seem to work. It seems like all the files are there, from what i can tell, but the system will not boot. It doesn't get past the Ubuntu splash screen.

i tried it several times, rebuilding Ubuntu every time because none of the backups i made would restore.

Damn shame. I really need to image my system to get the best results learning and installing packages. if i can't "roll back" my system by restoring an image, I am very hesitant to try new things.

there is one silver lining though. all of this has prompted me to make a "personal restore" document telling me what to do in what order so that I get everything back they way i want it if I have to scrub the drive.

:confused:

[HermanAB]

Well, you really only need to backup your data. There is not much point in backing up the system files. When a system breaks so badly that you need to restore from backup, you probably would want the latest version of Ubuntu, not the old one that you backed up. You may even want to build a new machine, in which case the old system backup won't work anyway since the hardware is different. So to recover from a disaster you should install the system the usual way and then restore your data.

Cheers,

Herman

[bomanizer]

well, after much disappointment, I think I will have to abandon this approach. The backing up seems to work, i know because I can "unpack" files I stuffed into the archive (I restored my music).
But the restoration part doesn't seem to work. It seems like all the files are there, from what i can tell, but the system will not boot. It doesn't get past the Ubuntu splash screen.


This sounds very familiar. I've done some backing up & restoring using tar some months ago. I remember similar problems, and the solution was to chmod 777 a certain folder, but I can't remember anymore which one!! The trouble was with X sessions, X needed some write-permissions, therefore the "chmod" was needed.

EDIT: Oh! oh! I got it! It was /tmp :)

[bomanizer]

you probably would want the latest version of Ubuntu, not the old one that you backed up.

Not true always. I got recently a Thinkpad, replacing my Aspire-crapiola-laptop. Now I sometimes regret that I started using Feisty. I had fever issues with Edgy. What's with the UUID anyway??

[lovefunelfish]

I recommend the "clonezilla" that can backup your system by partition or entire hard disk or network..

http://clonezilla.sourceforge.net/

[nomad5000]

I have backed up my entire ubuntu partition bacause I needed to repartition my hard drive. I did it exactly like this tutorial said. But after I repartitioned my hard drive and reinstalled grub. Linux wont boot, it doesn't tell me any error messages it only gets to the point where the ubuntu logo and the load bar are displayed and after that nothing happens. So what I want to know is if anybody can tell me what is wrong, and if not, how I can get ubuntu to give me a meaningfull error message.

Thanks very much for your help.

Michael

[mesh]

Helllo all,

I recently tried this.

-to restore i booted into my feisty live cd
-deleted my current partition , recreated and reformatted.
-Note: i had to kill gnome-volume-manager to do the step above.
-then then untarred the files to my newly created / formatted partition
-My hard drive what mounted to /media/disk-1 (cause i booted from the live cd?)
-after untarring and crating the necessary folders i rebooted.
-I then received the grub text (normal)
-Was greeted with the splash and got stuck there
-i let it sit and it eventually gets to the BusyBox v1.1.3 with error:
-/bin/sh: can't access tty; job control turned off
(initramfs) --- (ram drive?)


any insight? i wanna learn how to back up and restore successfully becuase everytime i try to use any linux as my primary os i get it working in a way that im comfortable and then install or something happens that will mess up my config and i revert back to windows. I want to stick with it for a while. Please give me some insight into the issue. Im thinking it has to do with the partitioning part becuase i untarred everything before without partitioning and it worked fine. I want to learn how to bring back a totally dead system as if the hard drive crashed. Thx to all in advance. Peace


Anyone have any experience with this?

[mesh]

anyone have any ideas?

[mesh]

well i had to revert to reinstalling Feisty and then just untarring my entire back up over it. System's back up with all my settings. Took forever though.

[Cyynic]

this is one of the main how to's i was looking for!

thanks a bunch, the backing up seems to work great... haven't tried the restore yet though :)

cheers

[cdblind]

Man I have always used windows and have recently converted to linux. Thank for this it is so simple. Hope I never use it but I will follow the procedure:popcorn: right away to back up my system.

[Midahed]

I've been looking for a guide to safe backup and restore practices and came across this thread. The original post is now over two years old....

Quite a few contributors want to be able to use this method to restore a system over a new or re-built installation.

My question is, how is this impacted by the move in Feisty to the use of UUIDs? If you re-partition a drive and then restore a backup into your new root filesystem, won't the UUID be different and won't this break the restored system?

I've been bitten by UUIDs recently, so it would be good to know if they have any impact on the methods discussed in this thread.

[EDIT] Having thought about this a bit more, I suppose one solution would be to keep a note of the system UUIDs and then, after re-partitioning and putting a filesystem on the target drive, use tune2fs -U <original uuid> /dev/hda1 to ensure that the UUIDs on the new system match the originals that appear in the backup.

Does that make sense? Has anyone had to do this to get over the problem of changing UUIDs when a new filesystem is created?

Mike

[markdjones82]

How big are the backups on these normally? Mine came up to like 54 GB and that seemed quite large to me. Is that normal?

[Midahed]

How big are the backups on these normally? Mine came up to like 54 GB and that seemed quite large to me. Is that normal?
It depends on how much data you're backing up in the first place, the compression algorithm used and, to some extent, the nature of the data you're backing up. For instance, if much of the data is already in a compressed format, that element of your data will hardly be further compressed at all.

Mike

[Desigen]

Great Article
Thanks

[ephman]

How big are the backups on these normally? Mine came up to like 54 GB and that seemed quite large to me. Is that normal?

are you backing up media like mp3's, movies, etc...? that could do it. i know it's all relative, however, my back up is only 4gig (i leave out all the media).

thanks for the bandwidth,
ephman

[markdjones82]

Hmm, I don't believe so, but I'll check again. Maybe I'll just use ghost and create an image of sorts.

[mpgarate]

i have to say very useful tut.. my favorite part is where its copying the files and theyre running down the terminal window really really fast!

[soeten]

Thanks, great guide. Now Im really starting to like ubuntu. <3

[force4]

HI...

I have a fresh install on my desktop.

/ is 10 gig on hda2 (hda1 is xp). hda3 is 33 gig and mounted as /home.

Everything works fine backing up my /home directory using tar.

Problem is when I try to backup the root system.

I log in as root user (not sudo from my user).

I unmount my xp partition (right clicking on the desktop disk and choosing unmount)

...and issue the following commands in the console (taken off the first post)

cd /
tar cvpzf backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz -- exclude=/home --exclude=/media --exclude=/mnt --exclude=/sys /

...press enter then tar takes off. The problem is...it doesn't exclude anthing.

My /home directory is included in the tarball and it is so big that the 10 gig root partition runs out of space and the process dies.

also this is the first few lines of the verbose output...

root@AMDx2:~# cd /
root@AMDx2:/# tar cvpzf backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz -- exclude=/home --exclude=/media --exclude=/mnt --exclude=/sys /

tar: Removing leading `/' from member names
/home/
/home/steve/
/home/steve/.gpilotd/
/home/steve/Downloads/
/home/steve/Downloads/stellarium_user_guide-0.9.0-1.pdf
/home/steve/Downloads/BusinessDisasterRecoveryebook.pdf
/home/steve/Downloads/envy_0.9.7-0ubuntu8_all.deb
/home/steve/Downloads/wicd_1.3.1-all.deb
/home/steve/Downloads/Foundations_2007_Desktops.zip
/home/steve/Downloads/stellarium-0.9.0.tar.gz
/home/steve/Downloads/nspluginwrapper-install-0-1.3.tar.gz
/home/steve/Downloads/dohickey_0.6.2_all.deb
/home/steve/Downloads/lw_conferencegrid.pdf
/home/steve/.themes/

What can I be doing wrong? :confused:

Only 3 weeks as a linux noob so I know it could be something simple.

Steve

[Midahed]

I think you need to use the --one-file-system option because you're /home is on a separate partition.


tar cvpzf backup.tgz --one-file-system --exclude=/proc --exclude=/lost+found....

One of the earlier posts in this thread mentioned it:-

http://ubuntuforums.org/showpost.php?p=2558003&postcount=286

Hope this helps.

Mike

[cdtech]

How big are the backups on these normally? Mine came up to like 54 GB and that seemed quite large to me. Is that normal?

I use the --files-from flag and the --exclude-from flag to include a file (a list of files, one per line, and treats it as if each file appeared separately on the command line) such as:

dont-backup file ::.
*.mp3
*.mpg
*.avi
*.wav
*.mov
*.asf
*.rm
*.iso
*.flac
*.zip
*.exe
data.bin
/usr/local/fonts
/usr/local/games
/var/cache/apt/archives
/mnt
/backup

do-backup file ::.
/boot
/etc
/home
/root
/usr/local
/var
/downloads
/tmp
/scripts

[kushagra]

Excellent post.. I backed up and restored my system several times by the technique you explained. It almost always worked. But I suppose this technique doesnt restore ur system to EXACTLY the same condition as before. This what I did:
1. I backed up my system as you expalined
2. I installed wine (on my feisty)
3. something went wrong and I untared the files that I had backed up. But, I still have the wine directory and related stuff on my system, amounting to several hundred MBs.

Isnt there a parameter that can be passed while untaring which will remove any files from the system that are not included in the archive?

On one occasion, I restored my system but could not boot into ubuntu. I got a message at startup, saying that some file in the /proc directory was missing. Should I have backed up that directory too?(I had not included it in the tar) Is it better to unmount all ur partitions and then backup, or to not include the whole /media directory itself.

Kindly clarify.

[ropenstein]

I didn't see where anybody has mentioned dd. That is what I use to back up disks and partitions in place of ghost, even with Win disks. Basically all you do is...

dd if=/dev/hda5 of=/somedir/mypart.img

and to restore to disk...

dd if=/somedir/mypart.img of=/dev/hda5

So basically it just copies whatever block device specified (/dev/fd0, /dev/hda1, /dev/sdb2, etc.; I think you can even /dev/hda to get all partitions on a disk) to a file and vice versa. I'm no expert, but it works great for copying hard disks, floppy disks, partitions, and the like.

The neat thing is, once you have an image of a partition you can actually mount it using...

mount /somedir/mypart.img /mnt/mymountpoint -o loop

As always, BE CAREFUL or you can really wipe some stuff out.

[srf21c]

dd will definitely get the job done. You can also pipe dd over an ssh connection to a remote machine, which is handy when you're short on local diskspace. However the biggest drawback to dd it's more or less a "dumb" command in that it copies everything, even empty sectors.

So if have a 80GB hard disk (pretty standard these days) with a single large partition on it containing only 15GB of information, with dd you'll still be backing up an extra 65GB of....well nothing really.

[synss]

to the OP:
you should exclude /dev, /var/lock, /var/run, /tmp and probably /var/tmp

/dev is dynamically populated, since, with recent kernels, udev is the rule. Backing it up may or may not give problems, and will probably put error messages at the console.
restoring /var/lock will cause trouble, an app puts a lock when only one instance should run, very likely, when you restore your system, it does not run and may be unable to start (X may not start)
/var/run is also used temporarily for the daemons

these two will be wiped anyway by the initscripts when you boot

/tmp and /var/tmp should not contain anything any app needs.

you can use wildcards to skip the re-creation of directories on restore, like:

--wildcards --exclude=/var/run/*

you should not use the "v" option as there are too many files to monitor what is going on and you might miss the error messages in the noise.

man tar
man udev
FHS: http://www.pathname.com/fhs/

[cdtech]

Thanks for that information synss. I've updated my dont-backup file to include these.

[.exe]

Hey, I just tried to backup my fresh install, and I noticed it was backing up files from other partitions. Is there a proper way to prevent it from doing this, besides excluding /media?

[cdtech]

Hey, I just tried to backup my fresh install, and I noticed it was backing up files from other partitions. Is there a proper way to prevent it from doing this, besides excluding /media?


--exclude=/mnt ?

Really need more information to answer this. What tar commands are you using?

[.exe]

I used the one on the first page. Ok, I think I see the problem: /mnt is empty, and /media is populated with all my drives/devices. So fixing is should be as simple as excluding /media?

[keithrennie]

THANKS As a recovering ex-windows user who finally took the plunge, ditched dual boot and happily reformatted my hard drive, I just spent several precious hours in frustration trying to make sure I didnt lose the results of all my hard work setting up Linux and apps the way I wanted it on my laptop. I couldnt get Keep to back up the system, couldnt burn a backup CD which would reflect all my new changes. - In desperation I turned to Ubuntu Forums, and yours was the first thing I found. You did a great job explaining things so clearly. Just what I needed, answers just about every question I had. I'm sure it's going to work fine - if not I will post again.
.
Now if you guys could just get a good thread going on Google Earth, that would make a perfect week.

[ropenstein]

Speaking of piping, I don't see why you couldn't pipe the output of dd into tar -z to squeeze the air out of it. This is in response to some of the earlier posts asking about a program similar to Ghost, whereby an actual image of the disk is created. By the way, I believe you can also dd /dev/hdb (entire disk) and not just /dev/hdb1 (first partition). On the other hand, can't you use tar on / and create an archive that encompasses data on multiple disks if you are using mount points? I guess each has its particular use depending on your needs.

[sk545]

Since this can be dangerous, I want to know what command to use. I have this in my /etc/fstab:

# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# /dev/hda5
UUID=79d58c5f-8f97-4c14-b5f1-61e3e31c5770 / reiserfs notail 0 1

# /dev/hda6
UUID=fff89e85-f9d0-4cca-987e-279770232f92 /home reiserfs defaults 0 2

# /dev/hda1
UUID=C65CFEAC5CFE9681 /media/hda1 ntfs defaults,nls=utf8,umask=007,gid=46 0 1

# /dev/hda2
UUID=2ABCFBE2BCFBA68B /media/hda2 ntfs defaults,nls=utf8,umask=007,gid=46 0 1

# /dev/hda3
UUID=B965-0200 /media/hda3 vfat defaults,utf8,umask=007,gid=46 0 1

# /dev/hdb1
UUID=674e3c0f-2483-4349-9cef-75ac0d16d002 /media/hdb1 ntfs defaults,nls=utf8,umask=007,gid=46 0 1

# /dev/hda7
UUID=08bc84f1-0b70-4dc6-9c5f-70ddc42d09e2 none swap sw 0 0

/dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0

/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0


I don't want to back up the windows partions (hda1-3) or hard drive 2 (hdb).
Whats the correct command for a backup? Also, if I restore this backup, will it mess up my MBR so I can't login to windows via grub?

And why are my partitions mounted on /media, not /mnt?

[tuxcantfly]

In case anybody's interested in making a bootable backup that can later be booted, restored, or archived, you might want to check out my project at http://lubi.sourceforge.net/bubakup.html . Basically what it does is store a backup in a loopmounted ext3 filesystem and it uses a patched Ubuntu initrd to boot directly from the disk image so that the backup can be booted and restored to an actual partition.

[synss]

Since this can be dangerous, I want to know what command to use. I have this in my /etc/fstab:

# /etc/fstab: static file system information.
# /dev/hda5
UUID=79d58c5f-8f97-4c14-b5f1-61e3e31c5770 / reiserfs notail 0 1
# /dev/hda6
UUID=fff89e85-f9d0-4cca-987e-279770232f92 /home reiserfs defaults 0 2
# /dev/hda1
UUID=C65CFEAC5CFE9681 /media/hda1 ntfs defaults,nls=utf8,umask=007,gid=46 0 1
# /dev/hda2
UUID=2ABCFBE2BCFBA68B /media/hda2 ntfs defaults,nls=utf8,umask=007,gid=46 0 1
# /dev/hda3
UUID=B965-0200 /media/hda3 vfat defaults,utf8,umask=007,gid=46 0 1
# /dev/hdb1
UUID=674e3c0f-2483-4349-9cef-75ac0d16d002 /media/hdb1 ntfs defaults,nls=utf8,umask=007,gid=46 0 1
# /dev/hda7
UUID=08bc84f1-0b70-4dc6-9c5f-70ddc42d09e2 none swap sw 0 0
/dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0


I don't want to back up the windows partions (hda1-3) or hard drive 2 (hdb).
Whats the correct command for a backup?

--exclude=/media and you should be fine

Also, if I restore this backup, will it mess up my MBR so I can't login to windows via grub? You may need to reinstall grub, that's probably all.

And why are my partitions mounted on /media, not /mnt?
FHS: http://www.pathname.com/fhs/

[mixersoft]

Helllo all,

-i let it sit and it eventually gets to the BusyBox v1.1.3 with error:
-/bin/sh: can't access tty; job control turned off
(initramfs) --- (ram drive?)


Anyone have any experience with this?

I have this problem on restore, too. I am going to try again and exclude the /var/run dir, but any help would be appreciated.

[Eicca]

Oh thanks alot! This will come useful! I'll tweak my system after I get it running as much as I can, and when it is stable enough, I'll make backups! :P

(I have to reinstall my whole system like twice in a year, so I think the backupping should be the answer for my problems :P)

[mahasmb]

I'm wondering about file compression. I don't really care for speed, but what's the most I can compress say 30 GB to? Or even 80 GB?

I know someone mentioned they were able to compress 2 GB to 1 GB.

[elinav]

Hi,

Relatively new to Ubuntu. After spending two weeks or so installing various packages I suddenly realized that if my hard disk fails, I have to do everything over. I don't mind the downloading, but I don't even remember all the configurations I made.

Anybody out there have a tool that will save the list pf packages I installed and just the configuration files?

Thanks,

Doron

[Daveski]

what's the most I can compress say 30 GB to? Or even 80 GB?

Compression ratios are a function of the content of your files. If you have 30Gb of already compressed music, videos or pictures (MP3, OGG, MP4, Jpeg etc.) then I doubt it would compress much at all. If you have 30Gb of HTML or text files then I would expect to compress down to as little as 1Gb or even less. Most people have a fair spread of content, so I would always guess at around a 50% decrease (say 15Gb or 40Gb for the full 80Gb of data).

[Phrawm48]

This is a great thread -- given the number of participants, questions, and rebuttals there's an obvious need for this information.

I'll add that I today managed to use Norton Ghost 2003 to image and restore my Feisty EXT3 system.

The trick was the use of the Norton Ghost 2003 program's "-ial" command line switch.

The "-ial" switch is a variation of the "-ia" switch that performs sector-by-sector copying of a source partition; the "-ial" switch limits sector-by-sector copying to Linux partitions whereas the "-ia" switch does not.

However, note first that the "-ial" switch (like the "-ia" switch) requires that the geometries of the source and destination partitions be identical! So the "-ial" solution is perhaps not as complete as one might wish.

Finally, also note that I've only done this once. So I for one don't at this point imagine that I have uncovered The One True Solution to the problem. In other words, I intend to proceed with caution and to periodically verify that these Ghost images/restores are 100% intact and reliable; I urge you to do the same...

Stated another way, I fully intend to keep researching alternatives such as the "Linux only" approach described in this Ubuntu Forum thread.

Nevertheless, it does preliminarily appear that one can indeed use Norton Ghost 2003 in conjunction with the program's "-ial" switch to create Feisty EXT3 Ghost images that can be restored and run.

That's worth knowing, I think...

Cheers & hope this helps,
Ric
SFO

[mannheim]

I don't know if this has been mentioned earlier in the thread, but I think that a command like

tar ... --exclude=/sys ...


will cause tar to exclude any file whose path contains /sys/ anywhere along the way. So it will exclude files like /usr/include/sys/time.h and others that you probably don't want to lose. Likewise, if you use --exlude=/media you will not back up such files as /usr/share/icons/gnome/16x16/stock/media/.

To avoid this, you need to supply the "--anchored" option to tar, I believe. If this is correct, the OP's script should be modified. (It was exactly this sort of uncertainty that made me switch from tar to rsync for backups, because the behavior of rsync seemed to me to better documented and I was more confident of getting what I wanted.)

[protenniser]

Hi,
First thanks for this great utility and I couldn't managed to read all of this thread (I was dying at 12th page :)) If so there is someone who followed up all of it, or have the answers to my questions, thanks a lot.
1-) I have ubuntu feisty fawn installed and have vmware workstation and have a XP in it. If I use the backup and restore method as described in the 1st page, will my XP in that virtual drive still work after recover?
2-) I have another pc, which has windows XP firstly installed in it and after that ubuntu feisty fawn. If I make the backup form ubuntu, will it also include the 2nd os (windows xp that is previously installed?) Or Should I back it up also?
3-) There was a lot discussion on that this method can be used only from live cd, not a running copy of ubuntu (recover process) is it true?
If you can answer these questions I will be glad.
Thanks, regards....

[mahasmb]

Question, would the steps in this how to in any cause the following?


http://ubuntuforums.org/showthread.php?p=3350485#post3350485

I only did the backup part and not restore part.

[bullgr]

in image backup the whole thing is that if you mess up your ubuntu installation and you can't fix it, you can restore it and all will be exactly like the last time you backed up.

with the tag thing if you mess up your ubuntu installation and you can't fix it, you must first install again ubuntu and then restore your data.

the big advantage of image backup is that you avoid the installation issue.

and symantec ghost 10 and later has no problem with the ext2 & ext3 filesystem. i restore ubuntu from symantec ghost 10 image last month after i messed up ubuntu with compiz fusion.
and after one month all works great with no problem.

but i don't like the dependence symantec ghost-winblows and i search for a linux image backup solution and i suggest to do the same.

[wirelessmonkey]

bullgr, et al.

Partimage works much like ghost, but is designed for linux. It works for a large number of filesystems/drive types.

[tech9]

Hi, and welcome to the Heliode guide to successful backing-up and restoring of a Linux system!


1: Backing-up

This was exactly what I was looking for... Many Thanks!
I tested, and it worked!

Cheers!:guitar:

"What should I use to backup my system then?" might you ask. Easy; the same thing you use to backup/compress everything else; TAR. Unlike Windows, Linux doesn't restrict root access to anything, so you can just throw every single file on a partition in a TAR file!

To do this, become root with

sudo su

and go to the root of your filesystem (we use this in our example, but you can go anywhere you want your backup to end up, including remote or removable drives.)

cd /


Now, below is the full command I would use to make a backup of my system:


tar cvpzf backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/sys /

Now, lets explain this a little bit.
The 'tar' part is, obviously, the program we're going to use.

'cvpfz' are the options we give to tar, like 'create archive' (obviously),
'preserve permissions'(to keep the same permissions on everything the same), and 'gzip' to keep the size down.

Next, the name the archive is going to get. backup.tgz in our example.

Next comes the root of the directory we want to backup. Since we want to backup everything; /

Now come the directories we want to exclude. We don't want to backup everything since some dirs aren't very useful to include. Also make sure you don't include the file itself, or else you'll get weird results.
You might also not want to include the /mnt folder if you have other partitions mounted there or you'll end up backing those up too. Also make sure you don't have anything mounted in /media (i.e. don't have any cd's or removable media mounted). Either that or exclude /media.

EDIT : kvidell suggests below we also exclude the /dev directory. I have other evidence that says it is very unwise to do so though.

Well, if the command agrees with you, hit enter (or return, whatever) and sit back&relax. This might take a while.

Afterwards you'll have a file called backup.tgz in the root of your filessytem, which is probably pretty large. Now you can burn it to DVD or move it to another machine, whatever you like!

EDIT2:
At the end of the process you might get a message along the lines of 'tar: Error exit delayed from previous errors' or something, but in most cases you can just ignore that.

Alternatively, you can use Bzip2 to compress your backup. This means higher compression but lower speed. If compression is important to you, just substitute
the 'z' in the command with 'j', and give the backup the right extension.
That would make the command look like this:


tar cvpjf backup.tar.bz2 --exclude=/proc --exclude=/lost+found --exclude=/backup.tar.bz2 --exclude=/mnt --exclude=/sys /


2: Restoring

Warning: Please, for goodness sake, be careful here. If you don't understand what you are doing here you might end up overwriting stuff that is important to you, so please take care!

Well, we'll just continue with our example from the previous chapter; the file backup.tgz in the root of the partition.

Once again, make sure you are root and that you and the backup file are in the root of the filesystem.

One of the beautiful things of Linux is that This'll work even on a running system; no need to screw around with boot-cd's or anything. Of course, if you've rendered your system unbootable you might have no choice but to use a live-cd, but the results are the same. You can even remove every single file of a Linux system while it is running with one command. I'm not giving you that command though! ;-)

Well, back on-topic.
This is the command that I would use:


tar xvpfz backup.tgz -C /


Or if you used bz2;


tar xvpfj backup.tar.bz2 -C /


WARNING: this will overwrite every single file on your partition with the one in the archive!

Just hit enter/return/your brother/whatever and watch the fireworks. Again, this might take a while. When it is done, you have a fully restored Ubuntu system! Just make sure that, before you do anything else, you re-create the directories you excluded:

mkdir proc
mkdir lost+found
mkdir mnt
mkdir sys
etc...


And when you reboot, everything should be the way it was when you made the backup!

2.1: GRUB restore
Now, if you want to move your system to a new harddisk or if you did something nasty to your GRUB (like, say, install Windows), You'll also need to reinstall GRUB.
There are several very good howto's on how to do that here on this forum, so i'm not going to reinvent the wheel. Instead, take a look here:

http://www.ubuntuforums.org/showthread.php?t=24113&highlight=grub+restore

There are a couple of methods proposed. I personally recommend the second one, posted by remmelt, since that has always worked for me.


Well that's it! I hope it was helpful!
As always, any feedback is appreciated!

This is exactly what I was looking for... Many Thanks!
I tested this, and it works! :guitar:

[gimpguy2000]

Just wondering, after two tries, the original method posted worked to back up and restore but with a slight problem.... Libs and other synaptic or otherwise software. All my software installed did not get written to my restore disk, nothing I put in on my own like Lexmark drivers, some libs, or anything of that sort. It did restore most everything else though. It did leave out the newer thunderbird install and things like that.

So is this just a back up of "approved" Ubuntu installs on the system? This is how it seems to me but thought i'd ask.

TX

[mr tim esquire]

Can i use this method to backup and reload my windows windows partition too? er.... how?
thanks!

[zurk]

i use debian etch.
zurk@DarkSMP:/$ cat backup
tar cvpjf - --one-file-system --exclude=/proc --exclude=/media --exclude=/var/lock --exclude=/var/run --exclude=/tmp --exclude=/var/tmp --exclude=/dev --exclude=/lost+found --exclude=/backup.tbz2.* --exclude=/mnt --exclude=/sys / | split -b 1024m - backup.tbz.


zurk@DarkSMP:/$ cat restore
cat backup.tbz.* | tar xvpfj - -C /
mkdir /proc
mkdir /lost+found
mkdir /mnt
mkdir /sys
mkdir /dev
mkdir /media
mkdir /var/lock
mkdir /var/run
mkdir /tmp
mkdir /var/tmp
chmod 777 /tmp

[mahasmb]

Question, would the steps in this how to in any cause the following?


http://ubuntuforums.org/showthread.php?p=3350485#post3350485

I only did the backup part and not restore part.

I ask again.

It seems I have lost my whole Ubuntu partition on my laptop with no means of recovering all my hard work AND also without a reason as to why it all happened.

I followed this HOW TO and did nothing else to my system that could possibly mess it up.

Yet, I can't access my Ubuntu partition in any way nor even the files within it. Everything is telling me one thing, my Ubuntu partition is an "unknown" partition.

COULD THIS BACKUP PROCEDURE HAVE POSSIBLY CAUSED MY PROBLEMS?

I need to know.

Please.

[tech9]

This is exactly what I was looking for... Many Thanks!
I tested this, and it works! :guitar:


I take this back...

See link....
http://ubuntuforums.org/showthread.php?t=553115

[onetb]

I've read over the thread and I am a bit confused why I cannot get this method to work. We had to send a laptop back to the manufacturer and they told us we would be charged is anything other than windows was on it. BS
Any way, when it came back, I tried to installed Ubuntu, then un tar my backup as root from the root directory, but when I restart, I get the following error
/bin/sh: can't access tty
Anyone know what I've done wrong?

[nox-Hand]

Hey!

I used your wonderful guide to backup AND restore my drive. Lovely stuff, I have Ubuntu up and running again, yet I have errors.



Now, I should probably tell you HOW I broke my system.

Imagine, if one were to run VMware sessions straight from drive in non-snapshot mode. Then imagine one was curious as to what'd happen if from INSIDE Ubuntu you load up Ubuntu again. In essence, the same file system, kernel, loggers, etc were all being used at the same time. Obviously it was going to break :lolflag:


Well, after reboot, it was all read-only. I opted to restore my backup (which I had luckily just made five hours ago! :) ) so it'd all be well and dandy.


However, now when I boot I get a series of messages telling me this:
ld_static: cannot open output file /lib/modules/2.6.20-16-generic/volatile/nvidia_new.ko : Readonly file system.

I get these with all modules in the volatile folder.

This means, I have NO NET on my laptop. Not even cable.


I could use a bit of help on fixing this one, the irc channel doesn't seem to know what is wrong :)

Cheers!

[shane2peru]

Well after running many backups and even writing a script to backup my system on a monthly basis, this system is for the birds! It has left me hanging so many times it is not even funny. I think my restore ratio is about 20% of the time that it has actually worked over a 2 year period! That is sorry. I have a separate partition for Home and only backup my / with this system. I tried a complete backup, including my home directory and went to restore after making some changes that messed up my system, and it was unbootable! Re-installed and then restored an earlier backup of just the filesystem and still unbootable!!! ARGHHH, why do I bother with this system of backing up!!! I'm off to find a new backup method that is more secure, more stable, and more reliable!

Shane

EDIT: Ok, I determined that the restore problem stems from the fact that I restore my Grub menu list, and that list is now run with the UUID method which changes if you have to do a re-install. This is unacceptable for a backup procedure and restore procedure. If you use this method, you cannot re-install and then restore without editing your Grub Menu list to be correct. I'm going to the partimage method described here. (http://ubuntuforums.org/showthread.php?t=287522) And plan on installing a second Linux installation on my hdd to use instead of a live CD as it boots much faster, and will run much faster. Looking at PuppyLinux or Xubuntu, so it doesn't require much space. I have used partimage before and find it to be a very good way of backing up and restoring.

[KrisWillis]

This command is working great for me, I have chucked it into a shell script and all is well - I thought I'd automate the whole process a little more and execute the script with cron.

This is where I'm having issues - When manually executing the script the archive is created perfectly at about 9GB, though when cron executes the script I'm left with a file that is about 8MB and only contains a very small portion of my data. Cron is scheduled to execute at 20:05, at this time, no users are logged in and the machine is left on the login screen.

Any ideas of what could be the reason for this failure?

My script looks like:
#!/bin/bash
tar cvpzf /media/backup/backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/media --exclude=/mnt --exclude=/sys /

My crontab (root):
5 20 * * * /home/kris/backup.sh

[shane2peru]

This command is working great for me, I have chucked it into a shell script and all is well - I thought I'd automate the whole process a little more and execute the script with cron.

This is where I'm having issues - When manually executing the script the archive is created perfectly at about 9GB, though when cron executes the script I'm left with a file that is about 8MB and only contains a very small portion of my data. Cron is scheduled to execute at 20:05, at this time, no users are logged in and the machine is left on the login screen.

Any ideas of what could be the reason for this failure?

My script looks like:
#!/bin/bash
tar cvpzf /media/backup/backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/media --exclude=/mnt --exclude=/sys /

My crontab (root):
5 20 * * * /home/kris/backup.sh
Your script looks fine, except I think you need to change tar to /bin/tar to show the exact location of the tar command. run this: which tar to find where your tar is located, it should be the same though. That should work, that is what I have used for a long time in a very similar script.
On a second note it really doesn't matter if someone is logged in or not, the script will still run since it is root, however it may slow the computer down a little.

Shane

[KrisWillis]

Thanks Shane, I'll give that a go...

[KrisWillis]

Ok, that didn't fix my problem - But I have figured out what did, I turned off the verbose switch and my file appears to be generating properly now. Thanks guys.

[mixersoft]

Followed the instructions to backup, but:

-after untarring and creating the necessary folders i rebooted.
-I then received the grub text (normal)
-Was greeted with the splash and got stuck there
-i let it sit and it eventually gets to the BusyBox v1.1.3 with error:

/bin/sh: can't access tty; job control turned off
(initramfs) --- (ram drive?)

Also tried to reinstall GRUB, but still got the same response. Any thoughts?

[shane2peru]

Followed the instructions to backup, but:

-after untarring and creating the necessary folders i rebooted.
-I then received the grub text (normal)
-Was greeted with the splash and got stuck there
-i let it sit and it eventually gets to the BusyBox v1.1.3 with error:

/bin/sh: can't access tty; job control turned off
(initramfs) --- (ram drive?)

Also tried to reinstall GRUB, but still got the same response. Any thoughts?

Yes your problem is UUID is in your menu list and your new UUID is not the same as your restored system. You need to go in and edit your menu.lst. It is almost impossible to recover without doing this. You will need to boot into a LiveCD then run sudo fdisk -l and look for the partition that has your Ubuntu on it. Then mount it with sudo mkdir /media/Ubuntu and then (You will need to replace the red part with the proper partition name)sudo mount /dev/hda1 /media/Ubuntu
then sudo cp /media/Ubuntu/boot/grub/menu.lst /media/Ubuntu/boot/grub/menu.lst_backup
and then finally edit your grub like this:sudo gedit /media/Ubuntu/boot/grub/menu.lst Now where you see
root=UUID=930jks0dka-d-sljhasd - or some long number like this You will need to replace the red part with: /dev/hda1You need to put the proper partition name into the green part.
That is exactly why I don't recommend this backup method. If you have any questions feel free to post back.

Shane

[mixersoft]

worked like a charm. Many thanks!

[Phrawm48]

The length and convolutions of this thread illustrate that Ubuntu's disk and files backup tools and processes remain overly complicated for many users (there's that word again).

First, observe that the word "backup" has no finite meaning. That's led to a lot of confusion between those who use the word to mean backing up files and those who use the word to mean backing up disks.

As for me, after extensive research and multiple "Linux moments", I retain a Windows partition on my hard drive so that I can continue to use Norton Ghost 2003. As I mentioned in an earlier post on this thread, I've found that Ghost 2003's -ial switch enables me to image and restore my Feisty system's EXT3 boot and root partitions.

I'll conclude by observing that for most users backing up a Linux system remains an experimental exercise. The problem with that is that Linux will quickly educate a user that experiments should only be performed after one has safely backed up up one's system, lest a failed experiment render the system unusable. And 'round and 'round we go...

Cheers & hope this helps,
Ric
SFO

[shane2peru]

I agree, backing up is an experimental thing. Hence I keep my /home partition separate with all my data. Now the best backup solution I have ever used is partimage. And I just came across this link (http://www.swerdna.net.au/linhowtorescuecd.html) that seems like the most perfect solution for me. I'm going to give it a try. I have used partimage many times and really like it, however I don't like being tied to a CD for my backup - restore solutions so that link is going to be the best solution if it works!

Shane

[snyper82]

I followed this method, made an oops...

chmod -R 777 /

I did this as root, now I can't restore my system at all, (it is a server)
I need to know if there is a way to restor the file system permissions before I reload the entire thing.

[donnguna]

:lolflag:

Yup,... why we've to pay while so many good people on the net who to share

[Nightwalker07]

it was great to read this superb guide to backing-up/restoring. The fact that linux allows your to access or delete any file (even if in use) was very impressive. As with any new backup method I come across, im naturally worried about its integrity, after-all if something does fail you the last thing you want is to turn to your backup and not have it work, so I have backed-up using your method and im interested in testing the restore out just to verify it works ok.

I have spent a week struggling with getting a MythTV-box working, with hardware problems etc, but now I've got it all working and stable! Worked hard to get it to that point so I want a reliable backup method to use so I can revert back to this working point. I have made the backup.tgz and would like to test it. What I would like to ask is it hardware-specific do you think? how would it deal with new hardware? ie. my MythTV-box is a desktop, and im wondering if I could test restoring the backup on a laptop, would it matter?

Also; regards: "You can even remove every single file of a Linux system while it is running with one command." - this intrigued me!.. is this really true? how would that work if you were in a desktop environment? Surely thats not possible? (I would love to be proved wrong!) lol. I tried this yesterday on a old ubuntu(7.04)-laptop, as root I sat at the root of the filesystem and executed my best guess at removing all files:
rm -rf *Starting ticking away and removing all the files on the system, but had a ton of access issues due to 'files in use' which was exactly what I thought wasnt supposed to happen!? Thats why I doubted the reliability of this info, just wondered if you could clear up a few things with me.

Thank you for your time and help, much appreciated.

[Nightwalker07]

Anyway, I've had a test-laptop to test this back & restore technique on. Dealing with ubuntu 7.04 I have followed the method precisely. I have tried it three times, the first time it seemed to have work and the last two times didnt work, they both produced the same result; a non booting system. It gets to the ubuntu OS loading screen loads about 1% of the orange bar and freezes, if I leave it for a while it then shows a screen saying:

BusyBox v1.1.3 (Debian 1:1.1.3-3ubuntu3) Built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/sh: can't access tty; job control turned off
(initramfs)

Why did the method work once? and why am I now getting these results that aren't working? As far as I can tell I have done a controlled textbook test of this backup/restore method. Any help greatly appreciated.

[shane2peru]

I don't remember the exact error message, but I believe this is the same error that I have seen many times. The problem is that Grub is not correct, or better stated your menu.lst is not correct, or to be more precise, your system changed the UUID numbers and can no longer locate the partition to boot from. When grub comes up, select the one you want to boot, and hit the 'e' key, this will allow you to temporarily edit that line for booting purposes. I'm assuming that you know what your hdd numbers are and the one that you are booting from. Replace the UUID=xkdaoi39829ajkd90 or whatever that very large group of alphanumeric string is with /dev/sda1 or /dev/hda1 or whatever your hdd partition that you want to boot into should be and it should boot right up. Once inside you need to fix your grub menu.lst again as that only allows you to boot into the system. This is the reason I don't use this backup method any longer. It is not reliable.

Shane

[Nightwalker07]

Thanks for your reply. I understand it could mess up GRUB a litte, I see what you mean, but its a test-laptop and I only have 1 HDD and no other hardware changes took place during the tests, so should this still have effected grub? Seeing as it didnt work for me im guessing the answer is going to be yes. Weird how it worked once and not the two other times.

There's only one thing I ask of a backup method and thats reliability! So do you think if you were willing to manually fix the grub boot-loader each time this would be a reliable method? Or is this method just worthless altogether?

[tech9]

=D>Anyway, I've had a test-laptop to test this back & restore technique on. Dealing with ubuntu 7.04 I have followed the method precisely. I have tried it three times, the first time it seemed to have work and the last two times didnt work, they both produced the same result; a non booting system. It gets to the ubuntu OS loading screen loads about 1% of the orange bar and freezes, if I leave it for a while it then shows a screen saying:

BusyBox v1.1.3 (Debian 1:1.1.3-3ubuntu3) Built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/sh: can't access tty; job control turned off
(initramfs)

Why did the method work once? and why am I now getting these results that aren't working? As far as I can tell I have done a controlled textbook test of this backup/restore method. Any help greatly appreciated.

I had the exact same problem make sure you exclude the /boot directory and you won't get this error anymore after restoring

[tech9]

Thanks for your reply. I understand it could mess up GRUB a litte, I see what you mean, but its a test-laptop and I only have 1 HDD and no other hardware changes took place during the tests, so should this still have effected grub? Seeing as it didnt work for me im guessing the answer is going to be yes. Weird how it worked once and not the two other times.

There's only one thing I ask of a backup method and thats reliability! So do you think if you were willing to manually fix the grub boot-loader each time this would be a reliable method? Or is this method just worthless altogether?

See this link for more info...

http://ubuntuforums.org/showthread.php?t=564836

[Nightwalker07]

=D>

I had the exact same problem make sure you exclude the /boot directory and you won't get this error anymore after restoringooo, that sounds superb! Hope it works, I'll give it a go tomorrow, Thanks!

[shane2peru]

See this link for more info...

http://ubuntuforums.org/showthread.php?t=564836

THANK YOU FOR MAKING A NEW THREAD!!! This thread has gotten way to long, and I don't think the original poster is even active. By the way, that is a very good tip, and I will include it in my new backup script that I use. Thanks!

ooo, that sounds superb! Hope it works, I'll give it a go tomorrow, Thanks!

Let us know how it goes for you!!! We are waiting to hear!

Shane

[indosupremacy]

bro...i have done exactly like what this tread said about back up...but,while i'm trying to restore it,it seem that i have trouble ....
after i try to make mkdir other directory i excluded etc: /proc ..., /lost+found...and other directory that i have excluded .it said...the directory u want is already exist...
any help...?
thanks broooo

[tech9]

ooo, that sounds superb! Hope it works, I'll give it a go tomorrow, Thanks!

no problem

[tech9]

bro...i have done exactly like what this tread said about back up...but,while i'm trying to restore it,it seem that i have trouble ....
after i try to make mkdir other directory i excluded etc: /proc ..., /lost+found...and other directory that i have excluded .it said...the directory u want is already exist...
any help...?
thanks broooo

are you making sure that you are using "--exclude"
That should work

[Nightwalker07]

Let us know how it goes for you!!! We are waiting to hear!
WARNING - Probably best randomers don't come along and repeat this... there's no need to wipe your system if its working, especially not if the backup fails afterwards. -PriceChild
My exact testing steps:
1. Freshly Installed 7.10 (Gusty-Gibbon-i386-desktop) on a Laptop.
2. Copied test data to system (600MB / 1,400 files) to both my home dir and desktop. And then installed KTron application (just a small-game to test if it works later).
3. Opened the Terminal and ran the following: cd /
sudo tar cvpzf backup.tgz --exclude=/etc/fstab --exclude=/boot --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/sys /4. I then inserted a USB flash-drive & backed up the 'backup.tgz' and removed the drive.
5. As part of the test I intended to serverly damage the filesystem, so again, still at root on the terminal I ran the following to destroy the system (WARNING: Dont do this if you value the system or it's data):sudo rm -fr *6. Manually powered-off the system.
7. Powered on system and booted with the ubuntu-7.10 disc, and performed a new, fresh installation.
8. Once booted into my new installation I then inserted my USB flash-drive and proceeded to extract/restore my system/files back onto the system drive:
sudo tar xvpfz /media/USB-DRIVE/backup.tgz -C /
9. Restarted the system and all was restored fine. System & files restored and working. :)

[indosupremacy]

:popcorn: for u my brooooo
now it actually work....
my fault on the last tried ist that i wasn't umount /media/sdax....
after i umount it...there's no problem at all....

offcourse theres trouble after kernel upgrade...but this thing make me back healthy again..
this thing is the simplest thing to back up....:)

[Bubs]

ok. I have separate root and /home partitions I made the backup.tgz but I made it in /home. si instead of
sudo tar cvpzf backup.tgz --exclude=/etc/fstab --exclude=/boot --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/sys /

I ran sudo tar cvpzf /home/backup.tgz --exclude=/etc/fstab --exclude=/boot --exclude=/proc --exclude=/lost+found --exclude=/home/backup.tgz --exclude=/mnt --exclude=/sys /

but the problem is that the file generated is bigger that the / partition.

so do I extract home and save it as another file. then add it after I restore the backup? how do the separate partitions work when backing up? I'm pretty new to backing up and restoring stuff...

[Bubs]

nevermind. it worked I just extracted from home to root and it worked fine!

Thank you for the tutorial! You saved my ***. My gutsy upgrade failed in the middle of installing and I backed up with your method right before I started that step.

[Nightwalker07]

Glad you got it sorted and the backup worked for you!

[Pay87]

Why do you --exclude=/etc/fstab ?
Is fstab generated automatically, I didn't think so.. ?

[mannheim]

9. Restarted the system and all was restored fine. System & files restored and working. :)

To some extent you were lucky. If you had installed gcc instead of KTron in your test, then you would have found that the "--exclude /sys" option in the backup script leads to the exclusion of such things as /usr/included/sys and everything else with "/sys" somewhere in the path. After your restore, gcc would have been broken because of missing header files. I posted earlier about this.

You need to add the --anchored option to tar to avoid this behavior.

(But even then, I've been bitten so many times by the unexpected behavior of tar's various options, that I've stopped using it for my backups.)

[Nightwalker07]

...You need to add the --anchored option to tar to avoid this behavior.

(But even then, I've been bitten so many times by the unexpected behavior of tar's various options, that I've stopped using it for my backups.)Interesting, thanks. Im new to tar myself, I'll check that out.

[kevinfitch83]

I have a dual-boot (Ubuntu and Windows XP) could I use this same method to back-up my copy of windows?

If so what dirs would a need to include?

[tech9]

:popcorn: for u my brooooo
now it actually work....
my fault on the last tried ist that i wasn't umount /media/sdax....
after i umount it...there's no problem at all....

offcourse theres trouble after kernel upgrade...but this thing make me back healthy again..
this thing is the simplest thing to back up....:)

Glad to hear you got it working

[tech9]

Why do you --exclude=/etc/fstab ?
Is fstab generated automatically, I didn't think so.. ?

yes... it is and if the UUID do not match up, then you will have problems booting that partition

[bluepowder7]

ha! this looks like the solution i should have used before mistakenly blowing away my work files from the /home partition. i never ever made backups, at least not since the days i had an external tape drive (which held a whopping 100megs i think).

time to start a "back up your crap" routine, i guess, and this one looks very well documented! thanks!!!

[Nightwalker07]

To some extent you were lucky. If you had installed gcc instead of KTron in your test, then you would have found that the "--exclude /sys" option in the backup script leads to the exclusion of such things as /usr/included/sys and everything else with "/sys" somewhere in the path. After your restore, gcc would have been broken because of missing header files. I posted earlier about this.

You need to add the --anchored option to tar to avoid this behavior.

(But even then, I've been bitten so many times by the unexpected behavior of tar's various options, that I've stopped using it for my backups.)This comment is the only thing thats got me worried about this backup method. I mean with a backup method the sole thing you need to know is that its integrity is good, can you shed any more light on what might be kept under /sys and what it might break? Fair enough I have tested this technique on fairly fresh installations, but im worried about the likes of my well-matured MythTV box, I need to know that the backups im doing of that are working if the day ever comes when I need to rely on them.

Is the '-anchored option' regards the tar following links in subdir's under sys? any info would be appreciated.

[tech9]

This comment is the only thing thats got me worried about this backup method. I mean with a backup method the sole thing you need to know is that its integrity is good, can you shed any more light on what might be kept under /sys and what it might break? Fair enough I have tested this technique on fairly fresh installations, but im worried about the likes of my well-matured MythTV box, I need to know that the backups im doing of that are working if the day ever comes when I need to rely on them.

Is the '-anchored option' regards the tar following links in subdir's under sys? any info would be appreciated.

Hi Nightwalker07,

You could try installing virtual box and install a test VM, then restore from there and see if it works.

[Nightwalker07]

Hi Nightwalker07,

You could try installing virtual box and install a test VM, then restore from there and see if it works.Funny you mention that I was using Gusty under a VMware Server on XP and the thought crossed my mind, but I didnt want to do it under those conditions because gusty dosent run well under vmware atm, but I will get a virtualbox going on my ubuntu-laptop, I know that runs smoother! :) I'll mature the VM and fill it with plenty of crap! Thanks again.

[tech9]

Funny you mention that I was using Gusty under a VMware Server on XP and the thought crossed my mind, but I didnt want to do it under those conditions because gusty dosent run well under vmware atm, but I will get a virtualbox going on my ubuntu-laptop, I know that runs smoother! :) I'll mature the VM and fill it with plenty of crap! Thanks again.

NP :)
Take Care

[mannheim]

This comment is the only thing thats got me worried about this backup method. I mean with a backup method the sole thing you need to know is that its integrity is good, can you shed any more light on what might be kept under /sys and what it might break? Fair enough I have tested this technique on fairly fresh installations, but im worried about the likes of my well-matured MythTV box, I need to know that the backups im doing of that are working if the day ever comes when I need to rely on them.

Is the '-anchored option' regards the tar following links in subdir's under sys? any info would be appreciated.

Well I take back some of what I wrote! Here, however, is what "man tar" says:

--anchored
force exclusion patterns to match initial subsequences
--no-anchored
allow exclusion patterns to match any substring (the default)

I interpret this to mean that "--exclude=/sys" will by default exclude anything whose path contains the string "/sys", such as "/usr/include/sys/foo", unless you supply the --anchored option. I just test this hypothesis, and it is wrong. If you do "--exclude=sys" then you will indeed exclude things like "/usr/include/sys/foo", but it seems "--exclude=/sys" (with the leading forward slash) is treated differently.

So my earlier post was wrong. I do stand by my statement that the options and documentation for tar are so hairy that it is almost impossible to know what it will do, without just testing or reading the source.

[Nightwalker07]

Well it was worth bringing up & thinking about, thanks for testing it out and letting us know. Im just about to get a ubuntu-7.10 installation going in VirtualBox, im gona try mature the installation with plenty of crap and test this backup method on that and see if it works or breaks stuff.

[mnml]

Does anyone know how to do the backup via ssh?
I would like to get the Backup of my serv on my local hd.
thanks

[mannheim]

Does anyone know how to do the backup via ssh?
I would like to get the Backup of my serv on my local hd.
thanks

I think you can do (on the local machine)
~$ ssh username@server.somewhere.com 'tar -cz /stuff' > /local/path/to/backup.tar.gz
to backup the directory /stuff from you remote machine to your local machine. Or you could use rsync.

[mnml]

thanks a lot, yep rsync might be a better solution : >

[ephman]

hi,

so i have a bit of a concern that i might be way off on, and your help figuring this out would be appreciated. ok i have 4 partitions on my drive. 1 for xp, a boot partition, an encrypted swap, and encrypted /root. i'm getting a new drive for my machine so i will have to remake the encrypted partitions, reinstall gutsy, and then restore using this method. i have a feeling things aren't going to go smoothly. maybe a grub boot problem after i restore not finding the gutsy partition, maybe some uid= issues? any advice or experiences with this?

thanks for the bandwidth,

ephman

[vietunion]

I have trouble ( sorry for my english )

I have install Ubuntu 7.10 ( Clear ), and i use this tutorial for backup ( call backup.tgz ), and then i install some app ( Amarok, KiBa-Dock, FileZilla...etc ) and backup again ( call backup2.tgz ), all files backup in root directory

I want restore PC to 7.10 ( clear ) and i use this command
tar xvpfz backup.tgz -C /

When i restart, Ubuntu not clear , it still have Amarock, Kiba-doc....etc


I want my PC clear ( never install any app ), please help me

Very sorry for my english

Thanks much

[mnml]

it's just unziping over it

[Nightwalker07]

Yeah, its just extracting the backup, so any additional files you put on after the backup will stay there. To do a clean restore, blank the filesystem, format it etc. and then either reinstall the OS and then extract the archive over the blank OS. Or you could do it from the live-boot of your ubuntu disc, a little tricker because you'll need to mount your HDD-disk and then point it to extract in slightly different place.

[tech9]

hi,

so i have a bit of a concern that i might be way off on, and your help figuring this out would be appreciated. ok i have 4 partitions on my drive. 1 for xp, a boot partition, an encrypted swap, and encrypted /root. i'm getting a new drive for my machine so i will have to remake the encrypted partitions, reinstall gutsy, and then restore using this method. i have a feeling things aren't going to go smoothly. maybe a grub boot problem after i restore not finding the gutsy partition, maybe some uid= issues? any advice or experiences with this?

thanks for the bandwidth,

ephman

you shouldn't have any issues if you exclude boot and fstab

[tech9]

I have trouble ( sorry for my english )

I have install Ubuntu 7.10 ( Clear ), and i use this tutorial for backup ( call backup.tgz ), and then i install some app ( Amarok, KiBa-Dock, FileZilla...etc ) and backup again ( call backup2.tgz ), all files backup in root directory

I want restore PC to 7.10 ( clear ) and i use this command
tar xvpfz backup.tgz -C /

When i restart, Ubuntu not clear , it still have Amarock, Kiba-doc....etc


I want my PC clear ( never install any app ), please help me

Very sorry for my english

Thanks much

uninstall any apps you don't want then take the backup... otherwise backup all of your data and perform a clean install, then restore just data :example home dir

[ephman]

you shouldn't have any issues if you exclude boot and fstab

hi,

i wish that was the case. however, just as i suspected the restore did not go so well. i never backup the /boot, and then fixed the fstab file. however on the reboot, not only did i get a grub "error 20", but when i fixed that error, and started to boot up i got another error. just as i originally suspected i get a uuid related error. the error points to the /dev/disk/by-uuid directory and says that the process is waiting for the root partition. has anybody actually restored using this backup method to a newly created encrypted partition? any advice would be great thanks.

ephman

[vietunion]

Thanks Nightwalker07 & tech9

[mdpalow]

Great How-To.

Can I make a suggestion?

Can you add a piece in the RESTORE section that gives a step-by-step on how to restore when something really bad happens and you can't boot the system. Restoring is easy when the system is up and running, but that's only 50% of the equation.

I screwed up my system and couldn't get back in. Had to use the Live Disk. However, I couldn't see my drives.

Tell us how to boot up and get to a command line, so we can do your restore. :)

Will you please private message me at 'mdpalow' if you make this addition?

thanks in advance..

[tech9]

hi,

i wish that was the case. however, just as i suspected the restore did not go so well. i never backup the /boot, and then fixed the fstab file. however on the reboot, not only did i get a grub "error 20", but when i fixed that error, and started to boot up i got another error. just as i originally suspected i get a uuid related error. the error points to the /dev/disk/by-uuid directory and says that the process is waiting for the root partition. has anybody actually restored using this backup method to a newly created encrypted partition? any advice would be great thanks.

ephman

that maybe the problem... may I ask why your partition is encrypted?

[tech9]

Great How-To.

Can I make a suggestion?

Can you add a piece in the RESTORE section that gives a step-by-step on how to restore when something really bad happens and you can't boot the system. Restoring is easy when the system is up and running, but that's only 50% of the equation.

I screwed up my system and couldn't get back in. Had to use the Live Disk. However, I couldn't see my drives.

Tell us how to boot up and get to a command line, so we can do your restore. :)

Will you please private message me at 'mdpalow' if you make this addition?

thanks in advance..

see this link...
http://ubuntuforums.org/showthread.php?t=564836

The idea here is create this backup... store it to an external disk source such as a networked PC or portable HD... (example: samba share) then when you need to get to a prompt - just rebuild your PC with a clean copy of ubuntu - and finally restore your backup from your samba share or portable HD.

Hope this helps!

T9:cool:

[tech9]

Thanks Nightwalker07 & tech9

Your Welcome:)

[ephman]

that maybe the problem... may I ask why your partition is encrypted?

both my root and swap partitions are encrypted.

thanks,

ephman

[markdjones82]

I just tried to open one of my archives and it says:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error exit delayed from previous errors


Any ideas? Also, can this be run in cron? if so do we need to put sudo in the command inside the script or will it run as root?

[markdjones82]

Anyone? I'd really like to make sure I am backing my data up properly.

[geekinput]

Can the backup/restore be saved on a windows network folder? I know you can mount windows network folders on linux. :guitar:


Hi, and welcome to the Heliode guide to successful backing-up and restoring of a Linux system!

Most of you have probably used Windows before you started using Ubuntu. During that time you might have needed to backup and restore your system. For Windows you would need proprietary software for which you would have to reboot your machine and boot into a special environment in which you could perform the backing-up/restoring (programs like Norton Ghost).
During that time you might have wondered why it wasn't possible to just add the whole c:\ to a big zip-file. This is impossible because in Windows, there are lots of files you can't copy or overwrite while they are being used, and therefore you needed specialized software to handle this.

Well, I'm here to tell you that those things, just like rebooting (http://www.ubuntuforums.org/showthread.php?t=34629), are Windows CrazyThings (tm). There's no need to use programs like Ghost to create backups of your Ubuntu system (or any Linux system, for that matter). In fact; using Ghost might be a very bad idea if you are using anything but ext2. Ext3, the default Ubuntu partition, is seen by Ghost as a damaged ext2 partition and does a very good job at screwing up your data.

1: Backing-up

"What should I use to backup my system then?" might you ask. Easy; the same thing you use to backup/compress everything else; TAR. Unlike Windows, Linux doesn't restrict root access to anything, so you can just throw every single file on a partition in a TAR file!

To do this, become root with

sudo su

and go to the root of your filesystem (we use this in our example, but you can go anywhere you want your backup to end up, including remote or removable drives.)

cd /


Now, below is the full command I would use to make a backup of my system:


tar cvpzf backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/sys /

Now, lets explain this a little bit.
The 'tar' part is, obviously, the program we're going to use.

'cvpfz' are the options we give to tar, like 'create archive' (obviously),
'preserve permissions'(to keep the same permissions on everything the same), and 'gzip' to keep the size down.

Next, the name the archive is going to get. backup.tgz in our example.

Next comes the root of the directory we want to backup. Since we want to backup everything; /

Now come the directories we want to exclude. We don't want to backup everything since some dirs aren't very useful to include. Also make sure you don't include the file itself, or else you'll get weird results.
You might also not want to include the /mnt folder if you have other partitions mounted there or you'll end up backing those up too. Also make sure you don't have anything mounted in /media (i.e. don't have any cd's or removable media mounted). Either that or exclude /media.

EDIT : kvidell suggests below we also exclude the /dev directory. I have other evidence that says it is very unwise to do so though.

Well, if the command agrees with you, hit enter (or return, whatever) and sit back&relax. This might take a while.

Afterwards you'll have a file called backup.tgz in the root of your filessytem, which is probably pretty large. Now you can burn it to DVD or move it to another machine, whatever you like!

EDIT2:
At the end of the process you might get a message along the lines of 'tar: Error exit delayed from previous errors' or something, but in most cases you can just ignore that.

Alternatively, you can use Bzip2 to compress your backup. This means higher compression but lower speed. If compression is important to you, just substitute
the 'z' in the command with 'j', and give the backup the right extension.
That would make the command look like this:


tar cvpjf backup.tar.bz2 --exclude=/proc --exclude=/lost+found --exclude=/backup.tar.bz2 --exclude=/mnt --exclude=/sys /


2: Restoring

Warning: Please, for goodness sake, be careful here. If you don't understand what you are doing here you might end up overwriting stuff that is important to you, so please take care!

Well, we'll just continue with our example from the previous chapter; the file backup.tgz in the root of the partition.

Once again, make sure you are root and that you and the backup file are in the root of the filesystem.

One of the beautiful things of Linux is that This'll work even on a running system; no need to screw around with boot-cd's or anything. Of course, if you've rendered your system unbootable you might have no choice but to use a live-cd, but the results are the same. You can even remove every single file of a Linux system while it is running with one command. I'm not giving you that command though! ;-)

Well, back on-topic.
This is the command that I would use:


tar xvpfz backup.tgz -C /


Or if you used bz2;


tar xvpfj backup.tar.bz2 -C /


WARNING: this will overwrite every single file on your partition with the one in the archive!

Just hit enter/return/your brother/whatever and watch the fireworks. Again, this might take a while. When it is done, you have a fully restored Ubuntu system! Just make sure that, before you do anything else, you re-create the directories you excluded:

mkdir proc
mkdir lost+found
mkdir mnt
mkdir sys
etc...


And when you reboot, everything should be the way it was when you made the backup!

2.1: GRUB restore
Now, if you want to move your system to a new harddisk or if you did something nasty to your GRUB (like, say, install Windows), You'll also need to reinstall GRUB.
There are several very good howto's on how to do that here on this forum, so i'm not going to reinvent the wheel. Instead, take a look here:

http://www.ubuntuforums.org/showthread.php?t=24113&highlight=grub+restore

There are a couple of methods proposed. I personally recommend the second one, posted by remmelt, since that has always worked for me.


Well that's it! I hope it was helpful!
As always, any feedback is appreciated!

[tech9]

both my root and swap partitions are encrypted.

thanks,

ephman

I know this is not recommended, but have you tried backing up as root?

[ephman]

I know this is not recommended, but have you tried backing up as root?
hold on i'm confused... i actually do back everything up as root, and then restore as root. am i missing something? (so glad i was able to pull my home directory out of the archive... whew i'm alive)

thanks,
ephman

[tech9]

hold on i'm confused... i actually do back everything up as root, and then restore as root. am i missing something? (so glad i was able to pull my home directory out of the archive... whew i'm alive)

thanks,
ephman

you can back everything up as user by using root privilege sudo -i

[thelatinist]

Thanks for the how-to. Now I have another question, though:

I am trying to create a script to run in cron which will automate weekly backups using this method, name them using the date in the filenames, and delete any backup files older than 30 days. This is what I have:

#!/bin/sh

tar cvpzf /path/to/backup-$(date +"%b-%d-%y").tgz --exclude=/proc --exclude=/lost+found --exclude=/mnt --exclude=/media --exclude=/sys --exclude=/home /

find /path/to/backup/*.tgz -mtime +30 -exec rm {} \;

exit

I'm new to shell scripting. If I schedule this in crontab as root, will it do what I'm expecting it to?

[myharshdesigner]

thanks dear :)

[jonasl]

How do i create the backup.tgz file directly on another hard drive that i have mounted?

[mdpalow]

How do i create the backup.tgz file directly on another hard drive that i have mounted?

cd to the location you want the back-up file to be placed and then run your tar command.

example:

cd /media/some_disk_name

then run your tar command

[decideci]

:'(
i used the method described in the first posting to save my ubuntu 7.10 settings and all before start fiddling with the ATI drivers from hell... i was not able to get either the binary drivers NOR the open source version to work correctly with my radeon 9600. no direct rendering. slowslow desktop. bla...

using the backup worked the first couple of times. i got my system back to working normal and all.
then something went horribly wrong and even after applying the backup the desktop was slow, unresponsive and simply not usable. so i figured i could just do a fresh install and load the backup over that....

worked fine until i rebooted the system after applying the backup... stops booting with a warning like "bla bla drive with UUID bla bla doesnt exist" and ends at a cryptic emergency command line. i did not change drives. used the same partitions as the previous install.... :'((

is there anything i can do to get my backup working again? *sob sob*

[Nightwalker07]

When restoring your system from your backup (using this method) I recommend that you do a fresh reinstall of the OS and then extract the backup.

Regards UUID stuff, see this post:
http://ubuntuforums.org/showthread.php?p=3860695#post3860695

Did you not exclude the /boot and fstab ?

[decideci]

thanks for the reply! i excluded /proc /mnt /sys and /media (all my windows partitions are mounted there so no need to include them in the ubuntu backup)

i did a fresh install from the live CD.... i figured that maybe i have to remove the /dev/ directory from the backup archive? i am no linux pro. but i am pretty used to work with computers and figuring stuff out. but that is a bit over my head. hehe... (just like the fact that activating the restricted binary drivers from ATI renders my gnome unusable instantly .... :( even though my card is in the compatible list)

the exact message that comes up while booting ubuntu (grub is working. ubuntu progress bar is starting, but standing still. switching to a different console shows the message):

check root=bootarg cat /proc/cmdline
or missing modules, devices: cat /proc/modules ls /dev
Alert! /dev/disk/by-uuid/75b.............


:(

probably the best idea would be to just take the time and just rebuild my system from scratch. but then i would like to have a backup option that really works the way it should be :( before i try to fiddle with those pesky ati drivers again...

[Nightwalker07]

Im pretty sure the problem is you didn't exclude your /boot which will store the UUID of your drive/partition, this is different with each install I think. Im new to all this myself! :(

When you take make your tar backup you need to exclude the /boot and all its information, because some of it unique, with it excluded you can do a fresh reinstall, extract the backup and then it will work. Im pretty sure thats the deal, anyone else got advice?

[hps]

Thanks this worked like a charm. I used your commands to back up ubuntu, and then proceeded to play with Compiz. When I finally screwed up my system so I could not get anything to work...I used the restore commands and bingo back to the way everything was. Now I just have to learn the commands meaning and get comfortable with them. Its just like 20 years ago learning the dos commands.

Thanks again and I have saved your post for future referance=D>=D>=D>

[Nightwalker07]

Good to hear another positive experience. Remember this method simply extracts the files from with the archive/backup you made. So if you extract/restore your backup-archive over the top of a damaged filesystem, it will leave files that were made anytime after the backup on the disk and will therefore not complete a clean restore.

To restore cleanly; reinstall the operating system to the same version as when you created your backup, make sure you format the partition (ie. blank the filesystem) when doing the new install. Then restore from your backup-archive onto a clean filesystem.

I hope I got that out right! Did that make sense? lol.

[decideci]

i will try my next backup with /boot excluded :) ... BUT!
i kinda forgot that i made an extra partition for /home ... ! so it fixed "itself" randomly. haha... first i formatted the partitions to make a fresh install, then applying my backup over it of course screwed it all up, because of the inclusion of /boot and the mismatch with the UUIDs.

then i decided "ok. lets to a really clean fresh install. all new. no backup :(" this time i accidently (didnt thought about it at that moment) unticked "format" for the second partition to me mounted as /home.... so. after the reboot from the install... most of my old settings were back, of course the installed software was missing, but since i figured they all store their settings in the /home dir... after reinstalling them, they had every settings and changes i made back!

i just dont have luck with my stupid radeon 9600 card.. but thats offtopic in this thread. hehe.

[tech9]

Thanks this worked like a charm. I used your commands to back up ubuntu, and then proceeded to play with Compiz. When I finally screwed up my system so I could not get anything to work...I used the restore commands and bingo back to the way everything was. Now I just have to learn the commands meaning and get comfortable with them. Its just like 20 years ago learning the dos commands.

Thanks again and I have saved your post for future referance=D>=D>=D>

excellent!

[commonplace]

How would I do a selective restore from the backed up tar.gz file? For example, I backed up my wife's entire laptop hard drive (using Heliode's method as described) and have since formatted and reinstalled Gutsy. Now I just want to restore her documents and things from her home folder to an alternate location (i.e. I don't want it to overwrite her existing home folder, but I want to be able to pick out what she needs copied over).

I'm sure it can be done... but how? :)

[Nightwalker07]

I believe you can go into the archive and extracts files, im talking about using the GUI, just double clicking the archive, can select only specific files? Im not sure, I've not tried.

[commonplace]

I believe you can go into the archive and extracts files, im talking about using the GUI, just double clicking the archive, can select only specific files? Im not sure, I've not tried.

I was pretty sure I could do that, and that ended up being what I did. It worked okay. It was just sloooooow. I was hoping for a command line way of doing it (for once -- usually I'm begging for a GUI!).

I have another backup file that's ~43GB and opening that through the GUI doesn't seem to work so well. ;-) Hopefully someone can steer me in the right direction as far doing it via the CLI.

Thanks!

[Nightwalker07]

Hell, 43GB backup file, yeah I can imagine that the GUI will struggle to open that, but given time it should work ok. All my backups have been below 2GB using this method. Im choosing to backup the system-state & its configurations, not my actual user-data.

[thelatinist]

I was pretty sure I could do that, and that ended up being what I did. It worked okay. It was just sloooooow. I was hoping for a command line way of doing it (for once -- usually I'm begging for a GUI!).

I have another backup file that's ~43GB and opening that through the GUI doesn't seem to work so well. ;-) Hopefully someone can steer me in the right direction as far doing it via the CLI.

Thanks!

I think any way you do it trying to extract certain files from a 43 GB tar archive is going to be slow. As I understand it, tar is a linear format...it has to read through the entire 43GB file to extract a file at the end of the file. This is a result of its having been developed originally as a tape archive format.

[mannheim]

I think any way you do it trying to extract certain files from a 43 GB tar archive is going to be slow.

That's one of the advantages of using rsync instead of tar. It's a trade-off. With rsync, you don't use compression and just get a perfect mirror of your filesystem on your backup media. Restoring individual files is then trivial. The downside is that you need more space for the backups.

[commonplace]

Care to point me in the direction of an rsync howto? :) It won't help me with my current problem, but it sounds ideal for future backups. I'm not so concerned with compression as I am with ease of restore.

[thelatinist]

I have been playing around with a script to automate backing up both / and /home. The script does the following things:


Backs up / and /home to folders named /Root and /Home in the folder of your choice (set in the variables section).
Names the files Backup-MMM-DD-YY-HH:MM:SS.tgz and Home-Backup-MMM-DD-YY-HH:MM:SS.tgz
Deletes backups older than a time specified in the variables section. You can set different lengths of time to preserve root and home backups.


It is easy to run this script regularly using cron.

#!/bin/sh

PATH=/usr/ucb:/usr/bin:/bin; export PATH

################################################## ################################################
# The following variables may be defined below: #
# #
# BACKUP_LOCATION is the folder where backups should be saved; the default is /Backup #
# ROOT_TIME is the length of time, in days, to preserve backups of /; the default is 90 days #
# ROOT_TIME is the length of time, in days, to preserve backups of /home; the default is 90 days #
################################################## ################################################

# Variables

BACKUP_LOCATION=/Backup
ROOT_TIME=90
HOME_TIME=90

# Export the variables set in the previous section

export BACKUP_LOCATION; export ROOT_TIME; export HOME_Time

# Tar and gzip the folder /; exclude unnecessary folders such as /proc /lost+found /mnt /media /sys; exclude /home

tar cvpzf $BACKUP_LOCATION/Root/Backup-$(date +"%b-%d-%y-%T").tgz --exclude=/proc --exclude=/lost+found --exclude=/mnt --exclude=/media --exclude=/sys --exclude=/home /

# Tar and gzip the folder /home; if you wish to exclude subdirectories, add the switch --exclude=/your_directory.

tar cvpzf $BACKUP_LOCATION/Home/Home-Backup-$(date +"%b-%d-%y-%T").tgz /home

# Delete backups older than the time specified in the variables section, then exit
find $BACKUP_LOCATION/Root/*.tgz -mtime +$ROOT_TIME -exec rm {} \;
find $BACKUP_LOCATION/Home/*.tgz -mtime +$HOME_TIME -exec rm {} \;
exit

Suggestions, corrections, etc. would be much appreciated. This is my first shell script.

[Nightwalker07]

Superb, I'll have to give this a go next time im on my work's laptop.

[tuproxy]

Would this be a good way to "clone" a system. I want to be able to image the install after I have done all the modifications that are needed such as having run the updates, setup the IPtables and installed the "plugins" that I needed.

I would assume this would work if system A (where the image was made) was the same as system B. If the two systems were slightly different, would the OS (say distro 7.04) be able to recognize the different drivers needed and download them?

I'm looking for an easy way to share a pre-configured router setup. It seems that installing the linux OS and then setting it up correctly to be a router isn't as easy as it is made out to be. So I thought that making an image of the system once it is up and running and then sharing it may make it easier for people to use.

I'm looking for ideas and opinions here. Thanks.

[Nightwalker07]

Would this be a good way to "clone" a system. I want to be able to image the install after I have done all the modifications that are needed such as having run the updates, setup the IPtables and installed the "plugins" that I needed.

I would assume this would work if system A (where the image was made) was the same as system B. If the two systems were slightly different, would the OS (say distro 7.04) be able to recognize the different drivers needed and download them?

I'm looking for an easy way to share a pre-configured router setup. It seems that installing the linux OS and then setting it up correctly to be a router isn't as easy as it is made out to be. So I thought that making an image of the system once it is up and running and then sharing it may make it easier for people to use.

I'm looking for ideas and opinions here. Thanks.Yes I imagine it will do that job well if the hardware was the same. You say they are but they have small differences? What exactly is different?

[tuproxy]

Yes I imagine it will do that job well if the hardware was the same. You say they are but they have small differences? What exactly is different?

Well, in XP if I swap the HD with the OS installed already, it will boot and use default drivers if the correct drivers aren't installed. The video will be 640x480 and it may use a generic NIC drive and a generic IDE or SATA driver.

Will Ubuntu do the same thing if say the MOBO is different? I'm just wondering if a generic image can be made to be shared with similar machines.

If system A has a differnet CD-Rom or different hard drives, will that be a problem?

[mdpalow]

Would this be a good way to "clone" a system. I want to be able to image the install after I have done all the modifications that are needed such as having run the updates, setup the IPtables and installed the "plugins" that I needed.

I would assume this would work if system A (where the image was made) was the same as system B. If the two systems were slightly different, would the OS (say distro 7.04) be able to recognize the different drivers needed and download them?

I'm looking for an easy way to share a pre-configured router setup. It seems that installing the linux OS and then setting it up correctly to be a router isn't as easy as it is made out to be. So I thought that making an image of the system once it is up and running and then sharing it may make it easier for people to use.

I'm looking for ideas and opinions here. Thanks.

I actually did something similar. I backed up my system. When all my parts came in for a new computer I was building, I built the computer and then restored the back-up files from the older machine. I upgraded MOBO, CPU, Video card, HD, and RAM. To be clear, I backed up my configuration files separately (xorg, fstab, etc..) and did not restore those files.

My system booted up properly, but I did experience some quirky things when trying to use it that I couldn't explain or remember now.

I decided to reinstall everything...

My thought is that when installing the Ubuntu OS I know it configures itself to your hardware towards the end of the install, which could be a problem. That being the case, you could always install while NOT being connected to the Internet and then certain system configuration settings might not occur, but then you might have other issues with having to set repositories and such just to name one.

One last thing... If I may RESPECTFULLY submit - this HowTo is good for a large percentage of users. However, using the Tar command suggested in the original post will eventually catch up with everyone and cause serious problems. I feel pretty confident saying it's not a matter of IF, but a matter of WHEN. That's not to say you would lose your data, but when 'WHEN' happens, you'll find you're going to have to do some research into getting a working OS and then how to restore your data.

Please don't read this a me trying to hijack this thread, it's not my intent by any means. However, our back-ups are very dear to us, right? We do them to save our treasured things and also to help us get up and running much more quickly. I'm just looking out for your/everyone's best interests.

I actually tried to contact the OP, but never heard back from him.

If you're looking to continue backing up your system using Tar (very good method), then consider checking out my script below. If you decide to download it, you can read the Help file, which will give you an indication of why it might be better to do it a different way. Also, my script will give you a better chance of working when restoring to different machines as I do the back-up a bit differently.

good luck...

[thelatinist]

However, using the Tar command suggested in the original post will eventually catch up with everyone and cause serious problems. I feel pretty confident saying it's not a matter of IF, but a matter of WHEN. That's not to say you would lose your data, but when 'WHEN' happens, you'll find you're going to have to do some research into getting a working OS and then how to restore your data.

What exactly do you mean? Exactly what problems do you foresee that the rest of us don't? Issuing cryptic warnings without any details doesn't help anyone.

Furthermore, I've examined your script, and it uses essentially the same tar command, with just a few added exclusions. Why not just tell us your logic for those exclusions and why you think they're important?

[mdpalow]

Sorry, I was in hurry to leave when I posted last.

I wouldn't go as far as to say "what I see and the rest don't." However, I think a lot of new users will not see the problem.

The problem lies in copying the configuration files into the system back-up. When doing this, you rope yourself into restoring these files along with the rest of the files being restored. Therefore, if you were to change some hardware after making a back-up, you put yourself in a bad position because the restore will copy the old configuration files into a computer that is using different hardware.

So, if you were to change to a different video card and then restored the old xorg file, you could very well not have a display. Even worse might be to add a hard drive and then put the back-up on a different mount (ex: from sda1 to sdb1) and then the system doesn't boot at all because the configuration files no longer match the original system's configuration.

For that reason, my script backs up the configuration files separately and allows the user to decide how they want to proceed with these files when restoring.

[mannheim]

Care to point me in the direction of an rsync howto? :) It won't help me with my current problem, but it sounds ideal for future backups. I'm not so concerned with compression as I am with ease of restore.

There's a very basic strategy here (http://ubuntuforums.org/showpost.php?p=3896536&postcount=9). A fuller discussion and an ingenious technique to create rotating "snapshots" using rsync and hard links is here (http://www.mikerubel.org/computers/rsync_snapshots/). (This is basically what I do for my backups.)

I've never tried "dirvish", but it is rsync-based, and is in the repositories. It is mentioned in this thread (http://ubuntuforums.org/showthread.php?t=209724).

[tech9]

There's a very basic strategy here (http://ubuntuforums.org/showpost.php?p=3896536&postcount=9). A fuller discussion and an ingenious technique to create rotating "snapshots" using rsync and hard links is here (http://www.mikerubel.org/computers/rsync_snapshots/). (This is basically what I do for my backups.)

I've never tried "dirvish", but it is rsync-based, and is in the repositories. It is mentioned in this thread (http://ubuntuforums.org/showthread.php?t=209724).

nice snippet, I will have to try this.

[LunarWolf]

Are there any good alternatives that will enable me to back up directly to a CD or DVD. Right now I am trying many new things and admitedly I screwed up a few times resulting in reinstalling Ubuntu. I just want a way to keep a standard ubuntu status as a control I can resort back to without having to reinstall it and go through a lengthy update process or take up room on an already small 20gb hard drive.

As it stands everything is pretty much at factory settings as possible, having only installed the updates at this point and configured Evolution mail to accept my gmail account as the only alteration.

[wildx005]

Just to confirm - the procedure given in the first post in this thread absolutely works (or else I would not be able to type this here). My upgrade from Feisty to Gutsy failed because of 'dependencies' and eventually the little upgrade window just went away leaving me with a very very not working system.

Because I have previously followed the instructions given in the 'First Post' - I was able to get everything back exactly how it was before:

Which, as an XP User with (practically) no Linux knowledge, seems pretty amazing,

Thanks

David

[tuproxy]

What happens if the backup command is run again and the original backup.tar file hasn't been moved or renamed. Will the command over-write the first file?

[hotrod6657]

Okay, this might be an odd question and if it's already been addressed i'm sorry but how would i go about deleting the file that's created, the backup.tgz file?

I'm really new to everything so if the answer is more obvious than expected please be nice. When i try and just delete it i'm told i don't have the proper permissions, even though i am the administrator.

Any suggestions?

[hotrod6657]

nevermind, i did a little poking and found my answer. Much simpler than i was expecting.

it was as easy as

sudo su
rm /backup.tgz

in terminal.

[milkyspit]

Guys, please excuse me if this is covered ground as this has become a VERY long thread... I tried to skim through it, but in the end am hoping someone could assist!

Preface: I'm a longtime software developer and somewhat familiar with Linux admin, at least on a small scale... thing is, most of my experience has been on CentOS and OS X, plus I've been out of that career path for well over a year, so no doubt I'm rusty, too! :(

The situation: our home network has used an OS X box as its fileserver for several years, but recently I migrated everything to a dual core box running the 64-bit version of Ubuntu 7.10. The server consists of a single partition on a terabyte drive (yes, I know not optimal but serves our needs for now, and I figure I can enable quotas in the near future if need be)... on the partition is the OS itself, plus a share exposed via Samba as 'Public' which all other machines on the network use to store and retrieve virtually all their data. I've got a pair of 250GB notebook hard drives in external enclosures and the general strategy is to run a full backup of the Public share once per week, alternating drives each week. For now the share stores less than 250GB so backup drive capacity isn't an issue. One drive (the more recent backup) resides in an airtight Otter Box in our detached garage, serving as the 'offsite' storage... the other drive (the less recent backup) is in another part of the house so the drive will be at room temperature when it's time to run the next backup onto it. I've followed this sort of scheme with our OS X server (a Mac G4 cube if it matters) for at least two years with good results.

Thing is, the backups are SLOW... ideally I'd like to speed the process without compromising integrity. Also, I'd like to use another external drive to store an image of the OS itself so I can do a bare metal recovery if needed. Let me emphasize that if the fileserver fails and we have no viable backup, we've lost pretty much everything... at the same time, if the backup process is too cumbersome I doubt I'll be vigilant about maintaining it, which also puts us in extreme jeopardy. It's got to be a robust procedure, and it's got to be easy.

Under my root account on the Ubuntu server I've got a backup alias that I kickoff via SSH overnight to run each backup...

alias bakpub='find /home/samba/shares/Public -xdev -depth -print | cpio -pdv /media/bak/.'


This may or may not be a good way to go about things... I have my suspicions that it's not the best way.

Could someone with far better grounding than I've got (there must be several of you out there) give me a critique and more importantly, suggest a better way to accomplish my goals? Script, some sort of open source software that might meet my needs, anything to improve the process. Please remember that my goals are maintenance of a valid image of the Public share; a sufficient image for rebuilding the fileserver itself; and a simple, reliable procedure.

Again, apologies if I'm covering the obvious! Please give this Ubuntu noob some guidance, and accept my sincere thanks!!

:confused:

[linux4909]

i just tried this method and got this error message:

gzip: stdout: No space left on device

it was in the middle of backing up.

my guess is, this may be cause because the size of my home/root isn't big enough to hold the backup and what i'm backing up in one?

i ran the df command. here's the output:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 9614148 9596616 0 100% /
varrun 512988 84 512904 1% /var/run
varlock 512988 0 512988 0% /var/lock
procbususb 512988 84 512904 1% /proc/bus/usb
udev 512988 84 512904 1% /dev
devshm 512988 0 512988 0% /dev/shm
lrm 512988 34696 478292 7% /lib/modules/2.6.22-14-generic/volatile
/dev/sda4 131751808 17707912 107351220 15% /home
/dev/sda1 10667128 6044348 4622780 57% /media/sda1

[frodemt]

I want to include the folders you exluded, but not files in these folders. Can I do that like this:

tar cvpjf backup.tar.bz2 --exclude=/proc/* --exclude=/lost+found/* --exclude=/backup.tar.bz2 --exclude=/mnt/* --exclude=/sys/* /

:confused:

[garret]

http://restore-backup.com might be worth a look too.

[m0rphex]

Just succesfully extracted everything on to fresh formatted harddrive from a usb stick with the help of knoppix (it was the only livecd i had lying around). However, I had excluded /dev and that turned out to be a bit hard to fix later since grub-install didn't work. Had to ask for help from a friend :D Maybe someone could make a howto for this empty /dev scenario.

[SpookyET]

Hello. I'm new to linux and bash scripting, but I do have programming knowledge. I've been looking for an easy to use backup program for Ubuntu. I am yet to find one. I'm hopping for the time machine clones to mature (Flyback and TimeVault).

I want to backup my laptop (running Ubuntu) to a desktop machine which is running Windows XP. I cannot use tools like rsnapshot and rdiff-backup because they cannot create symbolic links on NTFS, even though NTFS supports symbolic links (google for a tool called junction by Sys Internals).
It's best to create a tar archive since it can preserve permissions and symbolic links, and it can be stored on any file system.

Since a backup requires twice the disk space, it's best to write it directly to the desktop machine. So, the script mounts the machine using fuse and sshfs and creates the backup transparently over the network.

The script is by no means finished. It need to account for network failures, support incremental backups and restorations, but it's a start.

Since, I'm new to bash scripting, I'm looking for the community to help improve it.

Requirements:
SSH client, FUSE, SSHFS, etherwake on client (just apt-get them).
SSH server on backup server.


#! /bin/bash

################################################## ##############################
# CONFIGURATION
################################################## ##############################

# Backup server user name
USER=user

# Backup server password
PASSWORD=password

# Backup server host/IP
BACKUP_SERVER_HOST=192.168.1.100

# Backup server MAC address
BACKUP_SERVER_MAC=xx:xx:xx:xx:xx:xx

# Backup server boot wait time (seconds)
BACKUP_SERVER_BOOT_TIME=0

# Backup path on the backup server
BACKUP_DESTINATION="../All\ Users/Documents/Backup"

# This is the backup subdirectory, dictated by date and time
BACKUP_FILE_PREFIX=`date +%Y-%m-%dT%H:%M:%S`

# Backup server mount point
BACKUP_SERVER_MOUNT=/var/backup

# Log file
LOGFILE="/var/log/$BACKUP_FILE_PREFIX.backup.log"

# Include directories or files to backup
INCLUDES=(
"/"
)

# Exclude directories or files from backup
EXCLUDES=(
"/**/.thumbnails/"
"/**/.mozilla/**/Cache/"
"/**/.cache/tracker/"
"/**/.beagle/"
"/**/.Trash/"
"/**/.emerald/themecache/"
"/**/.fontconfig/*.cache*"
"/**/.java/deployment/cache/"
"/**/amarok/albumcovers/cache/"
"/**/amarok/albumcovers/large/"
"/**/.liferea*/mozilla/liferea/Cache/"
"/**/.liferea*/cache/"
"/**/.macromedia/Flash_Player/*SharedObjects/"
"/**/.macromedia/Macromedia/Flash\ Player/*SharedObjects/"
"/**/.metacity/sessions/"
"/**/.nautilus/saved*"
"/**/.mythtv/osdcache/"
"/**/.mythtv/themecache/"
"/**/var/cache/"
"/**/workspace/.metadata/"
"/**/.openoffice.org2/user/registry/cache/"
"/**/.openoffice.org2/user/uno_packages/cache/"
"/**/.grails/*/scriptCache/"
"/**/.wine/drive_c/windows/temp/"
"/**/lost+found"
"/cdrom/"
"/dev/"
"/media/"
"/mnt/"
"/proc/"
"/sys/"
"/tmp/"
"/var/log"
$BACKUP_SERVER_MOUNT
)

# Tar options
OPTIONS=(
"--create"
"--gzip"
"--verbose"
"--preserve"
"--file=${BACKUP_FILE_PREFIX}.backup.tar.gz"
)

# Populate the tar command
TAR_COMMAND="tar "

for OPTION in ${OPTIONS[@]};
do
TAR_COMMAND="$TAR_COMMAND $OPTION"
done

for EXCLUDE in ${EXCLUDES[@]};
do
TAR_COMMAND="$TAR_COMMAND --exclude=$EXCLUDE"
done

for INCLUDE in ${INCLUDES[@]};
do
TAR_COMMAND="$TAR_COMMAND $INCLUDE"
done

echo $TAR_COMMAND

################################################## ##############################
# Commands
################################################## ##############################

# Dump output to log
echo >> $LOGFILE 1>&2
echo >> $LOGFILE 1>&2
echo "Starting Backup On `date`" >> $LOGFILE 1>&2

# Wake up the backup server
echo "Waking up backup server with HOST ${BACKUP_SERVER_HOST} and MAC ${BACKUP_SERVER_MAC}" >> $LOGFILE 1>&2
etherwake $BACKUP_SERVER_MAC

# Wait an some time to allow the backup server to boot
echo "Waiting ${BACKUP_SERVER_BOOT_TIME} seconds for the backup server to boot" >> $LOGFILE 1>&2
sleep $BACKUP_SERVER_BOOT_TIME

# Test if the backup server is awake.
echo "Pinging backup server to check if it is awake" >> $LOGFILE 1>&2
ping -c1 -W2 $BACKUP_SERVER_HOST > /dev/null 2>&1
PING_RETURN=$?

if [ $PING_RETURN = "0" ] ; then
echo "The backup server is awake" >> $LOGFILE 1>&2
else
echo "The backup server is not awake" >> $LOGFILE 1>&2
echo "BACKUP FAILED" >> $LOGFILE 1>&2
exit;
fi

# Mount backup server
echo "Mounting backup server" >> $LOGFILE 1>&2
mkdir $BACKUP_SERVER_MOUNT
sshfs ssh://$USER:$PASSWORD@$BACKUP_SERVER_HOST:$BACKUP_DESTIN ATION $BACKUP_SERVER_MOUNT

# Back up files
echo "Backing up files" >> $LOGFILE 1>&2
exec `$TAR_COMMAND` >> $LOGFILE 1>&2
echo "Finished backing up files"

# Unmount backup server
echo "Unmounting backup server." >> $LOGFILE 1>&2
fusermount -u -z $BACKUP_SERVER_MOUNT
rm -R $BACKUP_SERVER_MOUNT

[ephman]

When restoring your system from your backup (using this method) I recommend that you do a fresh reinstall of the OS and then extract the backup.

Regards UUID stuff, see this post:
http://ubuntuforums.org/showthread.php?p=3860695#post3860695

Did you not exclude the /boot and fstab ?

yes i did exclude the boot. didn't matter including cause it'll be recreated when i reinstall. it's been a couple months since i had this issue, and since have been running with a non-encrypted drive. but i do remember modifying the fstab for the new uuid's. then i had a grub error. no worries i see now reading the thread you posted i might have to redo my grub. i'll try that in a few days.

thanks,
ephman

[mattforni]

The HOWTO as written with the edits worked beautifully for me. To all those thinking about searching for other backup methods, I don't recommend it; this is too easy and effective. Thanks!

[rugbert]

OK! This is great but for some reason I cant it to run daily even tho I added it to the cron.daily directory.

[Hopworks]

Originally Posted by Hopworks
Just wanted to say that I appreciate your post on backup and restore. I just have a question or two. I wouldn't even bother you except I'm really afraid I'll lose all my work over the past several months. This is my first trip through the wonderful world of Linux, and I had to jump through a lot of hoops to get my LAMP working.

I noticed the command line solution you offered...
tar cvpzf backup.tgz --exclude=/etc/fstab --exclude=/boot --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/sys /

Just wanted to put that backup file on my second drive. Would I just append to the front of that first part of that line? Like /video/backups/backup.tgz? The 'video' is because that drive was set up to handle my PVR files.

So then I pull the hard drives and put them into my p4 system, and what... install feisty fresh and then unarchive the file to that drive and get back where I was?

Sorry again to bother you. I'm so appreciative of your post and the help it is offering me during this very tense procedure for me. =)

Hop
Hi Hop,

Actaully all you need to do is change directories to /video/backups/ on your second drive and run the command. The backup file is created in whichever directory your in. By default the terminal loads you to /home, so that is where the backup is saved unless you specify otherwise... example cd /dev/sda2//video/backups/. (not sure if sda2 would be your second drive - just a guess- use the fdisk -l command to figure that out)

Anyway I hope this helps, but for future reference please asks these kinds of questions inside the post, so that everyone can benefit.

Take Care,

T9

Done =)

And I'm researching TAR now to find out how to span the archive over multiple files of a set size in case I go over the size of a DVD, and so I can include par files for recovery if needed.
EDIT: I got it. -M or --multi-volume. Just don't know how to specify the size of the spanned files yet. My Linux box is in transition atm, so I can't play with TAR right now.

Thanks again!

EDIT2: Could someone PM me or reply with a good ubuntuforums thread on using TAR? My search of TAR TUTORIAL turned up a lot of hits that are unrelated. =\

[rkillcrazy]

I'm using this to backup my 7.10 AMD64 system. It seems to work well. I'll give the script I use below...


# To use this open a terminal window and become root via:
# sudo su
# Then call this script via:
# sh backup.sh


# This will create a variable of today's date.
NOW=$(date +"%b-%d-%y")


# This will run the backup and name the file with today's date.
tar cvpzf /media/data/backup/backup-$NOW.tgz --exclude=/proc --exclude=/media --exclude=/lost+found --exclude=dvd.iso --exclude=/tmp --exclude=/home/rob/DVDFab --exclude=/mnt --exclude=/sys /


You'll note I excluded several files and directories. Some of them are just junk files I don't want or need to back up. However, I excluded /media because if I didn't exclude it, the script would backup the backup.tgz file which would be redundant and huge. My /media folder has mount points that point to my DVD drives, my second HDD and my file server (used to store other backups) so I wasn't worried about backing that stuff up. However, if you have important things listed there, you my feel different and remove that exclusion.

I do have a question. If one backs their system up and it has sensitive data on it, it would be bad if that tar file got out of the office. It's nice to have these things on DVDs but it also makes it easy to lose. Is there a way to set up the tar file with a password? I know I've done this with 7-Zip in Windows but I'm unsure how to do it with this setup.

01-01-08
1556 EST

[Kache]

Just wondering, why not just do the tar archiving through the use of a live CD? that way, wouldn't all the --exclude stuff not matter anymore (except backing up the archive itself, if needed), since you're no longer running off the hard drive at all?

I was also trying to add empty folders /media, /proc, /lost+found, etc to the backup archive so that I wouldn't have to deal with making them later, but can't seem to figure out how to do that. A quick search online and some experimentation didn't work, so just in case anyone knew....

[lilies34]

thanks for the tip guys! really learned a lot from reading this thread!

[cdtech]

This is a script I wrote you might want to check out.

.:: backup script ::. (http://www.bobs.homelinux.com/code/backup-script/mytarbackup)

:Note the --files-from and --exclude-from, these are text files that tell tar which directories and files to backup and not to backup.

I run this from cron and use a 4G USB stick for the backup (I leave it plugged in). I also backup my laptop with the same script, just changed the mount point to a storage drive.

Hope this helps you..

[OfMacandMen]

I'm trying to testing worst case recovery.

Here is the scenario:
- I have a backup.tar from last night
- My HD is bad and needs to be replace
- I replace my HD with a good one.

My question is how do I recover from this?

Do I need to load a fresh Ubuntu install on the new HD and then tar back?

Or can I use a Live CD? If yes How?

[KjSlag]

i backed up my Ubuntu partition with this method and later restored it and everything worked except for one thing:

any file that I had on my system before i restored but after i backed up was still there after i backed up. this then created some problems with installing packages.

what folders can i safely remove and still be able to tar to restore my partition?

for a temporary solution, i just removed my /etc and restored again. i would have removed more but i was afraid that i would remove something that's required for me to tar. is there such a file (i'm a new linux user)? or is it stored in my ram or outside the Ubuntu partition? can someone post a line that explains the use of the folders in the root directory?

[PJSingh5000]

I suggest creating a backup or restoring by booting-up with the Live CD...


BACKUP

Boot up with the Live CD. Then open a terminal and type the following...


$ sudo mkdir /mnt/backup
$ sudo mkdir /mnt/installation
$ sudo mount -tvfat /dev/sda4 /mnt/backup
$ sudo mount -text3 /dev/sda6 /mnt/installation
$ cd /mnt/installation
$ sudo tar -cvpzf /mnt/backup/Backup-Kubuntu-2008-01-26.tgz .

RESTORE

Boot up with the Live CD. Then open a terminal and type the following...


$ sudo mkdir /mnt/backup
$ sudo mkdir /mnt/installation
$ sudo mount -tvfat /dev/sda4 /mnt/backup
$ sudo mount -text3 /dev/sda6 /mnt/installation
$ sudo rm -r /mnt/installation/*
$ cd /mnt/installation
$ sudo tar -xvpzf /mnt/backup/Backup-Kubuntu-2008-01-26.tgz .

Notes

Replace sda4 with the partition or disk where you will keep your backups.
Replace sda6 with the partition where you've installed Kubuntu or Ubuntu.
If the partition or disk where you keep your backups is a Windows FAT file system, use mount -tvfat, as shown above.
If the partition or disk where you keep your backups is a Linux ext3 file system, be sure to use mount -text3 instead of mount -tvfat.
Replace Backup-Kubuntu-2008-01-26.tgz with the name of your backup archive.
Don't miss the "space" + "dot" at the end of the tar commands above.


There are a few benefits to this approach:

You don't have to unmount any file systems before you tar your Kubuntu or Ubuntu installation.
Mounted file systems are not archived, making your backup archive smaller.
You don't have to exclude any directories, especially if you've just setup your system exactly the way you want it.
The contents of directories that do not need to be archived are automatically empty (/sys, /proc).
The /media directory has the correct mount points and permissions after a restore.
The tar command becomes shorter and simpler.
During the restore process, you can completely wipe out your old installation, so you don't have to worry about unwanted files hanging around.
No tar errors.

[hoboken]

I suppose that somewhere, hidden in this post, there is a way to make the backup file go somewhere other than your filesystem folder, which is a bad partition to run out of space on.

How about you stick it on the original post?

[Ron O]

Easy backup? I installed the program Simple Backup Config with its counterpart Simple Backup Restore from the repository and allowed it to do it's automatic configuration, which selects the critical folders to back up, doing one full backup a week and daily incremental backups.

But I like having a backup on an external drive, and it took a little while to figure out how to do it. Instead of selecting the fully automatic version at the top in the General tab, select the Custom (not Manual) radio button. If you connect an external hard drive to a USB port (formatted to EXT3), when you try to choose an external destination, you cannot move up out of the file system folder to get to the external device. But WITHIN the file structure is a folder called Media. You will find a link in this folder to the external drive you just plugged in. Send the backup there.

When you are done, reset the program to automatic and it will continue to do incremental backups to your local drive.

But if you ever destroy your system by doing something dumb, you can reinstall your basic system from the distro disk, and get everything else back by restoring the full backup on your external drive.

You can also save incremental backups to the external drive. After placing the first (full) backup on the external drive, the next 6 backups you send there will automatically be the smaller incremental backup.

I have used this strategy with great sucess in Windows, and have been able to get my complete system back by reloading a basic Windows install then the external backup. I have not tried a similar restore in Linux yet, but I don't see any reason why it would not work.

[banda]

I suppose that somewhere, hidden in this post, there is a way to make the backup file go somewhere other than your filesystem folder, which is a bad partition to run out of space on.

How about you stick it on the original post?


its not exactly 'hidden', its actually in the the 1st post itself...





1: Backing-up

"What should I use to backup my system then?" might you ask. Easy; the same thing you use to backup/compress everything else; TAR. Unlike Windows, Linux doesn't restrict root access to anything, so you can just throw every single file on a partition in a TAR file!

To do this, become root with

sudo su

and go to the root of your filesystem (we use this in our example, but you can go anywhere you want your backup to end up, including remote or removable drives.)

cd /


Now, below is the full command I would use to make a backup of my system:


tar cvpzf backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/sys /

[dynamethod]

Hey there i tried this:

tar cvpzf backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/media --exclude=/sys /

i excluded media because i have a second IDE HD which just has music on it, and dont wish to back that up

i have plenty enough space however to back up the OS and settings on the primary IDE, but i get this error some way through:

tar: Error exit delayed from previous errors

im running the backup command as root btw(previous command i used is sudo su)

any help much appreciated

fyi, im backing up about 5.1gigs of stuff (the OS, Kubuntu7.10 and some installed app's/settings) on a 30gig HD, hope this helps, thanks

[m8zz]

Such a simply good way to backup. I tried the command line first, but 'Cowardly' refused to make an empty archive.
Here's my solution:
Since I use Ubuntu 7.10, I decided to use the file browser, pointed to the root, highlighted everything, right-clicked for 'create archive,' and saved to my /home/../Documents folder.
I had just made a clean Ubuntu install and completed customization recently, so only 5GB of 40GB has been used. A compressed archive should easily fit.
I realize using the GUI file browser is 'non-linuxy,' but I just wanted to get this backup done quickly.

[nofear1101]

I'm Running PcLinuxOs 2.21.2 and if someone could explain this to me alittle more i'm kinda a little lost :confused: and could use the help.... I moved my backup file to my other computer and it worked just fine but i dont know how to re-create the files listed below ans i have no idea what files i have to re-create :( . If someone could help me out that would be great!!

Thankx!! Hope to hear from someone real soon.. :)


re-create the directories you excluded:
Code:

mkdir proc
mkdir lost+found
mkdir mnt
mkdir sys
etc...

And when you reboot, everything should be the way it was when you made the backup!

[Patsoe]

Hi there, nice howto. I really like the use of standard tools like tar, because in the event of trouble you will always have access to them.

Since you're basically using tar to image the whole system here, there would be another simple option though: dump. I've chosen dump because of the following bit I found in the FreeBSD handbook (http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/backup-basics.html) (which I suppose applies to Ubuntu to some extent, too):
18.12.7 Which Backup Program Is Best?
dump(8) Period. Elizabeth D. Zwicky torture tested all the backup programs discussed here. The clear choice for preserving all your data and all the peculiarities of UNIX file systems is dump.

The command would be something like dump -0uf /media/usbdisk/{insert/filename} / which is what I use with a USB drive mounted on /media/usbdisk as the target - but I'd have to double check what happens if you have multiple mount points, so don't use this just now! ... I think with the above command dump would only copy the file system mounted on /, and would not recurse into file systems mounted on top of that. So if you have /home on a separate partition, that requires a second dump command.

I also haven't quite figured out what happens with /dev when you back up like this (or with tar which gives problems too I think?). I believe udev puts lots of stuff under /dev which you can circumvent by reading /dev/.static instead, but I haven't finished reading into it.