HOWTO: Create, Recover and Automate System Images

[GrammatonCleric]

Hi G~[thc],

So to be a little clearer. You want to image your current linux system and have the image file output to your USB drive, correct? If this is the case lets outline things.

Source Drive:

The drive that linux appears to be installed on appears to be /dev/sda so this is what we are going to take an image of.

Output Drive:

You stated that you'd like the image file to output to your USB drive there appears to be 2 attached to your system. they are...

/dev/sdb1             299G  255G   45G  86% /media/Storage
/dev/sde1             150G  112G   38G  75% /media/NEXT 160

.. or is one of these an internal hard drive? I ask because if there is data one either of these that you wish to "include" in your image this would require a second image of that drive (i.e. sdb1 or sde1).

So I'll assume that you want the image to go to /dev/sde or /media/NEXT 160.

Other:

There are three things that we need to discuss about before we can kick off the image. First being what file system does your USB drive use (i.e. FAT32, NTFS, EXT3, REISER, XFS, etc.)? I ask because if it is using FAT32 there is a maximum limit of file size that FAT32 can handle which is 4gb. So if the output image file is larger than 4gb, and chances are that it will be, then this may cause the the image job to fail.

Secondly, the name of the mounted USB dive contains a space (i.e. /media/NEXT 160). When you script out your imaging you will most likely need to include quotes around the full path, like....

"/media/NEXT 160/ubuntu_linux.img.gz"

Last but not least I would really consider running the "dd" job that fills the empty space of your /dev/sda drive with zeros first.

dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm -f /tmp/disk_zero_fill.tmp

This way you are going to get the smallest image file. Be sure to run this as root (i.e. sudo -i )!


Imaging:

If you are not scripting this out and going to run them interactively I would switch to root to run then rather that using sudo followed by the commands.

Step 1 - Switch to root.

sudo -i

Step 2 - Kick off the zero fill (this will take a while).

dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm -f /tmp/disk_zero_fill.tmp

Step 3 - Verify that the zero fill temp file has been deleted. Run a "df" and look to make sure that the root partition is not 100% full. The zero fill will fill this partition but it should remove the file that is filling the partition. If for some reason the the file is not deleted remove it manually with...

rm -f /tmp/disk_zero_fill.tmp

NOT VERIFYING THIS COULD RESULT IN A VERY LARGE IMAGE!!!!

Step 3 - Now that the zero fill is complete and the partition is ready to image. For your setup the command set should be. (again this should be run as root). Again this process could take a long time to complete.

dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 > "/media/NEXT 160/ubuntu_linux.img.gz"

Notice that since you are running as root you do not need the sudo at the begining of the command set.

Feel free to change the name of the output file to whatever you like. But in your case be sure to include the quotes around the output path (i.e. "/media/NEXT 160/ubuntu_linux.img.gz")

Hope this helps,

-GC

[G~[thc]]

Wow, that's great. You're really good at breaking down tasks and organizing instructions clearly.

The result of the zero-fill is:
---------------------------------------------------------
dd: writing `/tmp/disk_zero_fill.tmp': No space left on device
8430+0 records in
8429+0 records out
70713696256 bytes (71 GB) copied, 1390.79 seconds, 50.8 MB/s
---------------------------------------------------------
And the result of the df is:
---------------------------------------------------------
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 73742752 4591916 65404884 7% /
varrun 1029648 100 1029548 1% /var/run
varlock 1029648 0 1029648 0% /var/lock
procbususb 1029648 168 1029480 1% /proc/bus/usb
udev 1029648 168 1029480 1% /dev
devshm 1029648 0 1029648 0% /dev/shm
lrm 1029648 33788 995860 4% /lib/modules/2.6.20-17-generic/volatile
/dev/sdg1 2047488 723552 1323936 36% /media/disk
/dev/sdc1 156250144 114941440 41308704 74% /media/NEXT 160
---------------------------------------------------------
So it seems that, after z-f'ing the space, I've used about 7% of the drive.
Thereafter, the result of dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 > "/media/NEXT 160/ubuntu_linux.img.gz" is:
---------------------------------------------------------
5087+1 records in
5088+0 records out
80027320320 bytes (80 GB) copied, 2843.15 seconds, 28.1 MB/s
---------------------------------------------------------
So it seems to have worked. And the .img.gz file is indeed present in the portable drive. So has this created an image of only the fs partition, or the swap as well? Is it a -complete-
system image?
Thanks so much. Having folks like you around sure makes life a whole lot easier for those of us who are just starting to get the hang of it a bit.

[GrammatonCleric]

Hi G~[thc],

Everything looks good and to answer your question yes it is a complete image (i.e. all partitions and partition structures) of your /dev/sda drive. =)

Now that you have an image have you read through the restore process? In the event that the system that you just imaged is your only PC I would print out the restore process in case you need it.

Glad to help!

-GC

[sionghua]

I added the script for imaging to usb drive via 'scheduled task', which uses cron and anacron, but when I try to run it I encounter permission error when the script tries to dd /dev/sda4 , which is the root directory that I am running on. What could be the problem?

[GrammatonCleric]

Hi sionghua,

Are you calling the script via which users cron job? Yours or root? The script needs to be called from root's crontab. To list what jobs are in the root crontab run.

sudo crontab -l

now to edit the root crontab...

sudo crontab -e

Also are you trying to image the whole sda drive or just the single partition sda4? I ask because dd-ing just sda4 will only image data in that partition. Now if you want to image the the whole drive you will need to change your script to /dev/sda.

Can you post the output of "df -h" ?

-GC

[sionghua]

What name should I save the crontab file to? df -h produces

Filesystem Size Used Avail Use% Mounted on
/dev/sda4 20G 9.3G 9.5G 50% /
tmpfs 886M 0 886M 0% /lib/init/rw
varrun 886M 328K 885M 1% /var/run
varlock 886M 0 886M 0% /var/lock
udev 886M 2.7M 883M 1% /dev
tmpfs 886M 888K 885M 1% /dev/shm
lrm 886M 2.0M 884M 1% /lib/modules/2.6.27-7-generic/volatile
/dev/sda7 129G 15G 107G 13% /home
/dev/sdb1 466G 347G 120G 75% /media/FreeAgentDesktop

I am trying to backup sda4 and sda7 to sdb1, my script file is

####################
# set date variable
####################

tdy=`date +%m%d%Y`

##################################
# consider zeroing unused space
# before imaging.
#
# remove # in front of line below
# enable this fuction
##################################

dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm rm -f /tmp/disk_zero_fill.tmp

#########################
# image server to
#########################

dd bs=15M if=/dev/sda4 conv=sync,noerror | "gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda4.$tdy.img.gz"
dd bs=15M if=/dev/sda7 conv=sync,noerror | "gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda7.$tdy.img.gz"

Thanks again.

edit: Got it just ctrl+o and ctrl+x in nano, it will automatically update crontab. Thanks.

edit: upon running the script manually I get 'not found' error as follows:

root@sionghua-desktop:/home/sionghua# sh ddbackup.sh
dd: writing `/tmp/disk_zero_fill.tmp': No space left on device
1339+0 records in
1338+0 records out
11229851648 bytes (11 GB) copied, 192.981 s, 58.2 MB/s
ddbackup.sh: 21: gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda4.11122008.img.gz: not found
ddbackup.sh: 22: gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda7.11122008.img.gz: not found

edit: I think I know why, there should not be " before and after the gzip command unless it is used in ssh.

edit: Thanks it works.

[GrammatonCleric]

Hi sionghua,

First, lets talk about why you are splitting your image into two? Doing this will make restoring a little more tricky. Also since both partitions that you are creating images of are on the same drive it would make your life much easier if you just capture the whole drive in one image. Doing this will restore EVERYTHING (i.e. boot sector, partition tables, swap, all your apps, all your data, etc.). It would mean changing your script from...

dd bs=15M if=/dev/sda4 conv=sync,noerror | "gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda4.$tdy.img.gz"
dd bs=15M if=/dev/sda7 conv=sync,noerror | "gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda7.$tdy.img.gz"

To...

dd bs=15M if=/dev/sda conv=sync,noerror | "gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/linux_system.$tdy.img.gz"

Seriously think about it!!!

If you are breaking them up to capture incremental changes on your /home partition then I would consider using a tool like "Simple Backup Config" which is in the repos and can be installed by...

sudo apt-get install sbackup

...this is a very easy to use backup tool and is much better at capturing daily incremental changes then taking images of the partition.


That said.... Here's what you need would do to get your current script working. But seriously think about changing how you are going to image your system. Just my 2 cents. =)

###############################

Step 1 - Create a scripts directory.

sudo mkdir /etc/scripts

Step 2 - Take the contents of your imaging script...

####################
# set date variable
####################

tdy=`date +%m%d%Y`

##################################
# consider zeroing unused space
# before imaging.  
#
# remove # in front of line below 
# enable this fuction
##################################

dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm rm -f /tmp/disk_zero_fill.tmp

#########################
# image server to
#########################

dd bs=15M if=/dev/sda4 conv=sync,noerror | "gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda4.$tdy.img.gz"
dd bs=15M if=/dev/sda7 conv=sync,noerror | "gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda7.$tdy.img.gz"

.... and put them in a script file.

sudo nano /etc/scripts/image_backup.sh

Consider adding a auto remove of oldest images to the bottom of your script. Something like...

####################
# set date variable
####################

tdy=`date +%m%d%Y`

##################################
# consider zeroing unused space
# before imaging.  
#
# remove # in front of line below 
# enable this fuction
##################################

dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm rm -f /tmp/disk_zero_fill.tmp

#########################
# image server to
#########################

dd bs=15M if=/dev/sda4 conv=sync,noerror | "gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda4.$tdy.img.gz"
dd bs=15M if=/dev/sda7 conv=sync,noerror | "gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda7.$tdy.img.gz"

#######################################
# Remove img.gz  files older than 21 days
#######################################

find /media/FreeAgentDesktop/Ubuntu/Backup/*.img.gz -type f -mtime +21 -exec rm {} \;

Step 3 - Make script executable.

sudo chmod 700  /etc/scripts/image_backup.sh

Step 4 - Add your script to roots crontab.

Change the default editor to nano if you don't like or don't know how to use vi.

export EDITOR=nano

now edit the root crontab...

 sudo crontab -e

...and add line like...

0  23  * * 7 /path/to/script/server_imaging.sh  > /dev/null 2>&1

This will run your image script at 11:00pm every Sunday.

This should do it!

###############################

[sionghua]

Hi GrammatonCleric,

Thanks for your suggestions, but I don't want to image my windows partition and old data partitions which are also on the same harddisk. But thanks for the suggestion for auto-remove and the scrpt directory. I am using rsync to backup my data into daily/wekly/monthly/ folders so no need to use sbackup. BTW I think adding "" for the gzip command create problems.

Thanks again.

edit: by the way dunno why by running

sudo kill -SIGUSR1 thepidofdd

it doesn't show the progress here. But not a big deal.

[sionghua]

Hi, anyone knows how to configure this to run by anacron? Because my computer is not always on. Thanks.

edit: found it at
http://linux.die.net/man/8/anacron
http://linux.die.net/man/5/anacrontab

can't find the option to delete this post so I edit instead.

[sionghua]

It appears that only the free space on sda4 is filled with zero, while sda7 is not, because the size of the backup file for sda7 is bigger than the size of used space in sda7.

my script as follows:

####################
# set date variable
####################

tdy=`date +%d-%m-%Y`

##################################
# consider zeroing unused space
# before imaging.
#
# remove # in front of line below
# enable this fuction
##################################

dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm rm -f /tmp/disk_zero_fill.tmp

#########################
# image server to
#########################

dd bs=15M if=/dev/sda4 conv=sync,noerror | gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda4.$tdy.img.gz
dd bs=15M if=/dev/sda7 conv=sync,noerror | gzip -9 > /media/FreeAgentDesktop/Ubuntu/Backup/sda7.$tdy.img.gz

#######################################
# Remove img.gz files older than 90 days
#######################################

find /media/FreeAgentDesktop/Ubuntu/Backup/*.img.gz -type f -mtime +90 -exec rm {} \;

and running the script shows the following, but it was stopped before it complete.

root@sionghua-desktop:/home/sionghua# sh ddbackup.sh
dd: writing `/tmp/disk_zero_fill.tmp': No space left on device
1339+0 records in
1338+0 records out
11229851648 bytes (11 GB) copied, 204.268 s, 55.0 MB/s
1365+1 records in
1366+0 records out
21485322240 bytes (21 GB) copied, 2365.57 s, 9.1 MB/s
^C2159+0 records in
2158+0 records out
33942405120 bytes (34 GB) copied, 7346.55 s, 4.6 MB/s

Is zero filling working in this case?