HOWTO: Create, Recover and Automate System Images

**GrammatonCleric** · October 19th, 2007

A disk image is a computer file containing the complete contents and structure of a data storage medium or device, such as a Hard drive, CD or DVD. The term has been generalized to cover any such file, whether originated from an actual physical storage device or not. As such, a disk image contains all the information necessary to replicate the structure and contents layout, as well as the actual contents, of a storage device, and this is the distinguishing feature between an ordinary backup and a disk image. A disk image file is usually created based upon the sectors on the medium, ignoring its filing system.

Originally disk images were used for backup and disk cloning, where replication or storage of an exact structure was necessary or efficient. With the advent of optical drives such as CD-ROM and DVD, a more commonly encountered type of disk image is a CD/DVD image, often in the form of an .ISO file (or sometimes a .BIN/.CUE file), referring to the ISO 9660 file system commonly used on such disks. These provide an exact digital replica of a CD/DVD, whereby all of the data is stored in one file to completely preserve the data structure and integrity of the CD/DVD. The .ISO format is the most common format for software disk images, but does not support multi-track data or audio CDs. In general, disk imaging is essential for retaining copy-protection data and multi-track data/audio on CD/DVD.

This how to covers creating, recovering and automating the imaging of linux systems. Unlike other processes there is no need to unmount any partitions, run special agents, use only services like ftp, capture only one partition at time, and can be done with the server or workstation is up and running.

Step 1: Determine your hard drive device.

From the Ubuntu system that you wish to image. Drop to the command line and run:

Code:

    df -h

Look for the partition that is mounted on /.

Code:

    gcleric@libria:~# df
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /dev/sda1             92196516  18761636  68751536  22% /
    varrun                 1037236       132   1037104   1% /var/run
    varlock                1037236         0   1037236   0% /var/lock
    udev                   1037236       112   1037124   1% /dev
    devshm                 1037236         0   1037236   0% /dev/shm
    lrm                    1037236     34696   1002540   4% /lib/modules/2.6.22-14-generic/volatile

In the example above the / (or root partition) is mount to the file system device of /dev/sda1. So sda is your hard drive. Make note of that. When imaging your system you can either...

1 - Image a single partition by specifying a the partition number (i.e. /dev/sda1)

2 - Or image the entire drive capturing the mbr, swap, and data by leaving off any partition numbers (i.e. /dev/sda).

Step 2: Clear free space on hard drive.

One of the disadvantages of the dd over software like Ghost, Acronis or Partimage is that dd will store the entire partition, including blocks not currently used to store files, whereas Ghost/Acronis understand the file system and overlook these unallocated blocks. The overhead isn't too bad as long as you compress the image and the unallocated blocks have low entropy. In general this will not be the case because the emtpy blocks contain random junk from bygone files. To rectify this, it's best to blank all unused blocks before making the image. After doing that, the unallocated blocks will contain mostly zeros and will therefore compress down to almost nothing.

This will fill entire unused space disk with zeros, then delete it again.

Run as root or use sudo -i

Code:

dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm -f /tmp/disk_zero_fill.tmp

NOTE: To a the status of dd progress issue the following command from a new terminal window.

Code:

sudo kill -SIGUSR1 thepidofdd

This will send signal USR1 to any dd process you own every five seconds, triggering dd to tell you where it is in the transfer and how fast it’s going. Remember, look at dd’s terminal for the output! The output should look like...

Code:

    137+0 records in
    137+0 records out
    143654912 bytes (144 MB) copied, 5.66717 seconds, 25.3 MB/s
    249+0 records in
    249+0 records out
    261095424 bytes (261 MB) copied, 11.5736 seconds, 22.6 MB/s
    388+0 records in
    387+0 records out

    405798912 bytes (406 MB) copied, 18.8116 seconds, 21.6 MB/s

'
Step 3: Creating the hard drive image.

There are many ways that the image can be created. You can dump the image over SSH, to a USB device, Samba or NFS shares, etc.

A. Method 1 - SSH: using the hard drive example (sda) from step 1 drop to the command line and run:

Code:

    sudo dd bs=15M if=/dev/sda conv=sync,noerror | ssh user@dest "gzip -9 > /path/to/image/ubuntu_linux.img.gz"

B. Method 2 - USB Drive: using the hard drive example (sda) from step 1 drop to the command line and run:

Code:

   sudo dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 > /media/path/to/usbdrive/ubuntu_linux.img.gz

C. Method 3 - NFS Share: using the hard drive example (sda) from step 1 drop to the command line and run:

Code:

    sudo dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 > /media/path/to/nfs-share/ubuntu_linux.img.gz

To learn how to setup NFS shares please refer to NFSv4Howto.

D. Method 3 - SMB Share: using the hard drive example (sda) from step 1 drop to the command line and run:

Code:

   sudo  dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 > /media/path/to/smb-share/ubuntu_linux.img.gz

To learn how to setup Samba shares please refer to SettingUpSamba.

E. Method 4 - FTP Server: using the hard drive example (sda) from step 1 drop to the command line and run: (note this method requires that the package ncftp be installed.)

Code:

   sudo  dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 | ncftpput -f ftp_server.cfg -d /var/log/ftp.log -c /media/path/to/ubuntu_linux.img.gz

The ftp_server.cfg shown in the above exaple would contain the following information about the server that you are trying to ftp the image to.

Code:

host some.server.ip
user foo
pass foomanchu10101

This set of commands will create a dd image of the sda drive. Then pipe the output of the dd if=/dev/sda to either ssh, usb drive, nfs or smb share which then will gzip the data using the best compression to a path where the image will reside.

Step 4: Recovery of the image:

To use the restore methods for SSH, SMB and NFS below it is assumed that the system that is being booted is connected to a network with a functional DHCP Server.

A. From SSH.

1 - Using the Ubuntu Desktop CD boot from it on the system that you want to recover you image to.

2 - Once the Ubuntu Desktop CD is fully booted open a terminal:

Code:

        Applications -> Accessories -> Terminal

3 - Now verify the hard drive that you wish to restore your image to is seen by the Ubuntu Desktop CD.

Code:

        grep "sd" /var/log/dmesg

or if you have a non SATA or SCSI drive

Code:

        grep "hd"  /var/log/dmesg

You should see something like.

Code:

        [17179577.532000] sda: assuming drive cache: write through
        [17179577.532000] SCSI device sda: 52428800 512-byte hdwr sectors (26844 MB)
        [17179577.532000] sda: cache data unavailable
        [17179577.532000] sda: assuming drive cache: write through
        [17179577.532000]  sda: sda1 sda2 < sda5 >
        [17179577.556000] sd 0:0:0:0: Attached scsi disk sda
        [17179587.844000] sd 0:0:0:0: Attached scsi generic sg0 type 0
        [17179594.060000] Adding 5815488k swap on /dev/sda5.  Priority:-1 extents:1 across:5815488k
        [17179595.068000] EXT3 FS on sda1, internal journal

4 - Now restoring the image.

With the drive information of the local hard disk to restore from SSH is as follows.

Code:

        ssh user@dest "gzip -dc /media/path/to/ubuntu_linux.img.gz" | sudo dd of=/dev/sda

5 - Once the restore completes reboot and remove the Ubuntu Desktop CD And enjoy.

B. From USB

1 - Follow steps 1 thru 3 in section A above.

2 - In most cases your USB drive should automount to /media/disk or /dev/sdb1

3 - Restoring image.

Code:

        sudo gzip -dc /media/disk/to/usbdrive/ubuntu_linux.img.gz | sudo dd of=/dev/sda

4 - Once the restore completes reboot and remove the Ubuntu Desktop CD and enjoy.

C. From NFS

1 - Follow steps 1 thru 3 in section A above.

2 - Now install the NFS client. Yes install from the internet with the Ubuntu Desktop CD.

Code:

        sudo apt-get install nfs-client

3 - Create a mount point for the NFS share

Code:

        sudo mkdir /mnt/nfs

4 - Mount the NFS share that has the image that you want to restore.

Code:

        sudo mount 192.168.2.2:/path/to/image /mnt/nfs

5 - Restoring image

Code:

        sudo gzip -dc /mnt/nfs/ubuntu_linux.img.gz | sudo dd of=/dev/sda

6 - Once the restore completes reboot and remove the Ubuntu Desktop CD and enjoy.

D. From SMB

1 - Follow steps 1 thru 3 in section A above.

2 - Now install the SMB client. Yes install from the internet with the Ubuntu Desktop CD.

Code:

        sudo apt-get install smbfs

3 - Create a mount point for the SMB share.

Code:

sudo mkdir /mnt/smb

4 - Mount the SMB share that has the image that you want to restore.

Code:

        sudo mount -t smbfs  \\192.168.2.2\image_share /mnt/smb

5 - Restoring image.

Code:

        sudo gzip -dc /mnt/smb/path/to/ubuntu_linux.img.gz | sudo dd of=/dev/sda

6 - Once the restore completes reboot and remove the Ubuntu Desktop CD and enjoy.

Step 5: Automating the imaging

A. Over SSH:

1 - Read the "Public key authentication" section on...

Code:

        https://help.ubuntu.com/community/SSHHowto?highlight=%28SSH%29

..and setup public key authentication between the systems that you want to image to and from. I would recommend doing this as root on the system to image.

2 - Using your favorite editor create a bash script that looks like the following, changing the path and ip address to your needs:

Code:

####################
# set date variable
####################

tdy=`date +%m%d%Y`

##################################
# consider zeroing unused space
# before imaging.  
#
# remove # in front of line below 
# enable this fuction
##################################

# dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm -f /tmp/disk_zero_fill.tmp

#########################
# image server to
#########################

dd bs=15M if=/dev/sda conv=sync,noerror | ssh root@192.168.2.2 "gzip -9 > /media/path/to/ubuntu_servername_$tdy.img.gz"

3 - Make the script executable.

Code:

        chmod +x script_name.sh

4 - Add the script to crontab.

a. First read the following about crontab.

Code:

            http://en.wikipedia.org/wiki/Cron

b. I'm not a fan of vi so I've change the default editior to nano by running this command at the command line.

Code:

            sudo -i

then...

Code:

            export EDITOR=nano

c. Now execute

Code:

            crontab -e

d. Changing the path to your script, time, date and frequency to your needs add a line that looks like. The cron entry below will execute the imaging script every sunday at 11pm.

Code:

            0 23 * * 7 /path/to/script/server_imaging.sh > /dev/null 2>&1

B. To a USB drive:

1 - Using your favorite editor create a bash script that looks like the following, changing the path and ip address to your needs:

Code:

####################
# set date variable
####################

tdy=`date +%m%d%Y`

##################################
# consider zeroing unused space
# before imaging.  
#
# remove # in front of line below 
# enable this fuction
##################################

# dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm rm -f /tmp/disk_zero_fill.tmp

#########################
# image server to
#########################

dd bs=15M if=/dev/sda conv=sync,noerror | "gzip -9 > /media/disk/to/usbdrive/ubuntu_linux.$tdy.img.g

2 - Make the script executable.

Code:

        chmod +x script_name.sh

3 - Add the script to crontab.

a. First read the following about crontab.

Code:

            http://en.wikipedia.org/wiki/Cron

b. I'm not a fan of vi so I've change the default editor to nano by running this command at the command line.

Code:

            sudo -i

then...

Code:

            export EDITOR=nano

c. Now execute

Code:

crontab -e

d. Changing the path to your script, time, date and frequency to your needs add a line that looks like.

Code:

            0 * * * 7 /path/to/script/server_imaging.sh  > /dev/null 2>&1

C. To a SMB or NFS share:

1 - Using your favorite editor create a bash script that looks like the following, changing the path and ip address to your needs:

Code:


####################
# set date variable
####################

tdy=`date +%m%d%Y`
    

##################################
# consider zeroing unused space
# before imaging.  
#
# remove # in front of line below 
# enable this fuction
##################################

# dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm rm -f /tmp/disk_zero_fill.tmp

#########################
# image server to
#########################

dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 > /media/path/to/share/ubuntu_linux.$tdy.img.gz

2 - Make the script executable.

Code:

        chmod +x script_name.sh

3 - Add the script to crontab.

a. First read the following about crontab.

Code:

            http://en.wikipedia.org/wiki/Cron

b. I'm not a fan of vi so I've change the default editor to nano by running this command at the command line.

Code:

            sudo -i

then...

Code:

            export EDITOR=nano

c. Now execute

Code:

            crontab -e

d. Changing the path to your script, time, date and frequency to your needs add a line that looks like.

Code:

            0 * * * 7 /path/to/script/server_imaging.sh  > /dev/null 2>&1

Well there you have it,

UPDATE: If you'd like to remove images after a specified period of time you could setup a cron job to execute the following on the *TARGET* system where your images reside.

Step 6: Image Clean Up.

A. Using your favorite editor create a bash script that looks like the following, changing the path your needs:

Code:

#######################################
# Remove img.gz  files older than 21 days
#######################################

find /path/to/images/*.img.gz -type f -mtime +21 -exec rm {} \;

In the code above the mtime is the number of days and older of files that should be removed.

B. By now you should have read

Code:

            http://en.wikipedia.org/wiki/Cron

C. Again if you want to change the default editor to nano run this command at the command line. If you haven't already. =)

Code:

            sudo -i

then...

Code:

            export EDITOR=nano

D. Now execute

Code:

            crontab -e

E. Changing the path to your script, time, date and frequency to your needs add a line that looks like.

Code:

            0 * * * 7 /path/to/script/remove_old_images.sh  > /dev/null 2>&1

Update/FYI.

If you restore an image to a system that has a different NIC than the one that was in the system that the image(s) was created from you may need to remark out setting in the /etc/iftab to restore eth0.

The setting may look like...

Code:

eth0 mac xx:xx:xx:xx:xx:xx arp 1

By putting a # in front of this and rebooting will restore eth0.

**GrammatonCleric** · October 26th, 2007

Fixed a few typo's.

**thelocust** · October 26th, 2007

your avatar and username are awsome.

**dthomasdigital** · October 26th, 2007

Very nice, great work..

**Steve1961** · October 26th, 2007

Excellent how to. Thanks for this, one to bookmark I think

**aldonova65** · October 26th, 2007

I have a couple of questions.

First, I understand about backing up either specific partitions or entire drives by either including or excluding the partition number. (sda vs. sda1,sda2,etc..) My question is, how do I specify the partition number on the restore? Is it just the same way? For example, do I just say 'of=/dev/sda1'?

Second, it appears to me there is a closing double quote missing in the code sample for automating the imaging for a USB drive. Am I correct?

<ignore>Third, in the code samples for both USB and SMF/NFS, I don't think you are using the date variable in the file name. Is that correct?</ignore>

Nevermind. I see the 'tdy' now. It was just hiding from me.

Thanks for this great writeup!

**GrammatonCleric** · October 27th, 2007

aldonova65:

Ok, To answer your first question you would need to create the partition on the destination drive first. Making sure the maintain the same partition number(s) as the original drive.

As to your second question that is (was) a type-o which i've now corrected!

Thanks for pointing it out.

-GC

**GrammatonCleric** · October 27th, 2007

aldonova65,

The best way to do what you are looking for is to backup and restore the partition structure of the orginal drive to the new drive by running the following.

The Backup:

Code:

sfdisk -d /dev/sda > /path/to/share/partition_structure_sda.dump

The Restore:

Code:

sfdisk /dev/sda < /path/to/share/partition_structure_sda.dump

-GC

**GrammatonCleric** · November 5th, 2007

Added dd info for M$ Windows boxes.

-GC

**mnk0** · March 25th, 2008

Noticed in the how to there was no mention on doing this on a mounted file system, in your experience would this be safe enough to do on a mounted file system.??