Page 1 of 5 123 ... LastLast
Results 1 to 10 of 50

Thread: HOWTO: Create, Recover and Automate System Images

  1. #1
    Join Date
    Jun 2005
    Location
    Albuquerque, New Mexico
    Beans
    272
    Distro
    Ubuntu Development Release

    HOWTO: Create, Recover and Automate System Images

    A disk image is a computer file containing the complete contents and structure of a data storage medium or device, such as a Hard drive, CD or DVD. The term has been generalized to cover any such file, whether originated from an actual physical storage device or not. As such, a disk image contains all the information necessary to replicate the structure and contents layout, as well as the actual contents, of a storage device, and this is the distinguishing feature between an ordinary backup and a disk image. A disk image file is usually created based upon the sectors on the medium, ignoring its filing system.

    Originally disk images were used for backup and disk cloning, where replication or storage of an exact structure was necessary or efficient. With the advent of optical drives such as CD-ROM and DVD, a more commonly encountered type of disk image is a CD/DVD image, often in the form of an .ISO file (or sometimes a .BIN/.CUE file), referring to the ISO 9660 file system commonly used on such disks. These provide an exact digital replica of a CD/DVD, whereby all of the data is stored in one file to completely preserve the data structure and integrity of the CD/DVD. The .ISO format is the most common format for software disk images, but does not support multi-track data or audio CDs. In general, disk imaging is essential for retaining copy-protection data and multi-track data/audio on CD/DVD.

    This how to covers creating, recovering and automating the imaging of linux systems. Unlike other processes there is no need to unmount any partitions, run special agents, use only services like ftp, capture only one partition at time, and can be done with the server or workstation is up and running.


    Step 1: Determine your hard drive device.


    From the Ubuntu system that you wish to image. Drop to the command line and run:

    Code:
        df -h
    Look for the partition that is mounted on /.

    Code:
        gcleric@libria:~# df
        Filesystem           1K-blocks      Used Available Use% Mounted on
        /dev/sda1             92196516  18761636  68751536  22% /
        varrun                 1037236       132   1037104   1% /var/run
        varlock                1037236         0   1037236   0% /var/lock
        udev                   1037236       112   1037124   1% /dev
        devshm                 1037236         0   1037236   0% /dev/shm
        lrm                    1037236     34696   1002540   4% /lib/modules/2.6.22-14-generic/volatile
    In the example above the / (or root partition) is mount to the file system device of /dev/sda1. So sda is your hard drive. Make note of that. When imaging your system you can either...
    1 - Image a single partition by specifying a the partition number (i.e. /dev/sda1)
    2 - Or image the entire drive capturing the mbr, swap, and data by leaving off any partition numbers (i.e. /dev/sda).

    Step 2: Clear free space on hard drive.

    One of the disadvantages of the dd over software like Ghost, Acronis or Partimage is that dd will store the entire partition, including blocks not currently used to store files, whereas Ghost/Acronis understand the file system and overlook these unallocated blocks. The overhead isn't too bad as long as you compress the image and the unallocated blocks have low entropy. In general this will not be the case because the emtpy blocks contain random junk from bygone files. To rectify this, it's best to blank all unused blocks before making the image. After doing that, the unallocated blocks will contain mostly zeros and will therefore compress down to almost nothing.


    This will fill entire unused space disk with zeros, then delete it again.


    Run as root or use sudo -i
    Code:
    dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm -f /tmp/disk_zero_fill.tmp

    NOTE:
    To a the status of dd progress issue the following command from a new terminal window.

    Code:
    sudo kill -SIGUSR1 thepidofdd


    This will send signal USR1 to any dd process you own every five seconds, triggering dd to tell you where it is in the transfer and how fast it’s going. Remember, look at dd’s terminal for the output! The output should look like...


    Code:
        137+0 records in
        137+0 records out
        143654912 bytes (144 MB) copied, 5.66717 seconds, 25.3 MB/s
        249+0 records in
        249+0 records out
        261095424 bytes (261 MB) copied, 11.5736 seconds, 22.6 MB/s
        388+0 records in
        387+0 records out
    
        405798912 bytes (406 MB) copied, 18.8116 seconds, 21.6 MB/s
    '
    Step 3: Creating the hard drive image.


    There are many ways that the image can be created. You can dump the image over SSH, to a USB device, Samba or NFS shares, etc.
    A. Method 1 - SSH: using the hard drive example (sda) from step 1 drop to the command line and run:
    Code:
        sudo dd bs=15M if=/dev/sda conv=sync,noerror | ssh user@dest "gzip -9 > /path/to/image/ubuntu_linux.img.gz"
    B. Method 2 - USB Drive: using the hard drive example (sda) from step 1 drop to the command line and run:
    Code:
       sudo dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 > /media/path/to/usbdrive/ubuntu_linux.img.gz
    C. Method 3 - NFS Share: using the hard drive example (sda) from step 1 drop to the command line and run:
    Code:
        sudo dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 > /media/path/to/nfs-share/ubuntu_linux.img.gz
    To learn how to setup NFS shares please refer to NFSv4Howto.
    D. Method 3 - SMB Share: using the hard drive example (sda) from step 1 drop to the command line and run:
    Code:
       sudo  dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 > /media/path/to/smb-share/ubuntu_linux.img.gz
    To learn how to setup Samba shares please refer to SettingUpSamba.
    E. Method 4 - FTP Server: using the hard drive example (sda) from step 1 drop to the command line and run: (note this method requires that the package ncftp be installed.)
    Code:
       sudo  dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 | ncftpput -f ftp_server.cfg -d /var/log/ftp.log -c /media/path/to/ubuntu_linux.img.gz
    The ftp_server.cfg shown in the above exaple would contain the following information about the server that you are trying to ftp the image to.

    Code:
    host some.server.ip
    user foo
    pass foomanchu10101
    This set of commands will create a dd image of the sda drive. Then pipe the output of the dd if=/dev/sda to either ssh, usb drive, nfs or smb share which then will gzip the data using the best compression to a path where the image will reside.




    Step 4: Recovery of the image:


    To use the restore methods for SSH, SMB and NFS below it is assumed that the system that is being booted is connected to a network with a functional DHCP Server.
    A. From SSH.
    1 - Using the Ubuntu Desktop CD boot from it on the system that you want to recover you image to.
    2 - Once the Ubuntu Desktop CD is fully booted open a terminal:
    Code:
            Applications -> Accessories -> Terminal
    3 - Now verify the hard drive that you wish to restore your image to is seen by the Ubuntu Desktop CD.
    Code:
            grep "sd" /var/log/dmesg
    or if you have a non SATA or SCSI drive

    Code:
            grep "hd"  /var/log/dmesg
    You should see something like.


    Code:
            [17179577.532000] sda: assuming drive cache: write through
            [17179577.532000] SCSI device sda: 52428800 512-byte hdwr sectors (26844 MB)
            [17179577.532000] sda: cache data unavailable
            [17179577.532000] sda: assuming drive cache: write through
            [17179577.532000]  sda: sda1 sda2 < sda5 >
            [17179577.556000] sd 0:0:0:0: Attached scsi disk sda
            [17179587.844000] sd 0:0:0:0: Attached scsi generic sg0 type 0
            [17179594.060000] Adding 5815488k swap on /dev/sda5.  Priority:-1 extents:1 across:5815488k
            [17179595.068000] EXT3 FS on sda1, internal journal
    4 - Now restoring the image.
    With the drive information of the local hard disk to restore from SSH is as follows.

    Code:
            ssh user@dest "gzip -dc /media/path/to/ubuntu_linux.img.gz" | sudo dd of=/dev/sda
    5 - Once the restore completes reboot and remove the Ubuntu Desktop CD And enjoy.
    B. From USB
    1 - Follow steps 1 thru 3 in section A above.
    2 - In most cases your USB drive should automount to /media/disk or /dev/sdb1
    3 - Restoring image.
    Code:
            sudo gzip -dc /media/disk/to/usbdrive/ubuntu_linux.img.gz | sudo dd of=/dev/sda
    4 - Once the restore completes reboot and remove the Ubuntu Desktop CD and enjoy.
    C. From NFS
    1 - Follow steps 1 thru 3 in section A above.

    2 - Now install the NFS client. Yes install from the internet with the Ubuntu Desktop CD.
    Code:
            sudo apt-get install nfs-client
    3 - Create a mount point for the NFS share
    Code:
            sudo mkdir /mnt/nfs
    4 - Mount the NFS share that has the image that you want to restore.
    Code:
            sudo mount 192.168.2.2:/path/to/image /mnt/nfs
    5 - Restoring image
    Code:
            sudo gzip -dc /mnt/nfs/ubuntu_linux.img.gz | sudo dd of=/dev/sda
    6 - Once the restore completes reboot and remove the Ubuntu Desktop CD and enjoy.

    D. From SMB
    1 - Follow steps 1 thru 3 in section A above.

    2 - Now install the SMB client. Yes install from the internet with the Ubuntu Desktop CD.
    Code:
            sudo apt-get install smbfs
    3 - Create a mount point for the SMB share.
    Code:
    sudo mkdir /mnt/smb
    4 - Mount the SMB share that has the image that you want to restore.
    Code:
            sudo mount -t smbfs  \\192.168.2.2\image_share /mnt/smb
    5 - Restoring image.
    Code:
            sudo gzip -dc /mnt/smb/path/to/ubuntu_linux.img.gz | sudo dd of=/dev/sda
    6 - Once the restore completes reboot and remove the Ubuntu Desktop CD and enjoy.
    Step 5: Automating the imaging
    A. Over SSH:
    1 - Read the "Public key authentication" section on...
    Code:
            https://help.ubuntu.com/community/SSHHowto?highlight=%28SSH%29
    ..and setup public key authentication between the systems that you want to image to and from. I would recommend doing this as root on the system to image.
    2 - Using your favorite editor create a bash script that looks like the following, changing the path and ip address to your needs:
    Code:
    ####################
    # set date variable
    ####################
    
    tdy=`date +%m%d%Y`
    
    ##################################
    # consider zeroing unused space
    # before imaging.  
    #
    # remove # in front of line below 
    # enable this fuction
    ##################################
    
    # dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm -f /tmp/disk_zero_fill.tmp
    
    #########################
    # image server to
    #########################
    
    dd bs=15M if=/dev/sda conv=sync,noerror | ssh root@192.168.2.2 "gzip -9 > /media/path/to/ubuntu_servername_$tdy.img.gz"
    3 - Make the script executable.
    Code:
            chmod +x script_name.sh
    4 - Add the script to crontab.
    a. First read the following about crontab.
    Code:
                http://en.wikipedia.org/wiki/Cron
    b. I'm not a fan of vi so I've change the default editior to nano by running this command at the command line.
    Code:
                sudo -i
    then...
    Code:
                export EDITOR=nano
    c. Now execute
    Code:
                crontab -e
    d. Changing the path to your script, time, date and frequency to your needs add a line that looks like. The cron entry below will execute the imaging script every sunday at 11pm.
    Code:
                0 23 * * 7 /path/to/script/server_imaging.sh > /dev/null 2>&1
    B. To a USB drive:
    1 - Using your favorite editor create a bash script that looks like the following, changing the path and ip address to your needs:
    Code:
    ####################
    # set date variable
    ####################
    
    tdy=`date +%m%d%Y`
    
    ##################################
    # consider zeroing unused space
    # before imaging.  
    #
    # remove # in front of line below 
    # enable this fuction
    ##################################
    
    # dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm rm -f /tmp/disk_zero_fill.tmp
    
    #########################
    # image server to
    #########################
    
    dd bs=15M if=/dev/sda conv=sync,noerror | "gzip -9 > /media/disk/to/usbdrive/ubuntu_linux.$tdy.img.g
    2 - Make the script executable.
    Code:
            chmod +x script_name.sh
    3 - Add the script to crontab.
    a. First read the following about crontab.
    Code:
                http://en.wikipedia.org/wiki/Cron
    b. I'm not a fan of vi so I've change the default editor to nano by running this command at the command line.
    Code:
                sudo -i
    then...
    Code:
                export EDITOR=nano
    c. Now execute
    Code:
    crontab -e
    d. Changing the path to your script, time, date and frequency to your needs add a line that looks like.
    Code:
                0 * * * 7 /path/to/script/server_imaging.sh  > /dev/null 2>&1
    C. To a SMB or NFS share:
    1 - Using your favorite editor create a bash script that looks like the following, changing the path and ip address to your needs:
    Code:
    
    ####################
    # set date variable
    ####################
    
    tdy=`date +%m%d%Y`
        
    
    ##################################
    # consider zeroing unused space
    # before imaging.  
    #
    # remove # in front of line below 
    # enable this fuction
    ##################################
    
    # dd if=/dev/zero of=/tmp/disk_zero_fill.tmp bs=8M; rm rm -f /tmp/disk_zero_fill.tmp
    
    #########################
    # image server to
    #########################
    
    dd bs=15M if=/dev/sda conv=sync,noerror | gzip -9 > /media/path/to/share/ubuntu_linux.$tdy.img.gz
    2 - Make the script executable.
    Code:
            chmod +x script_name.sh
    3 - Add the script to crontab.
    a. First read the following about crontab.
    Code:
                http://en.wikipedia.org/wiki/Cron
    b. I'm not a fan of vi so I've change the default editor to nano by running this command at the command line.
    Code:
                sudo -i
    then...
    Code:
                export EDITOR=nano
    c. Now execute
    Code:
                crontab -e
    d. Changing the path to your script, time, date and frequency to your needs add a line that looks like.
    Code:
                0 * * * 7 /path/to/script/server_imaging.sh  > /dev/null 2>&1
    Well there you have it,


    UPDATE: If you'd like to remove images after a specified period of time you could setup a cron job to execute the following on the *TARGET* system where your images reside.

    Step 6: Image Clean Up.
    A. Using your favorite editor create a bash script that looks like the following, changing the path your needs:
    Code:
    #######################################
    # Remove img.gz  files older than 21 days
    #######################################
    
    find /path/to/images/*.img.gz -type f -mtime +21 -exec rm {} \;
    In the code above the mtime is the number of days and older of files that should be removed.
    B. By now you should have read
    Code:
                http://en.wikipedia.org/wiki/Cron
    C. Again if you want to change the default editor to nano run this command at the command line. If you haven't already. =)
    Code:
                sudo -i
    then...
    Code:
                export EDITOR=nano
    D. Now execute
    Code:
                crontab -e
    E. Changing the path to your script, time, date and frequency to your needs add a line that looks like.
    Code:
                0 * * * 7 /path/to/script/remove_old_images.sh  > /dev/null 2>&1
    Update/FYI.

    If you restore an image to a system that has a different NIC than the one that was in the system that the image(s) was created from you may need to remark out setting in the /etc/iftab to restore eth0.

    The setting may look like...


    Code:
    eth0 mac xx:xx:xx:xx:xx:xx arp 1
    By putting a # in front of this and rebooting will restore eth0.
    Last edited by GrammatonCleric; December 12th, 2009 at 04:28 PM. Reason: Adding a possbile config file change for iftab for restored images
    "Nice jail. Looks strong."
    - H. Houdini

  2. #2
    Join Date
    Jun 2005
    Location
    Albuquerque, New Mexico
    Beans
    272
    Distro
    Ubuntu Development Release

    Re: HOWTO: Create, Recover and Automate System Images

    Fixed a few typo's.
    "Nice jail. Looks strong."
    - H. Houdini

  3. #3
    Join Date
    Aug 2006
    Location
    BEHIND YOU!
    Beans
    435
    Distro
    Kubuntu 10.10 Maverick Meerkat

    Re: HOWTO: Create, Recover and Automate System Images

    your avatar and username are awsome.
    There are 10 types of people in the world: those who understand binary, and those who don't.

  4. #4
    Join Date
    Mar 2007
    Beans
    95

    Re: HOWTO: Create, Recover and Automate System Images

    Very nice, great work..

  5. #5
    Join Date
    Jul 2005
    Location
    /dev/UK
    Beans
    981
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: HOWTO: Create, Recover and Automate System Images

    Excellent how to. Thanks for this, one to bookmark I think

  6. #6
    Join Date
    Feb 2007
    Location
    Knoxville, TN
    Beans
    3
    Distro
    Ubuntu 7.04 Feisty Fawn

    Question Re: HOWTO: Create, Recover and Automate System Images

    I have a couple of questions.

    First, I understand about backing up either specific partitions or entire drives by either including or excluding the partition number. (sda vs. sda1,sda2,etc..) My question is, how do I specify the partition number on the restore? Is it just the same way? For example, do I just say 'of=/dev/sda1'?

    Second, it appears to me there is a closing double quote missing in the code sample for automating the imaging for a USB drive. Am I correct?

    <ignore>Third, in the code samples for both USB and SMF/NFS, I don't think you are using the date variable in the file name. Is that correct?</ignore>

    Nevermind. I see the 'tdy' now. It was just hiding from me.

    Thanks for this great writeup!
    Last edited by aldonova65; October 26th, 2007 at 11:10 PM.

  7. #7
    Join Date
    Jun 2005
    Location
    Albuquerque, New Mexico
    Beans
    272
    Distro
    Ubuntu Development Release

    Re: HOWTO: Create, Recover and Automate System Images

    aldonova65:

    Ok, To answer your first question you would need to create the partition on the destination drive first. Making sure the maintain the same partition number(s) as the original drive.

    As to your second question that is (was) a type-o which i've now corrected!

    Thanks for pointing it out.

    -GC
    "Nice jail. Looks strong."
    - H. Houdini

  8. #8
    Join Date
    Jun 2005
    Location
    Albuquerque, New Mexico
    Beans
    272
    Distro
    Ubuntu Development Release

    Re: HOWTO: Create, Recover and Automate System Images

    aldonova65,

    The best way to do what you are looking for is to backup and restore the partition structure of the orginal drive to the new drive by running the following.

    The Backup:
    Code:
    sfdisk -d /dev/sda > /path/to/share/partition_structure_sda.dump
    The Restore:
    Code:
    sfdisk /dev/sda < /path/to/share/partition_structure_sda.dump
    -GC
    Last edited by GrammatonCleric; October 27th, 2007 at 01:36 PM.
    "Nice jail. Looks strong."
    - H. Houdini

  9. #9
    Join Date
    Jun 2005
    Location
    Albuquerque, New Mexico
    Beans
    272
    Distro
    Ubuntu Development Release

    Re: HOWTO: Create, Recover and Automate System Images

    Added dd info for M$ Windows boxes.

    -GC
    "Nice jail. Looks strong."
    - H. Houdini

  10. #10
    Join Date
    Jul 2006
    Beans
    19

    Re: HOWTO: Create, Recover and Automate System Images

    Noticed in the how to there was no mention on doing this on a mounted file system, in your experience would this be safe enough to do on a mounted file system.??

Page 1 of 5 123 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •