Results 1 to 10 of 11

Thread: Compressing a dd image in a way that allows mounting

Threaded View

  1. #1
    Join Date
    May 2008
    Location
    Edmonton, Canada
    Beans
    Hidden!

    Compressing a dd image in a way that allows mounting

    One of the best techniques for backing up any linux system (in terms of data integrity) is dd.
    No matter what OS(s) you have, what boot loader you have or how it's configured, whether you have a USB, HDD, SSD, CD, or DVD, an MBR or GPT partition table, or almost anything else, dd will back it up.

    Everyone knows the downsides of using dd for backups:
    • It takes forever for obvious reasons
    • It's backups are huge for obvious reasons
    • If you want to compress it later, you need to fill the empty space with zeros ahead of time, which also takes a while


    A big advantage of dd backups however, is the ability to mount the dd image, giving you access to individual files from the backup. You would think that this would be something that every backup solution offered, but last I checked, many standard tools admins use to backup linux drives/partitions (partclone, partimage, etc.) do NOT allow you to restore files separately.

    Another great thing you can do with dd is compress the image on the fly by piping the output of dd through a compression algorithm. This can be done with 7zip, gzip, zip, rar, or bzip2, as seen here: https://wiki.archlinux.org/index.php/disk_cloning

    Unfortunately, using ANY compression algorithm I've ever come across on a dd image pretty much means you can no longer mount it to retrieve individual files.
    The only way to compress a dd image while still being able to mount it that I've ever seen is mksquashfs.

    Using squashfs to compress a dd image has 2 big disadvantages:
    1. Apparently, there's no way to pipe dd output to mksquashfs.
      This means if you're backing up a 500GB drive (which is small by today's standards), you basically need to create a 500GB dd image, wait until that finishes, and then start compressing it with mksquashfs, and wait until that finishes. Not only can you not do these tasks in parallel, but the temporary space you need to do the backup = FULL_SIZE_OF_HDD_OR_PARTITION_YOU_WANT_BACKED_UP + COMPRESSED_SIZE_OF_THE_SAME_HDD_OR_PARTITION_YOU_W ANT_BACKED_UP. If you happen to only have 30GB worth of stuff on your 500GB hard drive, you will need about 520GB of temp space, assuming your 30GB can compress to about 20GB. In contrast, if you were to pipe the dd output to gzip (for example), it would take a long time (as it processes the entire 500GB), but you would only ever need the 20GB of space to store your backup.
    2. To mount the dd image, you have to mount the squashfs image, and then mount the dd image from inside of that. This isn't a huge problem, but is annoying


    So my question is this: Is there a way to compress a dd image on the fly, while still retaining the ability to mount the compressed image so that individual files can be read and restored?
    I basically want to get rid of dd's "It's backups are huge" disadvantage while not effecting it's "it can be mounted" advantage.

    Note: I think I've clearly outlined the advantages and disadvantages of dd above - repeating them in the replies is not going to make me reconsider using dd for backups, so please don't turn this thread into a place to argue for your favorate backup software - I'm aware that dd isn't the only backup solution out there, and that it's disadvantages make it unpopular with many people.
    Last edited by terminator14; August 20th, 2014 at 02:44 PM.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •