Results 1 to 5 of 5

Thread: Checking backed up data for corruption?

  1. #1
    Join Date
    Jan 2009
    Beans
    244
    Distro
    Ubuntu 9.10 Karmic Koala

    Checking backed up data for corruption?

    I have a lot of information stored on an external hard drive which I periodically back up onto my computer's hard drive.

    When I say backup I mean I just delete all the current files and copy+paste the new ones. But this kind of worries me because of the possibility of some data being corrupted in the process.

    What's a simple way to backup my data but also have it check to make sure none of the files have been corrupted while in the process transferring?

    Creating a compressed archive of my external HD and then uncompressing it to check for errors sounds like it might work, but I want something that I can just hit a button to start backing it up and walk away without much effort on my part.

    It seems like there should be a checkable box in the file transfer progress bar that says something like "compare original and copied versions bit for bit" or something.
    CPU: AMD Phenom II X4 965 @ 3.5GHz
    GPU: MSI GTX 260 w/896MB DDR3 @ 655MHz
    RAM: Kingston HyperX 8GB (4x 2GB) DDR3 2000
    HDD: Samsung Spinpoint F3 1TB (2x fake RAID0)

  2. #2
    Join Date
    Apr 2009
    Beans
    1,173

    Re: Checking backed up data for corruption?

    Quote Originally Posted by blueshiftoverwatch View Post
    What's a simple way to backup my data but also have it check to make sure none of the files have been corrupted while in the process transferring?
    The hardware and OS will inform you of any error with an IO Error warning. I assume (and have never seen evidence to convince me otherwise) that the destination is a correct copy if I do not receive an error.

    If you want to run a proper verification then you can

    - run rsync with the -n (or --dry-run) flag to report any differences between source and destination

    - use Meld for graphical directory comparison

    - use Krusader's synchronise tool with compare by content

    - run diff, which has no output when the files match

    (and many more ways). My preference is rsync if I expect an absolute match, as with backup, and Krusader when I expect some differences, as with synchronising music or photo collections.

  3. #3
    Join Date
    Jan 2009
    Beans
    244
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: Checking backed up data for corruption?

    Quote Originally Posted by StuartN View Post
    - run rsync with the -n (or --dry-run) flag to report any differences between source and destination
    I get errors when trying to compare directories, even after adding quotations around files with spaces in the names it just says "skipping directory .".
    Quote Originally Posted by StuartN View Post
    - use Meld for graphical directory comparison
    Meld will only compare files, not directories. I have over 20,000 files and the only way I'd ever be able to compare them would be to make them zip archives and compare those files. But that's too much work. Also, everytime I go to compare two files (not directories) I get an error saying that my files "contain ascii nulls or perhaps it is a binary file".
    Quote Originally Posted by StuartN View Post
    - use Krusader's synchronise tool with compare by content
    I keep getting "could not parse output" errors. But I used Kdiff and it seemed to work, although it would not compare directories either.
    Quote Originally Posted by StuartN View Post
    - run diff, which has no output when the files match

    (and many more ways). My preference is rsync if I expect an absolute match, as with backup, and Krusader when I expect some differences, as with synchronising music or photo collections.
    When comparing small directories it seems to work, when comparing large directories I just get a bunch of messages about common subdirectories.
    CPU: AMD Phenom II X4 965 @ 3.5GHz
    GPU: MSI GTX 260 w/896MB DDR3 @ 655MHz
    RAM: Kingston HyperX 8GB (4x 2GB) DDR3 2000
    HDD: Samsung Spinpoint F3 1TB (2x fake RAID0)

  4. #4
    Join Date
    Apr 2009
    Beans
    1,173

    Re: Checking backed up data for corruption?

    Quote Originally Posted by blueshiftoverwatch View Post
    I get errors ...
    I get a detailed list of every different file, plus the type of change required, using this:
    Code:
    rsync --dry-run -avi source/ destination/
    Meld has a compare directories function that will compare a large file hierarchy and present a tree showing where different files occur, like a graphical version of the rsync command. Krusader's synchronise presents the same list, but allows you to operate on the list contents to correct differences, or to save the synchronisation as a profile for reuse. Diff, as I said, gives no output when files match.

    All these commands have good manual pages or online help.

  5. #5
    Join Date
    Jan 2009
    Beans
    244
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: Checking backed up data for corruption?

    I just spent the last 7 hours using the GTK frontend for Meld to copy 190GB worth of data from my external hard drive to the backup partition of my computer. If this were done by just copying the files it would have taken much less time, so I'm assuming this is because it not only copied the data but verified it bit for bit. Although I haven't read any of the detailed manual pages for any of the apps you listed as this process used up 100% of my RAM, so doing anything else while that was running was pretty much out of the question.
    Quote Originally Posted by StuartN View Post
    The hardware and OS will inform you of any error with an IO Error warning. I assume (and have never seen evidence to convince me otherwise) that the destination is a correct copy if I do not receive an error.
    I have reason to believe that isn't the case. A few months ago I was copying my music collection from my external hard drive onto my PS3. After the transfer was complete I had instances where I would go to select an album and if there were 12 songs, the first song would be listed 4 times, erasing songs 2-4 in the process. After looking through the music that had been transferred that happened more than a few times.
    CPU: AMD Phenom II X4 965 @ 3.5GHz
    GPU: MSI GTX 260 w/896MB DDR3 @ 655MHz
    RAM: Kingston HyperX 8GB (4x 2GB) DDR3 2000
    HDD: Samsung Spinpoint F3 1TB (2x fake RAID0)

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •