Results 1 to 6 of 6

Thread: Compare copied files between server and NAS

  1. #1
    Join Date
    Apr 2016
    Beans
    330

    Compare copied files between server and NAS

    Hello,

    I used the folling command to copy folder and files between server (ubuntu) and Qnap:

    Code:
    $ scp -rvp data/ user@192.168.1.50:/share/Public
    I want to know if all files have been successfully copied. Through windows I consulted the Properties window (using the right mouse button) and the result was as follows:

    --Server Ubuntu
    Size: 3.07 TB (3,384,578,889,263 bytes)
    Size on disk: 4.57 TB (5,027,498,351,616 bytes)
    Contains: 1,797,813 Files, 169,392 Folders

    --Qnap
    Size: 2.50 TB (2,757,571,816,744 bytes)
    Size on disk: 2.51 TB (2,761,236,037,632 bytes)
    Contains: 1,646,925 Files, 163,526 Folders

    I want to identify if this difference is due to some windows limitation or if not all files were copied after all.
    I searched for web solutions and read about the "test", "diff", "rsync" program, but I don't know if these are the best options for manipulating this amount of data.

    Thanks

  2. #2
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: Compare copied files between server and NAS

    rsync is what I'd use. Actually, I would have used rsync to perform the copy.
    Code:
    rsync -avz data user@192.168.1.50:/share/Public/
    rsync is picky about trailing '/' characters. My version above will create a "/share/Public/data/" directory. If you don't want the 'data' directory, replace 'data' with 'data/' in the command.
    I don't think the trailing '/' on the target is important, but I always add it for clarity.

    If the 'user' matches on both systems, then it isn't needed for the rsync or the scp.
    rsync uses ssh for authentication, so if ssh-keys have been setup, then rsync will authenticate like scp, sftp, ssh.

    rsync has test options, progress and statistics options. Just depends on what you'd like as output.

  3. #3
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,167
    Distro
    Ubuntu 18.04 Bionic Beaver

    Re: Compare copied files between server and NAS

    I'm with TheFu on this one. rsync can do the copy for you AND give you a verification report once the job is done so you can have peace of mind that it "actually" finished the entire task. Another nice thing is that if you want files to be identical on both systems, it can do that as well and ONLY copy the differences each time it runs...not the entire data set so it does not matter if you are adding a few megabytes to a terabyte repository.

    LHammonds

  4. #4
    Join Date
    Apr 2016
    Beans
    330

    Re: Compare copied files between server and NAS

    Hello,

    I executed this command:
    Code:
    sudo scp -rvp data/ user@192.168.1.50:/share/Public
    I want to make sure all files have been copied. I am trying execute this command:
    Code:
    sudo diff -ryq user@192.168.1.50:/share/Public/data/ /home/user/data/
    But I received this message:
    Code:
    diff: admin@192.168.1.4:/share/Public/data/: No such file or directory
    I was searching about this behaviour and I read about symlinks. So I tried to execute the command with the parameter:
    Code:
    sudo diff -ryq --no-dereference user@192.168.1.50:/share/Public/data/ /home/user/data/
    But the result it is the same. How can I handle this problem?

    Note: http://man7.org/linux/man-pages/man1/diff.1.html
    Regarding the ours suggestion I had not seen before trying again with the scp and diff program. But how can I generate a report with rsync? I've been reading the manual and searching but haven't found a command where the output was a server and client comparison report. (https://linux.die.net/man/1/rsync)

    Thanks

  5. #5
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: Compare copied files between server and NAS

    Use rsync instead of scp.

    If you don't want to actually copy any files, use the --dry-run option with all the normal options to rsync.

    I didn't know that diff supported remote system parameters and don't see any mention of that capability in the diff manpage. diff doesn't support --no-dereference according to the manpage either. Can't just make up options.

    Instead of looking at manpages on the internet, use the ones actually installed on YOUR system. It does take a few months to learn to read manpages, but once you do, they are usually brilliant. The layout is what someone who understands manpages needs. Summary at the top, to help people already familiar with a specific command, then more details as we get farther into the manpage.

    Anyway, running rsync shouldn't cause any harm if the directories are already all copied over.

    Might add --status --progress options to the already provided options.

  6. #6
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    2,766
    Distro
    Ubuntu Development Release

    Re: Compare copied files between server and NAS

    Quote Originally Posted by sed_faster View Post
    But how can I generate a report with rsync? I've been reading the manual and searching but haven't found a command where the output was a server and client comparison report. (https://linux.die.net/man/1/rsync)
    It sounds as though you want more than the standard output:
    Code:
    doug@s15:~/backup_smythies.com_03$ rsync --delete --archive --verbose --dry-run --progress --exclude 'ext-20*\.bin*' --exclude 'int-20*\.bin*' --exclude 'big' doug@smythies.com:/home/doug ./
    
    receiving incremental file list
    doug/
    doug/.bash_history
    doug/access.log
    doug/access.log.1
    doug/access.log.2
    doug/access.log.3
    doug/auth.log
    doug/auth.log.1
    doug/error.log
    doug/error.log.1
    doug/error.log.2
    doug/error.log.3
    doug/mail.err
    doug/mail.log
    doug/mail.log.1
    doug/zz.log
    doug/zz.txt
    doug/tcpdump/109/
    doug/test2/zzxxtest.data
    
    sent 3,914 bytes  received 20,239,890 bytes  941,572.28 bytes/sec
    total size is 117,705,867,687  speedup is 5,814.41 (DRY RUN)
    You can play with verbosity to get way way more information, if you want:

    Code:
    doug@s15:~/backup_smythies.com_03$ rsync --delete --archive -vv --dry-run --exclude 'ext-20*\.bin*' --exclude 'int-20*\.bin*' --exclude 'big' doug@smythies.com:/home/doug ./
    receiving incremental file list
    delta-transmission enabled
    doug/.bash_logout is uptodate
    doug/.bashrc is uptodate
    doug/.bzr.log is uptodate
    doug/.htkes is uptodate
    ...
    doug/config/etc/bind/named.conf.local is uptodate
    doug/config/etc/bind/named.conf.local.001 is uptodate
    doug/config/etc/bind/named.conf.local.doug is uptodate
    doug/config/etc/bind/named.conf.local.original is uptodate
    doug/config/etc/bind/named.conf.options is uptodate
    doug/config/etc/bind/named.conf.options.doug is uptodate
    doug/config/etc/bind/named.conf.options.doug1 is uptodate
    doug/config/etc/bind/named.conf.options.original is uptodate
    doug/config/etc/bind/named.conf.options.test is uptodate
    ...
    doug/public_html/linux/single-threaded/sweep/power.png is uptodate
    doug/public_html/linux/single-threaded/sweep/teo.csv is uptodate
    doug/public_html/linux/sort_order/index.html is uptodate
    doug/public_html/linux/ubuntu-docs/index.html is uptodate
    doug/public_html/linux/ubuntu-docs/langs.htm is uptodate
    doug/public_html/linux/ubuntu-docs/load-trace.html is uptodate
    doug/public_html/linux/ubuntu-docs/upstream_yelp.html is uptodate
    doug/public_html/linux/ubuntu-docs/help.ubuntu.com/compile_desktop_help.txt is uptodate
    ...
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •