Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: Fastest way to copy large amount of files? Rsync?

  1. #1
    Join Date
    Apr 2012
    Beans
    150
    Distro
    Ubuntu 13.04 Raring Ringtail

    Fastest way to copy large amount of files? Rsync?

    I'd like to copy a folder with all subfolders/files in it from partition sdc1 to partition sdb1. File permissions do not have to be kept as I copy from an NTFS drive. Nautilus will do the job, of course. But I wonder if there is a faster method to copy a large amount of data, assumably via command line? If possible, it should also give feedback about the progress. I had a look at rsync which seems to be a good choice. However, I wonder what's the fastest way.

  2. #2
    Join Date
    Feb 2009
    Location
    Dallas, TX
    Beans
    7,790
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: Fastest way to copy large amount of files? Rsync?

    Hi MrsUser.

    You can use 'cp' or 'rsync'. I don't think there's much performance difference between them (when rsync is used for coping). If I have to guess, I'd say cp is faster.

    A copy using 'cp':
    Code:
    cp -ivpr source/ destination/
    However, for a large directory tree, I would always prefer rsync because in case of any error, recovering with rsync would be a breeze (you just run it again).

    A typical'rsync' command would be:
    Code:
    rsync -av source/ destination/
    Hope it helps. Let us know how it goes.
    Regards.

  3. #3
    Join Date
    Jan 2010
    Beans
    261
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Fastest way to copy large amount of files? Rsync?

    The fastest way is not really limited by software, as nautilus is only an overlay of the command you'd use in a terminal anyway. GUI apps are all just that, running commands and scripts while you see yourself doing something else graphically.

    The fastest way would prbably be though nautilus so you can at least see what's going on and progress, but a simple "mv /pathtofile/tocopy/ormove /pathto/destination" will also do the trick, but in the background that's the command nautilus uses when you say "move to folder."

    live media might be faster since both the copied/moved file and destination are mounts and not the OS though... and flash will always be much faster than a rotational disk. Hardware is what limits this in the end, software can only be tuned to use the best efficiency of the hardware. a 6.0GB/s SSD will never be able to transfer at 6GB/s (relative speed) if the controller is maxed at Gen 2 3GB/s (relative speed)...

    technology is awesome.

    Oh, besides though, fastest means more room for error. Naultilus or a "mv" or "cp" command will be just fine and leave very little room for error as long as the system stays on. Faster than what they choose to move at may lead to serious corruption.

    rsync is something I don't use, but it wouldn't be any better or faster than what nautilus can do.
    When your entire life is measured in one moment, you change the concept of time.

  4. #4
    Join Date
    Jun 2006
    Location
    Brisbane Australia
    Beans
    713

    Re: Fastest way to copy large amount of files? Rsync?

    Quote Originally Posted by papibe View Post
    A typical'rsync' command would be:
    Code:
    rsync -av source/ destination/
    Actually I would instead say a typical rsync command would be:
    Code:
    rsync -av --delete source/ destination/
    I.e. often/normally you are intending a duplicated copy of the source dir so you also want files that were deleted in the source dir to be deleted in the destination dir. Unless it is a one off copy, rsync is better than cp because it only copies the changes and works the same over a network.

    Note when I type the above rsync command manually I always add a "-n" the first go to check what it will do before it does anything.

  5. #5
    Join Date
    Apr 2012
    Beans
    150
    Distro
    Ubuntu 13.04 Raring Ringtail

    Re: Fastest way to copy large amount of files? Rsync?

    I'll give rsync a try:
    Code:
    rsync -av --progress '/path to/the source folder' '/path to/the destination folder'
    The -n option is also very useful, by the way. Nice to see what will be copied before actually copying.

    Code:
    rsync -n -av --progress '/path to/the source folder' '/path to/the destination folder'
    And as you can see I put the path as a string by using " ' ". I just tried it without and it didn't work, because some directory names have blanks. Then I put it as string and now it works.

    Thanks a lot to you all!

    -----------

    EDIT:

    Quote Originally Posted by papibe View Post
    However, for a large directory tree, I would always prefer rsync because in case of any error, recovering with rsync would be a breeze (you just run it again).

    Just out of curiosity -- how does rsync keep track of already copied files? And do I not have to use the -P option then?
    Last edited by MrsUser; February 22nd, 2013 at 12:00 AM.

  6. #6
    Join Date
    Feb 2005
    Location
    Melbourne, Australia
    Beans
    13,510
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Fastest way to copy large amount of files? Rsync?

    Quote Originally Posted by MrsUser View Post
    I'd like to copy a folder with all subfolders/files in it from partition sdc1 to partition sdb1. File permissions do not have to be kept as I copy from an NTFS drive. Nautilus will do the job, of course. But I wonder if there is a faster method to copy a large amount of data, assumably via command line? If possible, it should also give feedback about the progress. I had a look at rsync which seems to be a good choice. However, I wonder what's the fastest way.
    Copying anything to/from NTFS in Linux is compromised because it is a non-Linux filesystem. People continually report that NTFS file IO is way slower than native Linux IO.
    Regards, David.
    Please use the Forum search and Wiki search for immediate help
    Please mark your thread as Solved when appropriate
    New to technical forums?: How To Ask Questions The Smart Way

  7. #7
    Join Date
    Feb 2009
    Location
    Dallas, TX
    Beans
    7,790
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: Fastest way to copy large amount of files? Rsync?

    Quote Originally Posted by MrsUser View Post
    how does rsync keep track of already copied files?
    It only does it on the fly while it's being run. The basic algorithm is based in filename, size, and modification date (although it can be tweak a bit).

    Quote Originally Posted by MrsUser View Post
    And do I not have to use the -P option then?
    Option -P implies two options --partial and --progress.

    'progress' works wonders when you are transferring big files. If you are transferring lots of medium to small files, I would recommend using only the verbose option (-v).

    'partial' is always useful. It doesn't add too much value if you are transferring small to medium size files. Its true value applies when transferring big files and/or when transferring over a network.

    Does that help?
    Regards.

  8. #8
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 22.04 Jammy Jellyfish

    Re: Fastest way to copy large amount of files? Rsync?

    Quote Originally Posted by papibe View Post
    It only does it on the fly while it's being run. The basic algorithm is based in filename, size, and modification date (although it can be tweak a bit).
    Indeed. You can also tell rsync to do it by checksum:
    Code:
     -c, --checksum              skip based on checksum, not mod-time & size
    But it can be considerably slower due to having to calculator the checksum of the source and destination files.

    I've always run rsync like this:

    Code:
    rsync -ai --delete /path/to/source /path/to/destination
    I don't have quotes around anything and it seems to handle spaces just fine.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  9. #9
    Join Date
    Apr 2012
    Beans
    150
    Distro
    Ubuntu 13.04 Raring Ringtail

    Re: Fastest way to copy large amount of files? Rsync?

    Quote Originally Posted by CharlesA View Post
    I've always run rsync like this:
    Code:
    rsync -ai --delete /path/to/source /path/to/destination
    I don't have quotes around anything and it seems to handle spaces just fine.
    What's the -i parameter for?

  10. #10
    Join Date
    Feb 2013
    Beans
    Hidden!

    Re: Fastest way to copy large amount of files? Rsync?

    From man rsync
    -i, --itemize-changes
    output a change-summary for all updates

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •