Page 2 of 2 FirstFirst 12
Results 11 to 15 of 15

Thread: Why does rsync slow down?

  1. #11
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    3,026
    Distro
    Ubuntu Development Release

    Re: Why does rsync slow down?

    I don't think there is anything wrong. As previous posters have mentioned, speed changes after buffers backup.

    For large file transfers, magnetic disk to different magnetic disk, I get:
    . ~175 megabytes per second, which is the disk read rate.
    . then the buffers backup, and the rate drops to ~85 to 90 megabytes per second.

    For many many small file (1024 bytes) transfers, magnetic disk to different magnetic disk, I get:
    . 1 megabyte per second for about 159 files.
    . then 500 kilo bytes per second until file 207
    . then 333 kilo bytes per second until file 246
    . then 250 kilo bytes per second until file 304
    . at this point the buffers are fully backup up.
    . then 3.26 kilo bytes per second on an exponential decay curve to 0.88 kilo bytes per second by file 40,000.

    EDIT: Note calculations done from rsync log file. Example:
    Code:
    ab00039997.txt
              1,024 100%    0.88kB/s    0:00:01 (xfr#39998, to-chk=2/40000)
    ab00039998.txt
              1,024 100%    0.88kB/s    0:00:01 (xfr#39999, to-chk=1/40000)
    ab00039999.txt
              1,024 100%    0.88kB/s    0:00:01 (xfr#40000, to-chk=0/40000)
    Yet the entire thing took only:

    Code:
    doug@s19:/media/sda$ time rsync -av --progress /media/sda/ab* /media/sdb > ~/bla5
    
    real    0m1.180s
    user    0m0.450s
    sys     0m0.930s
    So something is wrong with the math.
    Last edited by Doug S; June 25th, 2021 at 04:47 AM.
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

  2. #12
    Join Date
    Oct 2005
    Location
    Lab, Slovakia
    Beans
    10,432

    Re: Why does rsync slow down?

    You can use ionice on every rsync instance. You can also run rsync, watch it slow down, then run ionice and see whether it speeds up.

    There were bugs in old USB drivers (about a decade ago) that were very much improved with a default ionice (no parameters needed) on a copy action. Maybe you have one of those?

  3. #13
    Join Date
    Mar 2007
    Beans
    1,116

    Re: Why does rsync slow down?

    OK I've tried the ionice command and there was no noticeable change.

    Frequently checking the transfer of a large file (about 4GB) you can see that in the 6 second between 12:58:49 and 12:58:55 the file size increased by 262,144 bytes or a little over 43K per second.

    Code:
    # date; ls -l  .??*
    Fri Jun 25 12:58:49 CDT 2021
    -rw------- 1 root root 2438987776 Jun 25 12:58 .files.tgz.rw0UpH
    # date; ls -l  .??*
    Fri Jun 25 12:58:55 CDT 2021
    -rw------- 1 root root 2439249920 Jun 25 12:58 .files.tgz.rw0UpH
    # date; ls -l  .??*
    Fri Jun 25 12:59:04 CDT 2021
    -rw------- 1 root root 2439249920 Jun 25 12:58 .files.tgz.rw0UpH
    Note that from 12:58:55 to 12:59:04 there was no increase in size so the average rate would only be a little more than 17K/sec.

    People keep mentions buffers, but which buffers are they talking about. The buffers(cache) in the Hard Drive, the OS or somewhere else? The internal transfer rate for most hard drives is much higher than these numbers and assuming the buffers are operating efficiently the overall transfer should be higher. Even a USB2 connection can routinely operate at several megabytes/second.

    Adding another data point after allowing the transfer to continue;

    Code:
    # date; ls -l  .??*     Fri Jun 25 13:14:29 CDT 2021
    -rw------- 1 root root 2447114240 Jun 25 13:14 .files.tgz.rw0UpH
    This amounts to an elapsed time of 15:34 or 934 seconds resulting in a transfer rate of 8.7K/second. The longer time should allow for a closer approximation of the average transfer rate.

  4. #14
    Join Date
    Nov 2011
    Location
    /dev/root
    Beans
    Hidden!

    Re: Why does rsync slow down?

    It might be interesting to check, if you have the same slowdown in a live Ubuntu system (booted from USB) as you have in your installed system.

  5. #15
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    3,026
    Distro
    Ubuntu Development Release

    Re: Why does rsync slow down?

    Quote Originally Posted by rsteinmetz70112 View Post
    OK I've tried the ionice command and there was no noticeable change.
    me either.
    Quote Originally Posted by rsteinmetz70112 View Post
    People keep mentions buffers, but which buffers are they talking about. The buffers(cache) in the Hard Drive, the OS or somewhere else? The internal transfer rate for most hard drives is much higher than these numbers and assuming the buffers are operating efficiently the overall transfer should be higher. Even a USB2 connection can routinely operate at several megabytes/second.
    I am just talking about any buffers in the chain of things between sending the data and it actually getting onto the storage device, which is nowhere near my ram cash limits.
    I had not realized until the numbers you gave, that your issue was so severe.
    Example from my computer:
    Attached Images Attached Images
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

Page 2 of 2 FirstFirst 12

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •