Why does rsync slow down?

[Doug S]

I don't think there is anything wrong. As previous posters have mentioned, speed changes after buffers backup.

For large file transfers, magnetic disk to different magnetic disk, I get:
. ~175 megabytes per second, which is the disk read rate.
. then the buffers backup, and the rate drops to ~85 to 90 megabytes per second.

For many many small file (1024 bytes) transfers, magnetic disk to different magnetic disk, I get:
. 1 megabyte per second for about 159 files.
. then 500 kilo bytes per second until file 207
. then 333 kilo bytes per second until file 246
. then 250 kilo bytes per second until file 304
. at this point the buffers are fully backup up.
. then 3.26 kilo bytes per second on an exponential decay curve to 0.88 kilo bytes per second by file 40,000.

EDIT: Note calculations done from rsync log file. Example:

ab00039997.txt
          1,024 100%    0.88kB/s    0:00:01 (xfr#39998, to-chk=2/40000)
ab00039998.txt
          1,024 100%    0.88kB/s    0:00:01 (xfr#39999, to-chk=1/40000)
ab00039999.txt
          1,024 100%    0.88kB/s    0:00:01 (xfr#40000, to-chk=0/40000)

Yet the entire thing took only:

doug@s19:/media/sda$ time rsync -av --progress /media/sda/ab* /media/sdb > ~/bla5

real    0m1.180s
user    0m0.450s
sys     0m0.930s

So something is wrong with the math.

[HermanAB]

You can use ionice on every rsync instance. You can also run rsync, watch it slow down, then run ionice and see whether it speeds up.

There were bugs in old USB drivers (about a decade ago) that were very much improved with a default ionice (no parameters needed) on a copy action. Maybe you have one of those?

[rsteinmetz70112]

OK I've tried the ionice command and there was no noticeable change.

Frequently checking the transfer of a large file (about 4GB) you can see that in the 6 second between 12:58:49 and 12:58:55 the file size increased by 262,144 bytes or a little over 43K per second.

# date; ls -l  .??*
Fri Jun 25 12:58:49 CDT 2021
-rw------- 1 root root 2438987776 Jun 25 12:58 .files.tgz.rw0UpH
# date; ls -l  .??*
Fri Jun 25 12:58:55 CDT 2021
-rw------- 1 root root 2439249920 Jun 25 12:58 .files.tgz.rw0UpH
# date; ls -l  .??*
Fri Jun 25 12:59:04 CDT 2021
-rw------- 1 root root 2439249920 Jun 25 12:58 .files.tgz.rw0UpH

Note that from 12:58:55 to 12:59:04 there was no increase in size so the average rate would only be a little more than 17K/sec.

People keep mentions buffers, but which buffers are they talking about. The buffers(cache) in the Hard Drive, the OS or somewhere else? The internal transfer rate for most hard drives is much higher than these numbers and assuming the buffers are operating efficiently the overall transfer should be higher. Even a USB2 connection can routinely operate at several megabytes/second.

Adding another data point after allowing the transfer to continue;

# date; ls -l  .??*     Fri Jun 25 13:14:29 CDT 2021
-rw------- 1 root root 2447114240 Jun 25 13:14 .files.tgz.rw0UpH

This amounts to an elapsed time of 15:34 or 934 seconds resulting in a transfer rate of 8.7K/second. The longer time should allow for a closer approximation of the average transfer rate.

[sudodus]

It might be interesting to check, if you have the same slowdown in a live Ubuntu system (booted from USB) as you have in your installed system.

[Doug S]

OK I've tried the ionice command and there was no noticeable change.

me either.

People keep mentions buffers, but which buffers are they talking about. The buffers(cache) in the Hard Drive, the OS or somewhere else? The internal transfer rate for most hard drives is much higher than these numbers and assuming the buffers are operating efficiently the overall transfer should be higher. Even a USB2 connection can routinely operate at several megabytes/second.

I am just talking about any buffers in the chain of things between sending the data and it actually getting onto the storage device, which is nowhere near my ram cash limits.
I had not realized until the numbers you gave, that your issue was so severe.
Example from my computer: