Page 1 of 2 12 LastLast
Results 1 to 10 of 15

Thread: Why does rsync slow down?

  1. #1
    Join Date
    Mar 2007
    Beans
    1,116

    Why does rsync slow down?

    I've been using rsync to copy a lot of files between filesystems and it seems if copying a large amount of data that after running for a while rsync file transfers slow way down, they keep going but slow considerably.
    This seems to happen regardless of whether the transfer is over a network, the Internet or between two file systems on the same computer.
    I am using very basic commands like
    Code:
    rsync -av /file1/file3 /file2/file3
    Sometimes I use the "z" option when copying over the network.

    Looking at System Monitor I don't see excessive CPU or Memory consumption. Free shows plenty of RAM free and very little swap in use.

    What am I missing?

  2. #2
    Join Date
    Aug 2019
    Beans
    17

    Re: Why does rsync slow down?

    Copying a large number of small files will take longer and the small number of large files because the filesystem tables and metadata need to be updated for each file as well. Spinning disks can make that very slow. Also, -z will typically slow you down on a LAN due to the single threaded compression.

  3. #3
    Join Date
    Mar 2007
    Beans
    1,116

    Re: Why does rsync slow down?

    Thanks,

    What is happening is I am copying a mix of files and when I start the copy it moves along quickly with any size files but after a while it slows way down again with a mix of file sizes. This appears to happen regardless of whether there is a network involved or not.

  4. #4
    Join Date
    Oct 2005
    Location
    Lab, Slovakia
    Beans
    10,432

    Re: Why does rsync slow down?

    Buffers. In the beginning, it buffers to RAM, but eventually all buffers are full and what you see then is the real speed of the transfer.

  5. #5
    Join Date
    Nov 2011
    Location
    /dev/root
    Beans
    Hidden!

    Re: Why does rsync slow down?

    You can check the 'dirty' data which is not yet flushed from buffers with

    Code:
    grep -e 'Dirty:' /proc/meminfo
    I made a shellscript, watch-flush, that is part of the mkusb package. watch-flush watches the amount of dirty data at regular intervals and shows how the buffers are filled and flushed.
    Code:
    sudo add-apt-repository ppa:mkusb/ppa
    sudo apt install mkusb
    watch-flush
    Edit: You can tell the system to hurry up flushing the buffers with the command

    Code:
    sync
    Last edited by sudodus; June 23rd, 2021 at 11:02 AM.

  6. #6
    Join Date
    Mar 2007
    Beans
    1,116

    Re: Why does rsync slow down?

    Running the command it doesn't seem to be filling up buffers. and they seem to be flushed pretty quickly.

    Code:
    # grep -e 'Dirty:' /proc/meminfo
    Dirty:               576 kB
    # grep -e 'Dirty:' /proc/meminfo
    Dirty:               300 kB
    # grep -e 'Dirty:' /proc/meminfo
    Dirty:               300 kB
    # grep -e 'Dirty:' /proc/meminfo
    Dirty:                 0 kB
    Sync runs almost instantly.

    Code:
    # free
                  total        used        free      shared  buff/cache   available
    Mem:       16319620     2693748    12664452       30064      961420    12323888
    Swap:       8388604        4908     8383696
    # free -h
                  total        used        free      shared  buff/cache   available
    Mem:            15G        2.6G         12G         29M        940M         11G
    Swap:          8.0G        4.8M        8.0G
    free shows lots of memory available

    None of this seems to increase the copy speed.

  7. #7
    Join Date
    Oct 2005
    Location
    Lab, Slovakia
    Beans
    10,432

    Re: Why does rsync slow down?

    I wonder whether the command ionice will make a difference.

    Something like
    # ionice-c 2 -n 0 rsync yadda yadda
    Or
    $ sudo ionice -p PID

    I have seen in the past that ionice can clear up io access inefficiencies very effectively.
    Last edited by HermanAB; June 24th, 2021 at 06:53 AM.

  8. #8
    Join Date
    Mar 2007
    Beans
    1,116

    Re: Why does rsync slow down?

    In my current attempt to transfer files between two drives on the same computer rsync runs 3 processes all of them seem to be started by the original PID 3404

    Code:
    # ps -ef|grep rsync
    root      3423  3404  0 Jun21 pts/0    00:03:54 rsync -av /media/rob /backup
    root      3424  3423  0 Jun21 pts/0    00:00:00 rsync -av /media/rob /backup
    root      3689  3424  0 Jun21 pts/0    00:05:03 rsync -av /media/rob /backup
    So which PID would you use for ionice?
    Or would yadda yadda = rsync -av /media/rob /backup

    The copy process is proceeding slowly so I'm reluctant to start over and lose any large files currently in process although it's been running for days now - note the date on the process above.
    Last edited by rsteinmetz70112; June 24th, 2021 at 05:48 PM.

  9. #9
    Join Date
    Nov 2011
    Location
    /dev/root
    Beans
    Hidden!

    Re: Why does rsync slow down?

    Have you checked with top or htop if rsync is busy or maybe waiting for some other process?

    You can also [install and start] iotop in order to watch the write progress and what process(es) that are involved.

    Code:
    sudo iotop -od2
    What kind of drives are you reading from and writing to? How are they connected (SATA, eSATA, USB2, USB3, NVMe)?

    I really don't know what is slowing down your process, so I am guessing here: Could there be hardware problems (for example that the hard drive has problems writing some sectors)?

  10. #10
    Join Date
    Mar 2007
    Beans
    1,116

    Re: Why does rsync slow down?

    I've checked top and System Monitor neither show a significant loard on the computer., I haven't tried iotop.
    The drives are both external connected vi USB ports, but I've experienced similar issues with transfers over the Internet and over the LAN.
    There could be a problem with the drive I'm copying from as it is old and may have issues. The other drive is practically new and is showing not signs of problems.
    There could be a hardware problem somewhere else.

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •