Very slow speeds (185MB/s) when copying from one internal NVME to another

[shorthaul]

I am seeing incredibly slow speeds when copying from one internal NVME to another internal NVME. I am getting ~185 MB/s instead of an expected 1,000 to 3,500 MB/s, a very significant difference.

System specs:
Ubuntu 24.04.1
MB: Asus Proart X670E-Creator (PCIe Gen5) - latest available BIOS
CPU: Ryzen 7700X
Memory: 32GB
I have a video card in the x16 slot, no other PCIE slots used

NVME 1: Crucial P3 4TB Gen3 (~3,500 MB/s read speeds) - firmware updated
NVME 2: WD Black 8TB SN850X Gen4 (~6,600 MB/s write speeds) - firmware updated

Both NVME drives are formatted with EXT4 and SMART looks normal for both. I am trying to copy a large 3.5TB file from NVME 1 to NVME 2 by dragging and dropping the file in Nautilus. Speeds start out at 330 MB/s (instead of an expected ~3,500 MB/s) for a couple of minutes, then it drops to about 185 MB/s and continues at that speed. After the cache on the drive fills it should be able to maintain a solid 1,000 MB/s, about 5 times faster than what I am getting. A review of this drive from Toms Hardware shows that after the cache fills the drive should be able to maintain write speeds of ~1,000 MB/s:










The board has 2 NVME slots off of the CPU and 2 from the chipset. The Crucial NVME is in one of the slots from the CPU. I’ve tried putting the WD NVME in the other slots but it didn’t make a difference in the speeds. Any thoughts as why the copy speed is so slow?

[him610]

Any thoughts as why the copy speed is so slow?

IMOH, the buffers fill up.
A user in real world will never see speeds as fast as controlled lab-type tests.

[TheFu]

I'd look at the motherboard architecture and settings. Sometimes where a GPU is placed or how other HDD SATA/port settings are set can take away from other parts of a shared bus. In Asus MB manuals, I've seen have 2 tables which spell out the complexities of the bus setups.

[1fallen]

A user in real world will never see speeds as fast as controlled lab-type tests.

Very true.

@shorthaul, is there a reason you choose ext4?

If speed is a big factor, maybe read about BTRFS or better yet ZFS.

Also Drag and Drop, isn't in my habits, I mean nothing wrong with it but there are better ways.
1.The old "cp" average so so.
2. "rsync" is one part of my go to's

And Hardware is Key.

[TheFu]

Also Drag and Drop, isn't in my habits, I mean nothing wrong with it but there are better ways.
1.The old "cp" average so so.
2. "rsync" is one part of my go to's

The more GUI stuff there is in the middle, the more likely things seem to slow down.

Definitely run tests using cp with the time and known file size so that MBps can be calculated over the entire transfer. rsync can do this too - just add --stats to the options (I think that's the option).

I'm not a fan of BTRFS and ZFS has some overhead, but I haven't tested performance with them. COW file systems should be faster for reads, but I can't see how writes would be any different. I'd be more likely to switch to XFS for data, not booting!!!!, than use BTRFS. ZFS for huge amounts of data can make sense too, but I'd need 6 HDDs before I considered it and a good way to backup all that data.

If 1fallen has some real-world data, that could be helpful. Whenever phoronix.com does performance comparisons between file systems, ext4 is respectable and for large disks, xfs has a slight lead. The reasons to use btrfs and zfs are not for performance. It is for the other things those volume managers+file systems provide. As always, specific workloads matter. https://www.phoronix.com/review/ubuntu1910-ext4-zfs
And BTRFS lags according to this: https://www.phoronix.com/review/linux-611-filesystems

[shorthaul]

IMOH, the buffers fill up.
A user in real world will never see speeds as fast as controlled lab-type tests.

Before the cache on the drive filled up I was seeing 300 MB/s, about 10 times slower than it should. There may be some difference b/w 'lab' and 'real world', but not 10 times difference. Something else has to be going on.

I'd look at the motherboard architecture and settings. Sometimes where a GPU is placed or how other HDD SATA/port settings are set can take away from other parts of a shared bus. In Asus MB manuals, I've seen have 2 tables which spell out the complexities of the bus setups.

There is only one of the four NVME slots that share bandwidth with one of the PCIe slots, and that's only if there is a device in the PCIe slot. I am not using that slot and don't have a device in that PCIe slot either, so it isn't that.

Very true.

@shorthaul, is there a reason you choose ext4?

Not really, it's how the drive I'm reading from is formatted and was the default, so just went with that. Would ext4 cause a 10x drop in expected performance?

Once the current file copy finishes I will try using dd and bypass the file system cache and see if that makes a difference.

[shorthaul]

I tested using dd to bypass the file system cache but I'm still getting very slow speeds, even from the very start of the copy process when the cache isn't full. I used this command and it still was around 185 MB/s transfer speeds.

dd if=/path/to/source of=/path/to/destination bs=128M count=4000 oflag=direct status=progress

[brcisna-u]

@shorthaul

Is the second drive like a storage tank for your setup,or are the two drives Ubuntu used ?
If second drive is a storage tank what happens if you take that drive out,, then do a benchmark on your system drive?
I always just use the gnome-disks utility and do a benchmark from there,just because it is convenient and you are comparing apples to apples each time you run it.
If your performance goes up to about 2Gbps with system drive only,, swap that drive to the opposite slot and repeat.

[1fallen]

I'll start this with just basics, I'll copy a file un-compressed and the size is "6.1 GiB (6,506,741,844)1,105 files, 136 sub-folders"
That took:

 0:03:28 [21.7MiB/s] [==============================================================>] 100%

Now I'll compress that same folder and now:

 pv '/home/me/Downloads/1.All-DL'\''s.tar.gz'  > /media/me/WD-Blue-1TB/Downloads 
 643MiB 0:00:13 [47.1MiB/s] [==============================================================>] 100%      

The destination drive is ext4, the source drive is BTRFS with /home compressed ie:

less /etc/fstab|grep /home
UUID=3213b2e0-07ed-4f08-b6a6-d1f0fdc83db4 /home          btrfs   subvol=/@home,defaults,compress-force=zstd:3 0 2

On my Tank Drive with ZFS is way faster:

# pv '/home/me/Downloads/1.All-DL'\''s.tar.gz'  > /tank/speedtest 
 643MiB 0:00:00 [1.86GiB/s] [==============================================================>] 100%  

# df -T /tank 
Filesystem     Type 1K-blocks     Used Available Use% Mounted on 
tank           zfs  939123328 71617536 867505792   8% /tank

That should help, I think anyway.

[TheFu]

Exact commands would be useful, I bet.
Mixing so many different types of storage muddies the waters massively too.

Best to keep it simple.
1. Simplify the environment.
2. Isolate the different parts with separate tests.
3. Determine the slowest part, then ....
4. Repeat.

dd if=/path/to/source of=/path/to/destination bs=128M count=4000 oflag=direct status=progress

That blocksize is huge. Try either 4K or 4M. Larger block sizes can keep other tasks out. Larger isn't always better. Especially for I/O tasks. Too small is bad too.