Very slow speeds (185MB/s) when copying from one internal NVME to another

[ubfan1]

The "slow copy" or "decrease of transfer rate when copying large amount of data" are known problems, with many askubuntu questions and launchpad bugs filed. Previous postings have covered most of the things to check, but to reiterate: 1) Your MB has two PCIE5 M.2 slots and two PCIE4 M.2 slots, use a fast one for your target.
2)Ensure your partitions/filesystems are aligned properly -- 1MB or even 4MB.
3)System buffers fill up when writes are slower than reads, the usual case. Things slow down when buffers fill. Using the nocache command on a terminal copy will at least prevent the input stream from being buffered.
4)The "time" command is usless for determining write speeds, the copying process finishes minutes before the system buffers get flushed. Force the flush with a sync or watch the blinking lights for disk activity to determine the actual finish of the copy.
5)Some copy commands allow you to specify a buffer size -- this may help in limiting memory fragmentation, speeding things up a bit.
6)There are tweaks that may help for specific cases -- different scheduler, altering dirty write bits or rations, but the filling of system buffers will eventually drag things to a crawl. Slowing the reads down to the write speeds is the goal of such alterations. Most of the suggestions for tweaks were for hard disks, SSDs are likely to have completely different optimizations.

Follow the previous suggestions for seeing what helps your situation the most. Understand, this is an unresolved problem that prevents some people from using Linux for their work -- they simply cannot back up their data sets in a timely manner. Good luck, and interested in seeing what you find helps you the most.

[Doug S]

Just out of interest and because I just so happen to have two nvme drives installed in my main test server at the moment, I tried to see what I get.

Motherboard: ASUS PRIME Z490-A
Memory: 32G
Processor: Intel(R) Core(TM) i5-10600K CPU @ 4.10GHz (turbo to 4.8 GHz)
OS: Ubuntu 24.04.1 (Server, no GUI)
Kernel: 6.12.0-rc7
Format: EXT4 (both NVME drives)
NVME 1: WD_BLACK_SN850X_2000GB
NVME 2: WD_BLACK_SN850X_1000GB
CPU Frequency scaling driver and governor: intel_pstate / performance. HWP disabled.
Force CPU 5 only, to avoid CPU migration delays.

File: 536,870,912,000 random bytes.

Copy from NVME 1 to NVME 2: 1,416,545,942 bytes per second.
Copy from NVME 2 to NVME 1: 1,269,198,374 bytes per second.

Example of Script used:

doug@s19:/media/nvme/home/doug/tmp/tmp/tmp/tmp$ cat copy-big-pull
#!/bin/bash
date
taskset -c 5 cp ~/tmp/tmp/tmp/tmp/big ./
date
sync
date

EDIT: it doesn't seem to make a difference if the CPU frequency scaling governor is powersave and no forcing of CPU affinity.
Read only rate from:
NVME 1: 2791.103 Mega Bytes per second.
NVME 2: 1611.177 Mega Bytes per second.