Seemingly sporadic slow ZFS IO since 22.04

**tkae-lp** · February 9th, 2024

Finally.... restoring took quite a bit longer than the backup:

Backup:

Code:

11.4TiB 24:48:24 [ 134MiB/s]

Restore:

Code:

11.4TiB 31:40:01 [ 105MiB/s]

After reassigning mount points, exporting the backup and exporting/importing the pool...

Code:

  pool: Tank
 state: ONLINE
config:

    NAME                                 STATE     READ WRITE CKSUM
    Tank                                 ONLINE       0     0     0
      raidz2-0                           ONLINE       0     0     0
        ata-ST4000DM000-1F2168_S300XXXX  ONLINE       0     0     0
        ata-ST4000DM004-2CV104_ZTT4XXXX  ONLINE       0     0     0
        ata-ST4000DM004-2CV104_ZTT4XXXX  ONLINE       0     0     0
        ata-ST4000DM000-1F2168_W300XXXX  ONLINE       0     0     0
        ata-ST4000DM000-1F2168_W300XXXX  ONLINE       0     0     0
        ata-ST4000DM000-1F2168_W300XXXX  ONLINE       0     0     0
        ata-ST4000DM000-1F2168_W300XXXX  ONLINE       0     0     0
        ata-ST4000DM000-1F2168_W300XXXX  ONLINE       0     0     0
    logs    
      nvme0n1p2                          ONLINE       0     0     0
    cache
      nvme0n1p1                          ONLINE       0     0     0

errors: No known data errors

Code:

NAME                                  SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Tank                                 29.1T  16.0T  13.1T        -         -     0%    54%  1.00x    ONLINE  -
  raidz2-0                           29.1T  16.0T  13.1T        -         -     0%  54.8%      -    ONLINE
    ata-ST4000DM000-1F2168_S300XXXX      -      -      -        -         -      -      -      -    ONLINE
    ata-ST4000DM004-2CV104_ZTT4XXXX      -      -      -        -         -      -      -      -    ONLINE
    ata-ST4000DM004-2CV104_ZTT4XXXX      -      -      -        -         -      -      -      -    ONLINE
    ata-ST4000DM000-1F2168_W300XXXX      -      -      -        -         -      -      -      -    ONLINE
    ata-ST4000DM000-1F2168_W300XXXX      -      -      -        -         -      -      -      -    ONLINE
    ata-ST4000DM000-1F2168_W300XXXX      -      -      -        -         -      -      -      -    ONLINE
    ata-ST4000DM000-1F2168_W300XXXX      -      -      -        -         -      -      -      -    ONLINE
    ata-ST4000DM000-1F2168_W300XXXX      -      -      -        -         -      -      -      -    ONLINE
logs                                     -      -      -        -         -      -      -      -  -
  nvme0n1p2                           928G      0   928G        -         -     0%  0.00%      -    ONLINE
cache                                    -      -      -        -         -      -      -      -  -
  nvme0n1p1                           931G     8K   931G        -         -     0%  0.00%      -    ONLINE

I guess now I just use the pool as normal and monitor to see if the same issue occurs! Crossing all fingers!

Edit:

Code:

   READ: bw=4448MiB/s (4664MB/s), 4448MiB/s-4448MiB/s (4664MB/s-4664MB/s), io=10.0GiB (10.7GB), run=2302-2302msec

Code:

  WRITE: bw=808MiB/s (847MB/s), 808MiB/s-808MiB/s (847MB/s-847MB/s), io=10.0GiB (10.7GB), run=12678-12678msec

**MAFoElffen** · February 11th, 2024

Dang. That took a while. I was wondering if everything was okay...

Crossing fingers with you!!!

**tkae-lp** · February 20th, 2024

Originally Posted by MAFoElffen

Dang. That took a while. I was wondering if everything was okay...

Crossing fingers with you!!!

Thanks! Yeah it did a bit.... glad I didn't stick with rsync, it was still looking at what to do 5 hours in! I'm a ZFS send/recv convert lol

Sorry I haven't replied sooner - the last week has been busy (schools were off in the UK).

I am still monitoring and so far things seem to be good:

Code:

WRITE: bw=1552MiB/s (1627MB/s), 1552MiB/s-1552MiB/s (1627MB/s-1627MB/s), io=10.0GiB (10.7GB), run=6600-6600msec

Code:

READ: bw=3417MiB/s (3583MB/s), 3417MiB/s-3417MiB/s (3583MB/s-3583MB/s), io=10.0GiB (10.7GB), run=2997-2997msec

Watch this space.....

**tkae-lp** · April 18th, 2024

Right so hello again!

My apologies for taking almost 2 months to reply this time, but I think we can probably all agree that I have given it more than enough time to test lol

The good news is that I haven't had any issues since my last post. *yay* none at all

everything is playing like a champ.

So this time I think we can definitely mark this as solved.

I would like to thank both of you again for all your time and effort troubleshooting this, it means a lot and I thoroughly appreciate it. It's been total bliss clicking on something and it actually playing without constantly stopping!

I have also learnt a great deal throughout this endeavour and I would also like to thank you both for that again also.

I'm looking forward to ~~Numbat~~ Noble, and I will see you guys for the next adventure

**#&thj^%** · April 20th, 2024

We very much appreciate the kind feedback, and MAFoElffen did most of the heavy lifting here.

As far as 24.04 LTS goes I'd wait it out for a few months after it's release before *any* upgrade plans, this one has proven to have way too many moving pieces in progress.
Just a heads up is all.

**tkae-lp** · April 23rd, 2024

No problem

regardless of how much heavy lifting was done, I still appreciate you taking your free time to help me out

without people like you on forums, I'd be scratching my head and sobbing in a corner!

Thanks for the heads up - in fact, when everything is working, it's usually the EoL prompt that causes me to upgrade (as it was with Bionic to Jammy) so I'll most likely be on Jammy for a while before moving to Noble!

Edit:

I've come across a couple of other forum/reddit posts with users of the Asus variant of this board (X99WS) having issues with memory being recognised - Usually RDIMMs (like I dealt with here) so since I had such an issue with my ASRock variant of the X99 chip set, I will post this here again for others if they come across it:

You absolutely need to match manufacturer, rank and model number and in most cases part number. If your part numbers are one digit out (different at the end), ie. G, J, K like the micron modules I listed in this thread, you absolutely must make sure that like for like part numbers are paired in the recommended module add/placement order in the manual or at the very least quad channel will not work - and worst case they probably will not be detected. After correcting modules and positioning, a CMOS reset by means of the on board jumper must be performed. Et Voila!