Page 10 of 14 FirstFirst ... 89101112 ... LastLast
Results 91 to 100 of 136

Thread: Seemingly sporadic slow ZFS IO since 22.04

  1. #91
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Seemingly sporadic slow ZFS IO since 22.04

    Glad to hear everything is going so well.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  2. #92
    Join Date
    Nov 2023
    Beans
    76

    Re: Seemingly sporadic slow ZFS IO since 22.04

    I copied a file from one folder to another this evening (duplicate not move) and noticed it was slow... (20 secs for ~350MB)

    Code:
    fio --name TEST --eta-newline=5s --filename=temp.file --rw=write --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
    TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
    fio-3.28
    Starting 1 process
    Jobs: 1 (f=1): [W(1)][53.8%][w=107MiB/s][w=107 IOPS][eta 00m:06s]
    Jobs: 1 (f=1): [W(1)][59.1%][w=46.0MiB/s][w=46 IOPS][eta 00m:09s] 
    Jobs: 1 (f=1): [W(1)][59.4%][w=24.0MiB/s][w=24 IOPS][eta 00m:13s] 
    Jobs: 1 (f=1): [W(1)][61.0%][w=37.0MiB/s][w=37 IOPS][eta 00m:16s] 
    Jobs: 1 (f=1): [W(1)][63.3%][w=37.0MiB/s][w=37 IOPS][eta 00m:18s] 
    Jobs: 1 (f=1): [W(1)][66.1%][w=39.0MiB/s][w=39 IOPS][eta 00m:19s] 
    Jobs: 1 (f=1): [W(1)][71.7%][w=126MiB/s][w=126 IOPS][eta 00m:17s] 
    Jobs: 1 (f=1): [W(1)][81.7%][w=287MiB/s][w=287 IOPS][eta 00m:11s] 
    Jobs: 1 (f=1): [W(1)][98.2%][eta 00m:01s]                         
    Jobs: 1 (f=0): [f(1)][100.0%][w=271MiB/s][w=271 IOPS][eta 00m:00s]
    TEST: (groupid=0, jobs=1): err= 0: pid=3103281: Mon Jan 15 01:44:04 2024
      write: IOPS=174, BW=174MiB/s (183MB/s)(10.0GiB/58711msec); 0 zone resets
        slat (usec): min=191, max=61178, avg=5132.95, stdev=8453.58
        clat (usec): min=4, max=6125.3k, avg=176183.18, stdev=415840.98
         lat (usec): min=297, max=6126.8k, avg=181316.96, stdev=420835.36
        clat percentiles (msec):
         |  1.00th=[    8],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
         | 30.00th=[   10], 40.00th=[   15], 50.00th=[   53], 60.00th=[   81],
         | 70.00th=[  117], 80.00th=[  192], 90.00th=[  676], 95.00th=[  835],
         | 99.00th=[ 1083], 99.50th=[ 1284], 99.90th=[ 6141], 99.95th=[ 6141],
         | 99.99th=[ 6141]
       bw (  KiB/s): min=22528, max=4077568, per=100.00%, avg=196312.62, stdev=526535.26, samples=104
       iops        : min=   22, max= 3982, avg=191.71, stdev=514.19, samples=104
      lat (usec)   : 10=0.04%, 20=0.01%, 500=0.01%, 750=0.02%, 1000=0.03%
      lat (msec)   : 2=0.07%, 4=0.15%, 10=35.72%, 20=5.52%, 50=3.62%
      lat (msec)   : 100=19.67%, 250=18.72%, 500=4.99%, 750=1.90%, 1000=8.14%
      lat (msec)   : 2000=1.08%, >=2000=0.30%
      fsync/fdatasync/sync_file_range:
        sync (nsec): min=983, max=983, avg=983.00, stdev= 0.00
        sync percentiles (nsec):
         |  1.00th=[  980],  5.00th=[  980], 10.00th=[  980], 20.00th=[  980],
         | 30.00th=[  980], 40.00th=[  980], 50.00th=[  980], 60.00th=[  980],
         | 70.00th=[  980], 80.00th=[  980], 90.00th=[  980], 95.00th=[  980],
         | 99.00th=[  980], 99.50th=[  980], 99.90th=[  980], 99.95th=[  980],
         | 99.99th=[  980]
      cpu          : usr=1.02%, sys=13.79%, ctx=50057, majf=0, minf=16
      IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
         issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=32
    
    Run status group 0 (all jobs):
      WRITE: bw=174MiB/s (183MB/s), 174MiB/s-174MiB/s (183MB/s-183MB/s), io=10.0GiB (10.7GB), run=58711-58711msec
    Waiting 5 minutes and checking again gives:

    Code:
    fio --name TEST --eta-newline=5s --filename=temp.file --rw=write --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
    TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
    fio-3.28
    Starting 1 process
    Jobs: 1 (f=1): [W(1)][100.0%][eta 00m:00s]                          
    Jobs: 1 (f=1): [W(1)][100.0%][eta 00m:00s] 
    TEST: (groupid=0, jobs=1): err= 0: pid=3170929: Mon Jan 15 01:46:02 2024
      write: IOPS=1013, BW=1014MiB/s (1063MB/s)(10.0GiB/10102msec); 0 zone resets
        slat (usec): min=205, max=1925, avg=344.45, stdev=240.41
        clat (usec): min=3, max=6547.8k, avg=30515.32, stdev=359211.99
         lat (usec): min=260, max=6548.1k, avg=30860.17, stdev=359212.30
        clat percentiles (msec):
         |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[    9],
         | 30.00th=[    9], 40.00th=[    9], 50.00th=[    9], 60.00th=[    9],
         | 70.00th=[    9], 80.00th=[   10], 90.00th=[   18], 95.00th=[   24],
         | 99.00th=[   50], 99.50th=[   53], 99.90th=[ 6544], 99.95th=[ 6544],
         | 99.99th=[ 6544]
       bw (  MiB/s): min=  726, max= 3730, per=100.00%, avg=2848.29, stdev=1081.92, samples=7
       iops        : min=  726, max= 3730, avg=2848.29, stdev=1081.92, samples=7
      lat (usec)   : 4=0.04%, 10=0.01%, 500=0.05%, 750=0.05%, 1000=0.05%
      lat (msec)   : 2=0.20%, 4=0.35%, 10=81.28%, 20=11.36%, 50=5.71%
      lat (msec)   : 100=0.61%, >=2000=0.30%
      fsync/fdatasync/sync_file_range:
        sync (nsec): min=646, max=646, avg=646.00, stdev= 0.00
        sync percentiles (nsec):
         |  1.00th=[  644],  5.00th=[  644], 10.00th=[  644], 20.00th=[  644],
         | 30.00th=[  644], 40.00th=[  644], 50.00th=[  644], 60.00th=[  644],
         | 70.00th=[  644], 80.00th=[  644], 90.00th=[  644], 95.00th=[  644],
         | 99.00th=[  644], 99.50th=[  644], 99.90th=[  644], 99.95th=[  644],
         | 99.99th=[  644]
      cpu          : usr=4.39%, sys=62.32%, ctx=9354, majf=0, minf=15
      IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
         issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=32
    
    Run status group 0 (all jobs):
      WRITE: bw=1014MiB/s (1063MB/s), 1014MiB/s-1014MiB/s (1063MB/s-1063MB/s), io=10.0GiB (10.7GB), run=10102-10102msec
    and another 5 minutes:

    Code:
    WRITE: bw=1020MiB/s (1070MB/s), 1020MiB/s-1020MiB/s (1070MB/s-1070MB/s), io=10.0GiB (10.7GB), run=10038-10038msec
    Despite this, one thing I will say is that things have been perfect when streaming using Emby.

    A scrub was recently performed:

    Code:
    /mnt/Tank # zpool status
      pool: Tank
     state: ONLINE
      scan: scrub repaired 0B in 05:43:35 with 0 errors on Sun Jan 14 06:07:36 2024
    config:
    
        NAME                                                                STATE     READ WRITE CKSUM
        Tank                                                                ONLINE       0     0     0
          raidz2-0                                                          ONLINE       0     0     0
            ata-ST4000DM000-1F2168_S300xxxx                                 ONLINE       0     0     0
            ata-ST4000DM004-2CV104_ZTT4xxxx                                 ONLINE       0     0     0
            ata-ST4000DM004-2CV104_ZTT4xxxx                                 ONLINE       0     0     0
            ata-ST4000DM000-1F2168_W300xxxx                                 ONLINE       0     0     0
            ata-ST4000DM000-1F2168_W300xxxx                                 ONLINE       0     0     0
            ata-ST4000DM000-1F2168_W300xxxx                                 ONLINE       0     0     0
            ata-ST4000DM000-1F2168_W300xxxx                                 ONLINE       0     0     0
            ata-ST4000DM000-1F2168_W300xxxx                                 ONLINE       0     0     0
        logs    
          nvme-Samsung_SSD_980_PRO_with_Heatsink_2TB_S6WRNS0W53xxxxx-part2  ONLINE       0     0     0
        cache
          nvme0n1p1                                                         ONLINE       0     0     0
    
    errors: No known data errors
    Ksplice is not yet installed.

    Could this be within the realms of acceptable slow down? I appreciate that once in a while something like this 'could' happen, as long as it doesn't happen as much as it was before. I've had no other problems. Still, I'm a bit gutted. This is the first issue since replacing the OS drive.

    Another check in the 5-10 mins it's taken me to type/paste this:

    Code:
    WRITE: bw=1424MiB/s (1494MB/s), 1424MiB/s-1424MiB/s (1494MB/s-1494MB/s), io=10.0GiB (10.7GB), run=7189-7189msec
    Code:
    READ: bw=4525MiB/s (4745MB/s), 4525MiB/s-4525MiB/s (4745MB/s-4745MB/s), io=10.0GiB (10.7GB), run=2263-2263msec
    What do you guys think? Problem still here or just a blip?

    Edit: Gonna have to hit the hay as I have a long day tomorrow. Will check in ASAP.
    Last edited by tkae-lp; January 15th, 2024 at 03:02 AM.

  3. #93
    Join Date
    Nov 2023
    Beans
    76

    Re: Seemingly sporadic slow ZFS IO since 22.04

    No problems today:

    Code:
    READ: bw=4326MiB/s (4536MB/s), 4326MiB/s-4326MiB/s (4536MB/s-4536MB/s), io=10.0GiB (10.7GB), run=2367-2367msec
    Code:
    WRITE: bw=1377MiB/s (1443MB/s), 1377MiB/s-1377MiB/s (1443MB/s-1443MB/s), io=10.0GiB (10.7GB), run=7439-7439msec

    I'm thinking that this may have just been a freak occurrence. I continue to experience no issues with playback. However, I'm going to continue to monitor for a while.

    Sorry this is taking longer than expected!

  4. #94
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Seemingly sporadic slow ZFS IO since 22.04

    No problems. This is in your own time. LOL.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  5. #95
    Join Date
    Nov 2023
    Beans
    76

    Re: Seemingly sporadic slow ZFS IO since 22.04

    Very true!

    No problems so far today:

    Code:
    READ: bw=3561MiB/s (3733MB/s), 3561MiB/s-3561MiB/s (3733MB/s-3733MB/s), io=10.0GiB (10.7GB), run=2876-2876msec
    Code:
    WRITE: bw=1782MiB/s (1869MB/s), 1782MiB/s-1782MiB/s (1869MB/s-1869MB/s), io=10.0GiB (10.7GB), run=5745-5745msec
    Whatever it was seems to have been a freak occurrence. I even checked the new OS SSD's SMART data just in case something had flagged which might indicate a deeper SATA (board) issue but it's squeaky clean.

    Still monitoring....

  6. #96
    Join Date
    Nov 2023
    Beans
    76

    Re: Seemingly sporadic slow ZFS IO since 22.04

    Ok, this is not gone... last couple of days I've had slow downs.

    Just now I see:

    Code:
    READ: bw=2735MiB/s (2868MB/s), 2735MiB/s-2735MiB/s (2868MB/s-2868MB/s), io=10.0GiB (10.7GB), run=3744-3744msec
    But:

    Code:
    WRITE: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=10.0GiB (10.7GB), run=40824-40824msec
    So the main problem is on the write again. All the docker configs are on that tank which means the Emby DB - this could be why the write slow down is so apparent, It will want to update the DB with the current file and playback position.

    Ksplice is not installed, so I guess that rules that out.

    Where do we go from here? It has been so good up until this point. I've enjoyed a flawless experience until the hiccup the other day. It seems like we fix one thing and another pops up. I'm almost getting the feeling that I am dealing with multiple issues here....

    ARC is not quite maxed:

    Code:
        time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
    22:57:42     0     0      0     0    0     0    0     0    0   60G   64G    46G
    Free mem:

    Code:
    free -m  | grep ^Mem | tr -s ' ' | cut -d ' ' -f 3
    57147
    We're on the eve of Noble... Do I limp along and jump or try and fix this? The curiosity in me wants to fix this.... but idk. What do you guys think?

  7. #97
    Join Date
    Aug 2016
    Location
    Wandering
    Beans
    Hidden!
    Distro
    Xubuntu Development Release

    Re: Seemingly sporadic slow ZFS IO since 22.04

    Mine hasn't hit that yet:
    Code:
    Run status group 0 (all jobs):
      WRITE: bw=2297MiB/s (2409MB/s), 2297MiB/s-2297MiB/s (2409MB/s-2409MB/s), io=10.0GiB (10.7GB), run=4458-4458msec
    I'm going to talk to/with MAFoElffen to see if he is affected.

    I'm on Noble but I still don't advise a jump yet....getting closer but not yet.
    EDIT: Hold the press...this just in. I imported my tank and have a different showing now:
    Code:
    Run status group 0 (all jobs):
      WRITE: bw=57.3MiB/s (60.1MB/s), 57.3MiB/s-57.3MiB/s (60.1MB/s-60.1MB/s), io=3442MiB (3609MB), run=60027-60027msec
    Code:
    df -hT /tank
    Filesystem     Type  Size  Used Avail Use% Mounted on
    tank           zfs   450G  126G  325G  28% /tank
    Last edited by 1fallen; January 23rd, 2024 at 12:39 AM.
    With realization of one's own potential and self-confidence in one's ability, one can build a better world.
    Dalai Lama>>
    Code Tags | System-info | Forum Guide lines | Arch Linux, Debian Unstable, FreeBSD

  8. #98
    Join Date
    Nov 2023
    Beans
    76

    Re: Seemingly sporadic slow ZFS IO since 22.04

    Wow!!....... that certainly looks like the behaviour I have been seeing. I've never seen you post speeds anywhere near that low. Whatever this thing is, I would say it's a fairly safe bet that it seems you have it as well.

    I was just watching some old Star Trek TNG episodes and Emby was struggling to load each new episode so I checked, and this is from just now:

    Code:
    WRITE: bw=1073MiB/s (1125MB/s), 1073MiB/s-1073MiB/s (1125MB/s-1125MB/s), io=10.0GiB (10.7GB), run=9541-9541msec
    Code:
    READ: bw=563MiB/s (591MB/s), 563MiB/s-563MiB/s (591MB/s-591MB/s), io=10.0GiB (10.7GB), run=18174-18174msec
    Not being able to scrub through even SD without buffering is how bad it was when I first created the thread.

    My space usage has increased slightly since I had a clear out:

    Code:
    df -hT /mnt/Tank
    Filesystem     Type  Size  Used Avail Use% Mounted on
    Tank           zfs    21T   12T  8.7T  58% /mnt/Tank
    But in light of your slow down posted here (bearing in mind you are no where near 58%) I am not sure if it's the free space that's doing it. However, I will offload some stuff this evening and bring it back down a bit just to check.

    Just to be clear, I'm not on Noble, still on Jammy. I was just contemplating it.

    Edit: Just checked back in the thread, and I was originally at 69% full when opening this thread. I got it down to 57%.

    It seems unlikely this is the cause. However, I have talked to the kids and pruned another series they no longer want, freeing 192GB so back down to:

    Code:
    Filesystem     Type  Size  Used Avail Use% Mounted on
    Tank           zfs    21T   12T  8.9T  57% /mnt/Tank
    Last edited by tkae-lp; January 23rd, 2024 at 06:38 PM. Reason: Info about prune

  9. #99
    Join Date
    Aug 2016
    Location
    Wandering
    Beans
    Hidden!
    Distro
    Xubuntu Development Release

    Re: Seemingly sporadic slow ZFS IO since 22.04

    MAFoElffen has reported the same, and we are trying to find why.

    I get better write speeds in my /tank with btrfs.
    Code:
    Run status group 0 (all jobs):
      WRITE: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=10.0GiB (10.7GB), run=45526-45526msec
    This drive:
    Code:
    ─[/tank]
    └──╼ $df -hT
    Filesystem      Type      Size  Used Avail Use% Mounted on
    udev            devtmpfs  6.8G     0  6.8G   0% /dev
    tmpfs           tmpfs     1.4G  2.1M  1.4G   1% /run
    /dev/sda2       btrfs     457G  100G  356G  22% /
    tmpfs           tmpfs     6.8G  516K  6.8G   1% /dev/shm
    tmpfs           tmpfs     5.0M   16K  5.0M   1% /run/lock
    efivarfs        efivarfs  148K   71K   73K  50% /sys/firmware/efi/efivars
    /dev/sda2       btrfs     457G  100G  356G  22% /home
    /dev/sda1       vfat      300M  8.8M  291M   3% /boot/efi
    tank            zfs       450G  126G  325G  28% /tank
    tank/test       zfs       325G  128K  325G   1% /test
    tank/filesystem zfs       325G  128K  325G   1% /tank/filesystem
    tmpfs           tmpfs     1.4G  148K  1.4G   1% /run/user/1000
    I agree space dose not seem to be the main factor here at all.

    Still digging for a solution though.
    With realization of one's own potential and self-confidence in one's ability, one can build a better world.
    Dalai Lama>>
    Code Tags | System-info | Forum Guide lines | Arch Linux, Debian Unstable, FreeBSD

  10. #100
    Join Date
    Nov 2023
    Beans
    76

    Re: Seemingly sporadic slow ZFS IO since 22.04

    How odd! I did use btrfs for root a while back but am currently on ext4. However, your btrfs speeds are not what they should be even if they are faster. It's possible that this is simply random. I have found the issue to vary massively from check to check, sometimes fine [or not] for several hours/days. It may be giving the illusion that btrfs is better, when in fact you are just not in such a bad slow down right now. For example, this period of slow down I am currently experiencing is quite prevalent. I've had little relief from it. Still soldiering on through the SD TV episodes though lol

    I'm sorry you guys have been afflicted with this but - in some ways - this may be a good thing as you will be able to easier dig around and diagnose on your boxes, and you both know much more about ZFS that I do.

    Please let me know if there is anything I can do/try/test.
    Last edited by tkae-lp; January 25th, 2024 at 02:31 AM.

Page 10 of 14 FirstFirst ... 89101112 ... LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •