Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 29

Thread: Ubuntu 14 vs 20x File Copy Performance Slow

  1. #11
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Ubuntu 14 vs 20x File Copy Performance Slow

    Is the physical storage used by both the same? Same connections, same controller, same physical disk? I'm not talking about the virtual stuff.

  2. #12
    Join Date
    Feb 2024
    Beans
    15

    Re: Ubuntu 14 vs 20x File Copy Performance Slow

    I may have found the solution. Currently working through tests for confirmation.
    Basically, I boosted these and will post the final results.

    #Modify Dirty Parameters on Server2 Ubuntu23:
    echo 10 > /proc/sys/vm/dirty_background_ratio
    echo 20 > /proc/sys/vm/dirty_ratio
    echo 3000 > /proc/sys/vm/dirty_expire_centisecs
    echo 500 > /proc/sys/vm/dirty_writeback_centisecs
    echo $((64*1024*1024)) | sudo tee /proc/sys/vm/dirty_background_bytes
    echo $((512*1024*1024)) | sudo tee /proc/sys/vm/dirty_bytes
    sync; echo 3 > /proc/sys/vm/drop_caches

  3. #13
    Join Date
    Feb 2024
    Beans
    15

    Re: Ubuntu 14 vs 20x File Copy Performance Slow

    The issue is not resolved.
    I am now working through using your latest suggestion for a clearer benchmark.
    Thank you.

  4. #14
    Join Date
    Feb 2024
    Beans
    15

    Re: Ubuntu 14 vs 20x File Copy Performance Slow

    Yes, the 2 servers are on the same ESXi Host and both servers vmdk's are located on the same physical disks.
    I have been monitoring the ESXI host using vscsiStats -l and esxtop to which I can post here if there is something I might have overlooked.
    There are no special rules, configurations, priority groups, or reserved resources assigned to either box at the virtual level.

  5. #15
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Ubuntu 14 vs 20x File Copy Performance Slow

    On a 14.04 VM:
    Code:
    TEST_WRITE: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
    fio-2.1.3
    Starting 1 process
    TEST_WRITE: (groupid=0, jobs=1): err= 0: pid=2884: Tue Feb 13 08:34:27 2024
      write: io=1024.0MB, bw=6918.1MB/s, iops=6918, runt=   148msec
        slat (usec): min=25, max=1372, avg=128.72, stdev=83.95
        clat (usec): min=474, max=14618, avg=4415.14, stdev=2057.45
         lat (usec): min=588, max=15097, avg=4544.06, stdev=2072.32
        clat percentiles (usec):
         |  1.00th=[ 1416],  5.00th=[ 3344], 10.00th=[ 3440], 20.00th=[ 3536],
         | 30.00th=[ 3600], 40.00th=[ 3664], 50.00th=[ 3792], 60.00th=[ 3984],
         | 70.00th=[ 4128], 80.00th=[ 4384], 90.00th=[ 6240], 95.00th=[ 8032],
         | 99.00th=[13888], 99.50th=[14144], 99.90th=[14400], 99.95th=[14656],
         | 99.99th=[14656]
        lat (usec) : 500=0.10%, 750=0.49%
        lat (msec) : 2=0.88%, 4=58.69%, 10=36.23%, 20=3.61%
      cpu          : usr=62.59%, sys=27.21%, ctx=21, majf=0, minf=8
      IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=97.0%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
         issued    : total=r=0/w=1024/d=0, short=r=0/w=0/d=0
    Run status group 0 (all jobs):
      WRITE: io=1024.0MB, aggrb=6918.1MB/s, minb=6918.1MB/s, maxb=6918.1MB/s, mint=148msec, maxt=148msec
    Disk stats (read/write):
      vda: ios=0/987, merge=0/0, ticks=0/1280, in_queue=1308, util=56.00%
    TEST_READ: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
    fio-2.1.3
    Starting 1 process
    TEST_READ: (groupid=0, jobs=1): err= 0: pid=2888: Tue Feb 13 08:34:27 2024
      read : io=1024.0MB, bw=6606.5MB/s, iops=6606, runt=   155msec
        slat (usec): min=24, max=929, avg=112.97, stdev=86.92
        clat (usec): min=435, max=23661, avg=4637.92, stdev=3753.76
         lat (usec): min=509, max=23763, avg=4751.25, stdev=3752.69
        clat percentiles (usec):
         |  1.00th=[ 1160],  5.00th=[ 1720], 10.00th=[ 2096], 20.00th=[ 2608],
         | 30.00th=[ 2928], 40.00th=[ 3152], 50.00th=[ 3408], 60.00th=[ 3600],
         | 70.00th=[ 4128], 80.00th=[ 5536], 90.00th=[ 8768], 95.00th=[13888],
         | 99.00th=[20352], 99.50th=[23680], 99.90th=[23680], 99.95th=[23680],
         | 99.99th=[23680]
        lat (usec) : 500=0.10%, 750=0.29%, 1000=0.20%
        lat (msec) : 2=8.11%, 4=59.57%, 10=23.14%, 20=7.32%, 50=1.27%
      cpu          : usr=0.00%, sys=80.52%, ctx=80, majf=0, minf=8200
      IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=97.0%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
         issued    : total=r=1024/w=0/d=0, short=r=0/w=0/d=0
    Run status group 0 (all jobs):
       READ: io=1024.0MB, aggrb=6606.5MB/s, minb=6606.5MB/s, maxb=6606.5MB/s, mint=155msec, maxt=155msec
    Disk stats (read/write):
      vda: ios=967/0, merge=0/0, ticks=3176/0, in_queue=3220, util=59.44%
    Note that option '--io_size=' was not in the 14.04 version of fio, so was removed for that test...

    On a Mantic VM:
    Code:
    mafoelffen@minatuar-02:~/Documents$ fio --name TEST_WRITE --eta-newline=5s --filename=temp.file --rw=write --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting && fio --name TEST_READ --eta-newline=5s --filename=temp.file --rw=read --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
    TEST_WRITE: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
    fio-3.35
    Starting 1 process
    TEST_WRITE: Laying out IO file (1 file / 2048MiB)
    Jobs: 1 (f=1): [W(1)][25.0%][w=85.9MiB/s][w=85 IOPS][eta 00m:24s]
    Jobs: 1 (f=1): [W(1)][41.9%][w=292MiB/s][w=292 IOPS][eta 00m:18s] 
    Jobs: 1 (f=1): [W(1)][61.3%][w=338MiB/s][w=338 IOPS][eta 00m:12s] 
    Jobs: 1 (f=1): [W(1)][78.1%][w=310MiB/s][w=310 IOPS][eta 00m:07s] 
    Jobs: 1 (f=1): [W(1)][93.9%][w=381MiB/s][w=381 IOPS][eta 00m:02s] 
    Jobs: 1 (f=1): [W(1)][100.0%][w=64.1MiB/s][w=64 IOPS][eta 00m:00s]
    TEST_WRITE: (groupid=0, jobs=1): err= 0: pid=3492: Tue Feb 13 08:51:10 2024
      write: IOPS=318, BW=318MiB/s (334MB/s)(10.0GiB/32160msec); 0 zone resets
        slat (usec): min=100, max=1088.7k, avg=3063.82, stdev=43769.40
        clat (usec): min=2, max=1137.9k, avg=96870.10, stdev=235523.45
         lat (usec): min=156, max=1139.5k, avg=99933.91, stdev=238866.44
        clat percentiles (msec):
         |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
         | 30.00th=[    8], 40.00th=[   11], 50.00th=[   15], 60.00th=[   21],
         | 70.00th=[   27], 80.00th=[   36], 90.00th=[  211], 95.00th=[  827],
         | 99.00th=[  995], 99.50th=[ 1020], 99.90th=[ 1116], 99.95th=[ 1133],
         | 99.99th=[ 1133]
       bw (  KiB/s): min=16384, max=1058816, per=100.00%, avg=486089.93, stdev=290007.48, samples=42
       iops        : min=   16, max= 1034, avg=474.69, stdev=283.22, samples=42
      lat (usec)   : 4=0.05%, 250=0.01%, 500=0.04%, 750=0.07%, 1000=0.03%
      lat (msec)   : 2=0.20%, 4=0.43%, 10=37.58%, 20=21.40%, 50=22.85%
      lat (msec)   : 100=2.44%, 250=6.74%, 500=0.30%, 750=0.52%, 1000=6.44%
      lat (msec)   : 2000=0.92%
      fsync/fdatasync/sync_file_range:
        sync (nsec): min=626603k, max=626603k, avg=626603257.00, stdev= 0.00
        sync percentiles (msec):
         |  1.00th=[  625],  5.00th=[  625], 10.00th=[  625], 20.00th=[  625],
         | 30.00th=[  625], 40.00th=[  625], 50.00th=[  625], 60.00th=[  625],
         | 70.00th=[  625], 80.00th=[  625], 90.00th=[  625], 95.00th=[  625],
         | 99.00th=[  625], 99.50th=[  625], 99.90th=[  625], 99.95th=[  625],
         | 99.99th=[  625]
      cpu          : usr=1.31%, sys=4.82%, ctx=17751, majf=13, minf=10
      IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
         issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=32
    
    Run status group 0 (all jobs):
      WRITE: bw=318MiB/s (334MB/s), 318MiB/s-318MiB/s (334MB/s-334MB/s), io=10.0GiB (10.7GB), run=32160-32160msec
    TEST_READ: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
    fio-3.35
    Starting 1 process
    Jobs: 1 (f=1): [R(1)][83.3%][r=2216MiB/s][r=2216 IOPS][eta 00m:01s]
    TEST_READ: (groupid=0, jobs=1): err= 0: pid=3559: Tue Feb 13 08:51:15 2024
      read: IOPS=2128, BW=2128MiB/s (2232MB/s)(10.0GiB/4811msec)
        slat (usec): min=173, max=4160, avg=453.57, stdev=146.60
        clat (nsec): min=1494, max=49629k, avg=13964383.67, stdev=3321470.00
         lat (usec): min=175, max=51683, avg=14417.95, stdev=3412.82
        clat percentiles (usec):
         |  1.00th=[ 6718],  5.00th=[10683], 10.00th=[12649], 20.00th=[13042],
         | 30.00th=[13173], 40.00th=[13304], 50.00th=[13566], 60.00th=[13698],
         | 70.00th=[14091], 80.00th=[14877], 90.00th=[16188], 95.00th=[17171],
         | 99.00th=[26346], 99.50th=[32900], 99.90th=[47449], 99.95th=[49021],
         | 99.99th=[49021]
       bw (  MiB/s): min= 1412, max= 2310, per=99.24%, avg=2112.38, stdev=274.00, samples=9
       iops        : min= 1412, max= 2310, avg=2112.22, stdev=274.00, samples=9
      lat (usec)   : 2=0.03%, 4=0.02%, 250=0.05%, 500=0.05%, 750=0.05%
      lat (usec)   : 1000=0.05%
      lat (msec)   : 2=0.14%, 4=0.26%, 10=3.99%, 20=92.83%, 50=2.53%
      cpu          : usr=0.94%, sys=91.62%, ctx=10394, majf=0, minf=8204
      IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
         issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=32
    
    Run status group 0 (all jobs):
       READ: bw=2128MiB/s (2232MB/s), 2128MiB/s-2128MiB/s (2232MB/s-2232MB/s), io=10.0GiB (10.7GB), run=4811-4811msec
    Hmmm. I do see a difference, at minimum specs allocated. Both of those VM's vdisks are on the same Host, on the same storage pool (same disks)...

    I confirm that I see a difference, but I can't explain why.
    Last edited by MAFoElffen; February 13th, 2024 at 05:57 PM.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  6. #16
    Join Date
    Feb 2024
    Beans
    15

    Re: Ubuntu 14 vs 20x File Copy Performance Slow

    Using fio to get a more detailed Benchmark:
    fio --name TEST_WRITE --eta-newline=5s --filename=temp.file --rw=write --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting && fio --name TEST_READ --eta-newline=5s --filename=temp.file --rw=read --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
    Code:
    TEST_WRITE: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
    fio-3.35
    Starting 1 process
    Jobs: 1 (f=1): [W(1)][21.9%][w=283MiB/s][w=283 IOPS][eta 00m:25s]
    Jobs: 1 (f=1): [W(1)][39.4%][w=280MiB/s][w=280 IOPS][eta 00m:20s]
    Jobs: 1 (f=1): [W(1)][55.9%][w=274MiB/s][w=274 IOPS][eta 00m:15s]
    Jobs: 1 (f=1): [W(1)][73.5%][w=302MiB/s][w=302 IOPS][eta 00m:09s]
    Jobs: 1 (f=1): [W(1)][91.2%][w=220MiB/s][w=220 IOPS][eta 00m:03s]
    Jobs: 1 (f=1): [W(1)][100.0%][w=287MiB/s][w=287 IOPS][eta 00m:00s]
    TEST_WRITE: (groupid=0, jobs=1): err= 0: pid=1780: Tue Feb 13 17:10:47 2024
      write: IOPS=295, BW=296MiB/s (310MB/s)(10.0GiB/34622msec); 0 zone resets
        slat (usec): min=85, max=130670, avg=217.52, stdev=2591.22
        clat (msec): min=8, max=333, avg=107.82, stdev=32.53
         lat (msec): min=8, max=333, avg=108.03, stdev=32.43
        clat percentiles (msec):
         |  1.00th=[   23],  5.00th=[   56], 10.00th=[   78], 20.00th=[   89],
         | 30.00th=[   94], 40.00th=[   99], 50.00th=[  103], 60.00th=[  109],
         | 70.00th=[  118], 80.00th=[  131], 90.00th=[  148], 95.00th=[  167],
         | 99.00th=[  199], 99.50th=[  215], 99.90th=[  288], 99.95th=[  317],
         | 99.99th=[  330]
       bw (  KiB/s): min=219136, max=584558, per=100.00%, avg=302971.26, stdev=51309.14, samples=69
       iops        : min=  214, max=  570, avg=295.84, stdev=50.04, samples=69
      lat (msec)   : 10=0.19%, 20=0.72%, 50=2.46%, 100=42.12%, 250=54.36%
      lat (msec)   : 500=0.16%
      fsync/fdatasync/sync_file_range:
        sync (nsec): min=233773k, max=233773k, avg=233773249.00, stdev= 0.00
        sync percentiles (msec):
         |  1.00th=[  234],  5.00th=[  234], 10.00th=[  234], 20.00th=[  234],
         | 30.00th=[  234], 40.00th=[  234], 50.00th=[  234], 60.00th=[  234],
         | 70.00th=[  234], 80.00th=[  234], 90.00th=[  234], 95.00th=[  234],
         | 99.00th=[  234], 99.50th=[  234], 99.90th=[  234], 99.95th=[  234],
         | 99.99th=[  234]
      cpu          : usr=2.32%, sys=2.75%, ctx=9498, majf=0, minf=13
      IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
         issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=32
    
    
    Run status group 0 (all jobs):
      WRITE: bw=296MiB/s (310MB/s), 296MiB/s-296MiB/s (310MB/s-310MB/s), io=10.0GiB (10.7GB), run=34622-34622msec
    
    
    Disk stats (read/write):
      sdc: ios=0/17646, merge=0/592, ticks=0/1852003, in_queue=1852002, util=99.71%
    TEST_READ: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
    fio-3.35
    Starting 1 process
    Jobs: 1 (f=1): [R(1)][70.0%][r=889MiB/s][r=889 IOPS][eta 00m:03s]
    Jobs: 1 (f=1): [R(1)][100.0%][r=922MiB/s][r=921 IOPS][eta 00m:00s]
    TEST_READ: (groupid=0, jobs=1): err= 0: pid=1787: Tue Feb 13 17:10:58 2024
      read: IOPS=959, BW=960MiB/s (1006MB/s)(10.0GiB/10669msec)
        slat (usec): min=78, max=5005, avg=143.03, stdev=213.65
        clat (usec): min=10039, max=78712, avg=33013.74, stdev=5933.09
         lat (usec): min=10569, max=78793, avg=33156.77, stdev=5898.63
        clat percentiles (usec):
         |  1.00th=[16581],  5.00th=[27395], 10.00th=[28443], 20.00th=[28705],
         | 30.00th=[29230], 40.00th=[30016], 50.00th=[31327], 60.00th=[33162],
         | 70.00th=[35914], 80.00th=[38011], 90.00th=[40109], 95.00th=[41157],
         | 99.00th=[52167], 99.50th=[61080], 99.90th=[67634], 99.95th=[71828],
         | 99.99th=[76022]
       bw (  KiB/s): min=821248, max=1134592, per=99.99%, avg=982734.14, stdev=101143.49, samples=21
       iops        : min=  802, max= 1108, avg=959.62, stdev=98.68, samples=21
      lat (msec)   : 20=1.68%, 50=96.81%, 100=1.51%
      cpu          : usr=0.39%, sys=13.31%, ctx=7255, majf=0, minf=8204
      IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
         issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=32
    
    
    Run status group 0 (all jobs):
       READ: bw=960MiB/s (1006MB/s), 960MiB/s-960MiB/s (1006MB/s-1006MB/s), io=10.0GiB (10.7GB), run=10669-10669msec
    
    
    Disk stats (read/write):
      sdc: ios=12573/3, merge=3058/1, ticks=399820/169, in_queue=399989, util=99.23%

  7. #17
    Join Date
    Feb 2024
    Beans
    15

    Re: Ubuntu 14 vs 20x File Copy Performance Slow

    Yes, this is a head scratcher. Process of elimination has been quite extensive but still have not found the root cause. I have reviewed the Ubuntu online documentation to search for any notes, updates, vm specific required changes, or if NFS3 is simply the root of the problem in the way Ubuntu updated versions interacts/writes/caches/etc... with an nfs filesystem. I currently trying to get access to an esxi host and datastore that isn't being used within my company to create a test case where nfs is updated to the latest version. But this is proving to be a challenge due to cost and all current resources being utilized.
    Thus, am hopeful for a temporary solution to this issue until I can get the backend upgraded which could take a long time.

    Any other suggestions are much appreciated and thank you for your time thus far in attempting to help me.

  8. #18
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Ubuntu 14 vs 20x File Copy Performance Slow

    Okay... I found where my difference was. I spun u[p a new Mantic server VM. I ran it on an existing, and when I looked at it, it was on an encrypted filesystem. Not a fair comparison. Dang.

    I also removed --io_size from my Mantic VM run, so that the commands were the same for both:
    Code:
    TEST_WRITE: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
    fio-3.35
    Starting 1 process
    TEST_WRITE: (groupid=0, jobs=1): err= 0: pid=5687: Tue Feb 13 18:31:26 2024
      write: IOPS=3531, BW=3531MiB/s (3703MB/s)(1024MiB/290msec); 0 zone resets
        slat (usec): min=68, max=1872, avg=279.60, stdev=147.21
        clat (usec): min=131, max=16588, avg=8583.53, stdev=2666.41
         lat (usec): min=393, max=16843, avg=8863.14, stdev=2732.06
        clat percentiles (usec):
         |  1.00th=[ 5080],  5.00th=[ 6783], 10.00th=[ 6849], 20.00th=[ 6980],
         | 30.00th=[ 7046], 40.00th=[ 7177], 50.00th=[ 7308], 60.00th=[ 7635],
         | 70.00th=[ 8160], 80.00th=[11994], 90.00th=[13173], 95.00th=[14222],
         | 99.00th=[15533], 99.50th=[15795], 99.90th=[15926], 99.95th=[16581],
         | 99.99th=[16581]
      lat (usec)   : 250=0.10%, 500=0.10%, 1000=0.10%
      lat (msec)   : 2=0.20%, 4=0.29%, 10=77.83%, 20=21.39%
      cpu          : usr=18.69%, sys=77.85%, ctx=964, majf=0, minf=13
      IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=97.0%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
         issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=32
    Run status group 0 (all jobs):
      WRITE: bw=3531MiB/s (3703MB/s), 3531MiB/s-3531MiB/s (3703MB/s-3703MB/s), io=1024MiB (1074MB), run=290-290msec
    Disk stats (read/write):
        dm-0: ios=0/574, merge=0/0, ticks=0/144, in_queue=144, util=58.06%, aggrios=0/8192, aggrmerge=0/0, aggrticks=0/1791, aggrin_queue=1790, aggrutil=66.52%
      sda: ios=0/8192, merge=0/0, ticks=0/1791, in_queue=1790, util=66.52%
    
    TEST_READ: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
    fio-3.35
    Starting 1 process
    TEST_READ: (groupid=0, jobs=1): err= 0: pid=5691: Tue Feb 13 18:31:26 2024
      read: IOPS=3835, BW=3835MiB/s (4022MB/s)(1024MiB/267msec)
        slat (usec): min=149, max=406, avg=258.00, stdev=35.20
        clat (usec): min=84, max=10964, avg=7905.76, stdev=938.24
         lat (usec): min=350, max=11366, avg=8163.76, stdev=948.02
        clat percentiles (usec):
         |  1.00th=[ 2704],  5.00th=[ 7439], 10.00th=[ 7504], 20.00th=[ 7570],
         | 30.00th=[ 7701], 40.00th=[ 7832], 50.00th=[ 7898], 60.00th=[ 8029],
         | 70.00th=[ 8225], 80.00th=[ 8356], 90.00th=[ 8717], 95.00th=[ 8717],
         | 99.00th=[ 9765], 99.50th=[10421], 99.90th=[10814], 99.95th=[10945],
         | 99.99th=[10945]
      lat (usec)   : 100=0.10%, 500=0.10%, 750=0.10%, 1000=0.10%
      lat (msec)   : 2=0.39%, 4=0.78%, 10=97.56%, 20=0.88%
      cpu          : usr=1.13%, sys=98.50%, ctx=10, majf=0, minf=8206
      IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=97.0%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
         issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=32
    Run status group 0 (all jobs):
       READ: bw=3835MiB/s (4022MB/s), 3835MiB/s-3835MiB/s (4022MB/s-4022MB/s), io=1024MiB (1074MB), run=267-267msec
    Disk stats (read/write):
        dm-0: ios=555/0, merge=0/0, ticks=184/0, in_queue=184, util=61.29%, aggrios=8192/0, aggrmerge=0/0, aggrticks=1223/0, aggrin_queue=1223, aggrutil=65.23%
      sda: ios=8192/0, merge=0/0, ticks=1223/0, in_queue=1223, util=65.23%
    So the results for 14.04:
    Code:
      WRITE: io=1024.0MB, aggrb=6918.1MB/s, minb=6918.1MB/s, maxb=6918.1MB/s, mint=148msec, maxt=148msec
      READ: io=1024.0MB, aggrb=6606.5MB/s, minb=6606.5MB/s, maxb=6606.5MB/s, mint=155msec, maxt=155msec
    And for 23.10:
    Code:
      WRITE: bw=3531MiB/s (3703MB/s), 3531MiB/s-3531MiB/s (3703MB/s-3703MB/s), io=1024MiB (1074MB), run=290-290msec
      READ: bw=3835MiB/s (4022MB/s), 3835MiB/s-3835MiB/s (4022MB/s-4022MB/s), io=1024MiB (1074MB), run=267-267msec
    Yes, it shows as faster with 23.10, by those results... That is on the same host & storage pool for both, with 23.10 using 'levator-noop' as a boot parameter.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  9. #19
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Ubuntu 14 vs 20x File Copy Performance Slow

    If you just mount with -t nfs option, you get nfsv4 in Debian-based OSes. The only way I'd expect NFSv3 to be used is if the client specifically requests it or if the NFS server is on hardware from 2005.

  10. #20
    Join Date
    Feb 2024
    Beans
    15

    Re: Ubuntu 14 vs 20x File Copy Performance Slow

    Please confirm for my understanding from your results that 23.10 is faster? For the 14.4 server, mint/maxt = 148/155ms and the 23.10 server, run=290/267ms.
    For my continued education, in the output, what is showing 23 is faster than 14?

Page 2 of 3 FirstFirst 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •