Code:
mike@beastie:/$ sudo zpool historyHistory for 'testpool':
2024-09-18.16:39:05 zpool create -o ashift=12 testpool raidz2 sda1 sdb1 sdc1 sdd1 sde1 -f
2024-09-18.16:40:24 zfs set compress=lz4 testpool
mike@beastie:/$ sudo zpool status
pool: testpool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sda1 ONLINE 0 0 0
sdb1 ONLINE 0 0 0
sdc1 ONLINE 0 0 0
sdd1 ONLINE 0 0 0
sde1 ONLINE 0 0 0
errors: No known data errors
mike@beastie:/$ sudo zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
testpool 2.27T 1.23M 2.27T - - 0% 0% 1.00x ONLINE -
mike@beastie:/testpool$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=write --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
TEST: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [W(1)][38.9%][w=554MiB/s][w=553 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [W(1)][56.5%][w=208MiB/s][w=208 IOPS][eta 00m:10s]
Jobs: 1 (f=1): [W(1)][76.0%][w=436MiB/s][w=436 IOPS][eta 00m:06s]
Jobs: 1 (f=1): [W(1)][89.3%][w=199MiB/s][w=199 IOPS][eta 00m:03s]
Jobs: 1 (f=1): [W(1)][100.0%][eta 00m:00s]
Jobs: 1 (f=1): [W(1)][97.4%][eta 00m:01s]
Jobs: 1 (f=1): [W(1)][97.4%][eta 00m:01s]
TEST: (groupid=0, jobs=1): err= 0: pid=319677: Wed Sep 18 16:45:12 2024
write: IOPS=267, BW=268MiB/s (281MB/s)(10.0GiB/38246msec); 0 zone resets
slat (usec): min=274, max=9307, avg=2764.48, stdev=1544.85
clat (usec): min=2, max=9938.6k, avg=115141.89, stdev=542234.64
lat (usec): min=320, max=9940.5k, avg=117907.03, stdev=542325.14
clat percentiles (msec):
| 1.00th=[ 11], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 24],
| 30.00th=[ 56], 40.00th=[ 70], 50.00th=[ 85], 60.00th=[ 104],
| 70.00th=[ 123], 80.00th=[ 138], 90.00th=[ 150], 95.00th=[ 155],
| 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 9866], 99.95th=[ 9866],
| 99.99th=[10000]
bw ( KiB/s): min=200704, max=2181120, per=100.00%, avg=358184.42, stdev=311594.61, samples=57
iops : min= 196, max= 2130, avg=349.79, stdev=304.29, samples=57
lat (usec) : 4=0.02%, 10=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.04%, 4=0.08%, 10=0.87%, 20=12.54%, 50=11.25%
lat (msec) : 100=33.48%, 250=41.37%, >=2000=0.30%
fsync/fdatasync/sync_file_range:
sync (nsec): min=1168, max=1168, avg=1168.00, stdev= 0.00
sync percentiles (nsec):
| 1.00th=[ 1176], 5.00th=[ 1176], 10.00th=[ 1176], 20.00th=[ 1176],
| 30.00th=[ 1176], 40.00th=[ 1176], 50.00th=[ 1176], 60.00th=[ 1176],
| 70.00th=[ 1176], 80.00th=[ 1176], 90.00th=[ 1176], 95.00th=[ 1176],
| 99.00th=[ 1176], 99.50th=[ 1176], 99.90th=[ 1176], 99.95th=[ 1176],
| 99.99th=[ 1176]
cpu : usr=1.44%, sys=10.42%, ctx=68611, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=268MiB/s (281MB/s), 268MiB/s-268MiB/s (281MB/s-281MB/s), io=10.0GiB (10.7GB), run=38246-38246msec
mike@beastie:/testpool$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
Jobs: 1 (f=0): [f(1)][-.-%][r=3682MiB/s][r=3682 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=364773: Wed Sep 18 16:45:37 2024
read: IOPS=3561, BW=3562MiB/s (3735MB/s)(10.0GiB/2875msec)
slat (usec): min=229, max=890, avg=278.54, stdev=37.48
clat (usec): min=2, max=23476, avg=8613.45, stdev=1134.15
lat (usec): min=262, max=24368, avg=8892.27, stdev=1164.29
clat percentiles (usec):
| 1.00th=[ 5342], 5.00th=[ 8291], 10.00th=[ 8291], 20.00th=[ 8356],
| 30.00th=[ 8356], 40.00th=[ 8356], 50.00th=[ 8356], 60.00th=[ 8356],
| 70.00th=[ 8455], 80.00th=[ 8455], 90.00th=[10028], 95.00th=[10159],
| 99.00th=[12649], 99.50th=[13042], 99.90th=[20317], 99.95th=[21890],
| 99.99th=[23200]
bw ( MiB/s): min= 2844, max= 3714, per=99.12%, avg=3530.40, stdev=384.01, samples=5
iops : min= 2844, max= 3714, avg=3530.40, stdev=384.01, samples=5
lat (usec) : 4=0.05%, 500=0.05%, 750=0.05%, 1000=0.05%
lat (msec) : 2=0.20%, 4=0.34%, 10=85.87%, 20=13.29%, 50=0.11%
cpu : usr=1.08%, sys=98.82%, ctx=8, majf=0, minf=8205
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=3562MiB/s (3735MB/s), 3562MiB/s-3562MiB/s (3735MB/s-3735MB/s), io=10.0GiB (10.7GB), run=2875-2875msec
mike@beastie:/testpool$
on this attempt (test) the drives did pick up a bit in write, and drop a bit in read. looking over the data from a striped, raidZ1 vs the raidZ2. 
Adv Reply
Originally Posted by sgt-mike
Bookmarks