99.99% NFS io for jbd2 on /home partition

[wkuhns]

Seems to be NFS related.

Running 12.04 64-bit on server, 12.10 32 bit client. Server provides /home to client via NFS. Client hangs often for MANY seconds. Root cause appears to be writes to /home file system causing 99.99% io usage by jbd2 for /home partition on server for extended periods.

I've reviewed a bunch of threads on this topic, and none of the situations or solutions seem to fit.

Server and client mount options include noatime for this partition. Server shows no tasks other than NFS doing disk writes to /home.

Smartctl shows no health issues. No log files or databases on /home.

System was fine for many weeks.

[wkuhns]

Update: NFS write to /home partition is definitely the trigger. Writing a 1gb file causes the client process to hang for several minutes with server iotop showing 99% IO for the jbd2 process. The system was fine before attempting a file save, and there's WAY more NFS disk write activity than can be explained by saving a 1gb file. There is no one logged into the server at this point, and only a single NFS client, writing a single 1gb file:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 981.28 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
886 be/3 root 0.00 B/s 0.00 B/s 0.00 % 99.92 % [jbd2/sdb3-8]
1901 be/4 root 0.00 B/s 54.52 K/s 0.00 % 0.11 % [nfsd]
1903 be/4 root 0.00 B/s 54.52 K/s 0.00 % 0.10 % [nfsd]
1899 be/4 root 0.00 B/s 54.52 K/s 0.00 % 0.09 % [nfsd]
1904 be/4 root 0.00 B/s 46.73 K/s 0.00 % 0.09 % [nfsd]
1902 be/4 root 0.00 B/s 54.52 K/s 0.00 % 0.09 % [nfsd]
1905 be/4 root 0.00 B/s 46.73 K/s 0.00 % 0.08 % [nfsd]
1900 be/4 root 0.00 B/s 38.94 K/s 0.00 % 0.07 % [nfsd]
1898 be/4 root 0.00 B/s 38.94 K/s 0.00 % 0.05 % [nfsd]

[SeijiSensei]

Is the server configured to use "sync" (the default) or "async"? The latter can be considerably faster. It does create the possibility of data loss if the client or server crashes before the write is complete, but on reliable networks the probability of that happening is remote.

Also how big are rsize and wsize set on the client? I usually set them to 32768 over wifi, and 65536 over ethernet.

[wkuhns]

Server is "sync" - I've used that setting with several generations of Linux servers with perfectly acceptable performance and I'd rather not mess with it. I changed rsize and wsize from 8192 to 65536 - made no difference. I have several client machines with a wide range of operating system vintages mounting /home via nfs, but there *should* be almost no disk activity. This has all worked for years. The recent change is replacing my old server with a new one running xubuntu 12.04. I used ext4 for most partitions, while the previous server had used ext3. The new server runs NFSv4 which necessitated an extra mount option for some older clients (nfsvers=2).

I'm suspicious of the large number of nfs processes in the iotop snapshot above. More notes on server configuration:
sda is an SSD with boot, root(/), and swap partitions
sdb is a 2tb Hitachi SATA drive with ext4 partitions for /var, /tmp, /home, and a media partition. Only the /home partition is exported via NFS.

[capscrew]

Server is "sync" - I've used that setting with several generations of Linux servers with perfectly acceptable performance and I'd rather not mess with it...

The recent change is replacing my old server with a new one running xubuntu 12.04. I used ext4 for most partitions, while the previous server had used ext3.

The ext3 file system handles sync differently than ext4. This most likely is the difference you see in performance. Ted Tso is the principal developer of ext3 and ext4 file systems. You can see Ted Tso's thoughts on sync and fsync here and here.

The bottom line is to go with async in your mount directive if you are using ext4 file systems for your partitions.

[wkuhns]

Thanks - changing both the mount options in fstab and specifying async in /etc/exports solved the problem. As far as I can tell, changing the partition mount options was not enough.

[capscrew]

I believe you are correct. It's the exports that need the async setting.