Results 1 to 7 of 7

Thread: 99.99% NFS io for jbd2 on /home partition

  1. #1
    Join Date
    Apr 2011
    Beans
    11

    99.99% NFS io for jbd2 on /home partition

    Seems to be NFS related.

    Running 12.04 64-bit on server, 12.10 32 bit client. Server provides /home to client via NFS. Client hangs often for MANY seconds. Root cause appears to be writes to /home file system causing 99.99% io usage by jbd2 for /home partition on server for extended periods.

    I've reviewed a bunch of threads on this topic, and none of the situations or solutions seem to fit.

    Server and client mount options include noatime for this partition. Server shows no tasks other than NFS doing disk writes to /home.

    Smartctl shows no health issues. No log files or databases on /home.

    System was fine for many weeks.
    Last edited by wkuhns; August 27th, 2013 at 12:45 PM.

  2. #2
    Join Date
    Apr 2011
    Beans
    11

    Re: 99.99% io for jbd2 on /home partition

    Update: NFS write to /home partition is definitely the trigger. Writing a 1gb file causes the client process to hang for several minutes with server iotop showing 99% IO for the jbd2 process. The system was fine before attempting a file save, and there's WAY more NFS disk write activity than can be explained by saving a 1gb file. There is no one logged into the server at this point, and only a single NFS client, writing a single 1gb file:

    Total DISK READ: 0.00 B/s | Total DISK WRITE: 981.28 K/s
    TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
    886 be/3 root 0.00 B/s 0.00 B/s 0.00 % 99.92 % [jbd2/sdb3-8]
    1901 be/4 root 0.00 B/s 54.52 K/s 0.00 % 0.11 % [nfsd]
    1903 be/4 root 0.00 B/s 54.52 K/s 0.00 % 0.10 % [nfsd]
    1899 be/4 root 0.00 B/s 54.52 K/s 0.00 % 0.09 % [nfsd]
    1904 be/4 root 0.00 B/s 46.73 K/s 0.00 % 0.09 % [nfsd]
    1902 be/4 root 0.00 B/s 54.52 K/s 0.00 % 0.09 % [nfsd]
    1905 be/4 root 0.00 B/s 46.73 K/s 0.00 % 0.08 % [nfsd]
    1900 be/4 root 0.00 B/s 38.94 K/s 0.00 % 0.07 % [nfsd]
    1898 be/4 root 0.00 B/s 38.94 K/s 0.00 % 0.05 % [nfsd]

  3. #3
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    9,612
    Distro
    Kubuntu 14.04 Trusty Tahr

    Re: 99.99% NFS io for jbd2 on /home partition

    Is the server configured to use "sync" (the default) or "async"? The latter can be considerably faster. It does create the possibility of data loss if the client or server crashes before the write is complete, but on reliable networks the probability of that happening is remote.

    Also how big are rsize and wsize set on the client? I usually set them to 32768 over wifi, and 65536 over ethernet.
    If you ask for help, please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

  4. #4
    Join Date
    Apr 2011
    Beans
    11

    Re: 99.99% NFS io for jbd2 on /home partition

    Server is "sync" - I've used that setting with several generations of Linux servers with perfectly acceptable performance and I'd rather not mess with it. I changed rsize and wsize from 8192 to 65536 - made no difference. I have several client machines with a wide range of operating system vintages mounting /home via nfs, but there *should* be almost no disk activity. This has all worked for years. The recent change is replacing my old server with a new one running xubuntu 12.04. I used ext4 for most partitions, while the previous server had used ext3. The new server runs NFSv4 which necessitated an extra mount option for some older clients (nfsvers=2).

    I'm suspicious of the large number of nfs processes in the iotop snapshot above. More notes on server configuration:
    sda is an SSD with boot, root(/), and swap partitions
    sdb is a 2tb Hitachi SATA drive with ext4 partitions for /var, /tmp, /home, and a media partition. Only the /home partition is exported via NFS.

  5. #5
    Join Date
    Dec 2008
    Beans
    Hidden!
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: 99.99% NFS io for jbd2 on /home partition

    Quote Originally Posted by wkuhns View Post
    Server is "sync" - I've used that setting with several generations of Linux servers with perfectly acceptable performance and I'd rather not mess with it...

    The recent change is replacing my old server with a new one running xubuntu 12.04. I used ext4 for most partitions, while the previous server had used ext3.
    The ext3 file system handles sync differently than ext4. This most likely is the difference you see in performance. Ted Tso is the principal developer of ext3 and ext4 file systems. You can see Ted Tso's thoughts on sync and fsync here and here.

    The bottom line is to go with async in your mount directive if you are using ext4 file systems for your partitions.
    Last edited by capscrew; August 29th, 2013 at 06:01 PM.

  6. #6
    Join Date
    Apr 2011
    Beans
    11

    Re: 99.99% NFS io for jbd2 on /home partition

    Thanks - changing both the mount options in fstab and specifying async in /etc/exports solved the problem. As far as I can tell, changing the partition mount options was not enough.

  7. #7
    Join Date
    Dec 2008
    Beans
    Hidden!
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: 99.99% NFS io for jbd2 on /home partition

    I believe you are correct. It's the exports that need the async setting.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •