Results 1 to 7 of 7

Thread: 16.04.02 Cratered IO perf under load

  1. #1
    Join Date
    Mar 2006
    Beans
    214

    16.04.02 Cratered IO perf under load

    Thanks in advance for any pointers!

    Just yesterday I installed ubuntu desktop 16.04.02 32 bit on an existing system previously running 64 bit 12.04. I installed system and user data to a new SSD in the expectation of performance improvements.

    Symptoms are severe lagging & unresponsiveness under any kind of sustained IO load. Rebooting the system seems to clear the symptoms for a while.
    - Using rsync to copy 00's of GB of user data from a HDD to the user partition on the SSD took all afternoon and that was only after deliberately interrupting rsync and rebooting, periodically
    - Using mksquashfs as part of preparing an ltsp installation caused multiple running procs to be killed, then the system froze completely, requiring a hard reset. The operation did complete afterwards, but only by running it on a completely quiet system. There's 16 GB RAM in the box, so available RAM wasn't an issue!
    - From a user point of view applications "pause" when starting, and UI effects seem to wait a few seconds before completing.

    Monitoring IOWAIT shows readings of 25 - 50% typically under load, and the write speed to the SSD drops from maybe 200 - 300 MB/s to 2 MB/s!

    How would I go about starting to investigate the causes and do something about it? There must be something stupid I've done or not done along the way.

    My fstab is:

    Code:
    # <file system> <mount point>   <type>  <options>       <dump>  <pass>
    # / was on /dev/sdd1 during installation
    UUID=09979c08-9b79-4c49-a5b3-9138dc872623 /               ext4    noatime,discard,data=ordered,errors=remount-ro 0       1
    # /home was on /dev/sdd6 during installation
    UUID=83d3174a-99a1-4f3f-b469-9098ff601b26 /home           ext4    noatime,discard,data=ordered,defaults        0       2
    # swap was on /dev/sda1 during installation
    UUID=b8ee76ff-a1df-45ce-961e-bf53c9a37992 none            swap    sw              0       0
    # swap was on /dev/sdd5 during installation
    UUID=bf886f6a-1407-4a69-b4bd-d06ea5232b02 none            swap    sw              0       0
    The SSD was on /dev/sdd during the installation, but I moved some SATA cables around inside the box and now it is on /dev/sda. The /dev/sdaa in the fstab is swap on a legacy raid 0 partition md1 which is mount on demand (no idea at all why that is there, but the installer seems to have insisted on putting it there).

    Some useful information:

    Code:
    ~$ df -kh
    Filesystem      Size  Used Avail Use% Mounted on
    udev            7.9G     0  7.9G   0% /dev
    tmpfs           1.6G  9.6M  1.6G   1% /run
    /dev/sda1        92G  8.3G   79G  10% /
    tmpfs           7.9G  178M  7.8G   3% /dev/shm
    tmpfs           5.0M  4.0K  5.0M   1% /run/lock
    tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
    /dev/sda6       1.8T  614G  1.1T  37% /home
    tmpfs           1.6G   60K  1.6G   1% /run/user/1000
    Code:
    ~$ sudo hdparm -Tt /dev/sda
    
    /dev/sda:
     Timing cached reads:   18580 MB in  2.00 seconds = 9302.80 MB/sec
     Timing buffered disk reads: 860 MB in  3.00 seconds = 286.42 MB/sec
    Code:
    ~$ sudo hdparm -v /dev/sda 
    
    /dev/sda:
     multcount     =  0 (off)
     IO_support    =  0 (default) 
     readonly      =  0 (off)
     readahead     = 256 (on)
     geometry      = 249281/255/63, sectors = 4004704368, start = 0
    Forgot to add the SSD is a Crucial_CT2050MX300SSD1 (M0CR031), and root and /home are both on the SSD, under separate partitions.

    Here's the output from bonnie++ running as default:

    Code:
    Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
    Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
    Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
    aammscott       32G   751  99 23076   2 91499  11  2553 100 280175  18  3697  43
    Latency             22050us     127ms     160ms    3606us   39999us   15383us
    Version  1.97       ------Sequential Create------ --------Random Create--------
    aammscott           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
                  files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                     16  6710   5 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
    Latency              1154us     854us    1232us     375us      12us      32us
    1.97,1.97,aammscott,1,1496730064,32G,,751,99,23076,2,91499,11,2553,100,280175,18,3697,43,16,,,,,6710,5,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,22050us,127ms,160ms,3606us,39999us,15383us,1154us,854us,1232us,375us,12us,32us
    Last edited by howefield; June 6th, 2017 at 09:57 AM. Reason: posts combined and code tags added..

  2. #2
    Join Date
    Mar 2006
    Beans
    214

    Re: 16.04.02 Cratered IO perf under load

    Assistance with this will be appreciated.

    Currently investigating any possible link to occasional boot failures.

    When the system has been running a while (more than say 12 hours under use) then the iowait issue is more likely to manifest itself. A reboot "cleans" the issue.

  3. #3
    Join Date
    Mar 2006
    Beans
    214

    Re: 16.04.02 Cratered IO perf under load

    Output of hdparm -i

    Code:
    sudo hdparm -i /dev/sda
    
    /dev/sda:
    
     Model=Crucial_CT2050MX300SSD1, FwRev=M0CR031, SerialNo=16391409EAD9
     Config={ Fixed DTR>10Mbs }
     RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=0
     BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=off
     CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=4004704368
     IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
     PIO modes:  pio0 pio3 pio4 
     DMA modes:  mdma0 mdma1 mdma2 
     UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
     AdvancedPM=yes: unknown setting WriteCache=enabled
     Drive conforms to: unknown:  ATA/ATAPI-3,4,5,6,7
    
     * signifies the current active mode

  4. #4
    Join Date
    Mar 2006
    Beans
    214

    Re: 16.04.02 Cratered IO perf under load

    Output from tune2fs:

    Code:
    sudo tune2fs -l /dev/sda1
    tune2fs 1.42.13 (17-May-2015)
    Filesystem volume name:   <none>
    Last mounted on:          /
    Filesystem UUID:          09979c08-9b79-4c49-a5b3-9138dc872623
    Filesystem magic number:  0xEF53
    Filesystem revision #:    1 (dynamic)
    Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
    Filesystem flags:         signed_directory_hash 
    Default mount options:    user_xattr acl
    Filesystem state:         clean
    Errors behavior:          Continue
    Filesystem OS type:       Linux
    Inode count:              6111232
    Block count:              24413696
    Reserved block count:     1220684
    Free blocks:              20769714
    Free inodes:              5737913
    First block:              0
    Block size:               4096
    Fragment size:            4096
    Reserved GDT blocks:      1018
    Blocks per group:         32768
    Fragments per group:      32768
    Inodes per group:         8192
    Inode blocks per group:   512
    Flex block group size:    16
    Filesystem created:       Mon Jun  5 10:18:40 2017
    Last mount time:          Thu Jun  8 08:34:22 2017
    Last write time:          Thu Jun  8 08:34:22 2017
    Mount count:              55
    Maximum mount count:      -1
    Last checked:             Mon Jun  5 10:18:40 2017
    Check interval:           0 (<none>)
    Lifetime writes:          217 GB
    Reserved blocks uid:      0 (user root)
    Reserved blocks gid:      0 (group root)
    First inode:              11
    Inode size:	          256
    Required extra isize:     28
    Desired extra isize:      28
    Journal inode:            8
    First orphan inode:       5508067
    Default directory hash:   half_md4
    Directory Hash Seed:      665e7dca-7d56-4355-b02f-a1b379bb7ea2
    Journal backup:           inode blocks
    Code:
    sudo tune2fs -l /dev/sda6
    tune2fs 1.42.13 (17-May-2015)
    Filesystem volume name:   <none>
    Last mounted on:          /home
    Filesystem UUID:          83d3174a-99a1-4f3f-b469-9098ff601b26
    Filesystem magic number:  0xEF53
    Filesystem revision #:    1 (dynamic)
    Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
    Filesystem flags:         signed_directory_hash 
    Default mount options:    user_xattr acl
    Filesystem state:         clean
    Errors behavior:          Continue
    Filesystem OS type:       Linux
    Inode count:              118923264
    Block count:              475685376
    Reserved block count:     23784268
    Free blocks:              307283499
    Free inodes:              118141843
    First block:              0
    Block size:               4096
    Fragment size:            4096
    Reserved GDT blocks:      910
    Blocks per group:         32768
    Fragments per group:      32768
    Inodes per group:         8192
    Inode blocks per group:   512
    Flex block group size:    16
    Filesystem created:       Mon Jun  5 10:18:42 2017
    Last mount time:          Thu Jun  8 08:34:23 2017
    Last write time:          Thu Jun  8 08:34:23 2017
    Mount count:              55
    Maximum mount count:      -1
    Last checked:             Mon Jun  5 10:18:42 2017
    Check interval:           0 (<none>)
    Lifetime writes:          3743 GB
    Reserved blocks uid:      0 (user root)
    Reserved blocks gid:      0 (group root)
    First inode:              11
    Inode size:	          256
    Required extra isize:     28
    Desired extra isize:      28
    Journal inode:            8
    First orphan inode:       59774267
    Default directory hash:   half_md4
    Directory Hash Seed:      46a892c0-d03a-42a0-96bc-ddb8271c45d2
    Journal backup:           inode blocks

  5. #5
    Join Date
    Mar 2006
    Beans
    214

    Re: 16.04.02 Cratered IO perf under load

    Have been backing up files with rsynch -av to an external HDD sized to match /home formatted ext4. This is over USB 2.0.

    There are a lot of large and small files to back up. Everything from mailboxes to thumbnails.

    For the first few minutes, rsynch does its job, transferring data at a solid 30 - 40 MB / s. However, after some time, the file transfer rate drops off a cliff, down to 3 MB / s if I am lucky. At the same time, general responsiveness on the computer becomes more of an issue.

    This never used to happen under 12.04.

    I find that *REBOOTING* seems to 'recover' the performance, and I can re-mount the HDD and start transferring files again. Until the next time ...

    This is obviously unworkable in anything other than the short term.

    Some pointers will be really appreciated!

  6. #6
    Join Date
    Mar 2006
    Beans
    214

    Re: 16.04.02 Cratered IO perf under load

    UPDATE:

    While I won't mark this thread as '[SOLVED]' just yet, the following does seem to be helping with maintaining IO perf without frequent reboots.

    Clues for me were the realisation that (a) the SSD was not the common factor in IO perf issues, and (b) a reboot temporarily cleared the issues. Obviously, I don't want to keep rebooting the system to assure half way reasonable performance and responsiveness, especially when away, but this helped me frame the right question for Google: "Ubuntu 16.04 IO performance reboot".

    The following links have been instrumental:

    https://askubuntu.com/questions/6092...es-doesnt-work

    https://unix.stackexchange.com/quest...-drop-in-linux

    The underlying cause seems to be related to limitations in the way that 32 bit kernels handle caching. It would appear that if there are a lot of entries in the cache - not necessarily that there is a large cache - then IO performance suffers.

    I installed 32 bit as a consequence of applying LTSP-PNP with 32-bit clients.

    I've added the following to /etc/crontab - after having backed it up first:

    Code:
    # Added since install
    # 10 June 2017: Clear the cache every 5 minutes to stop iowait blocking
    */5 *	* * *	root	sync && echo "3" | tee /proc/sys/vm/drop_caches
    What this does is every 5 minutes first syncs to storage, then clears the cache.

    So far, this seems to be keeping the users happy.

  7. #7
    Join Date
    Mar 2006
    Beans
    214

    Re: 16.04.02 Cratered IO perf under load


Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •