Results 1 to 10 of 11

Thread: IOwait high, no disk writes, server hangs

Threaded View

  1. #1
    Join Date
    Feb 2011
    Beans
    7

    IOwait high, no disk writes, server hangs

    I run a website that a very steady flow of traffic and Im seeing recent issues that I just dont like.

    Server is 10.04.2 on a supermicro i7-950/6gb RAM with two 500gb Samsung F3 drives in a software RAID1 (1x5400, 1x7200) and for several weeks, its been running very well. Recently, Im seeing the server hang for 5-20 seconds. IOwait goes through the roof, nothing can write to the disk. Apache logs stop, redis fails to rebuild caches, mysql errors and then it continues and moves back to normal operation

    / is ext4, the kernel was 2.6.32-server-x64 but since updating to 2.6.38-server-x64, the issue has dropped from maybe once per 10 minutes to once per 15 minutes.

    3 IOstat copy/pastes show this when it hangs.

    Code:
    Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
      TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                                                                                                          
    11811 be/4 redis       0.00 B/s    0.00 B/s  0.00 % 99.99 % redis-server /etc/redis/redis.conf
        1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    
    Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
      TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                                                                                                          
      273 be/3 root        0.00 B/s    0.00 B/s  0.00 % 97.39 % [jbd2/md2-8]
    13149 be/4 redis       0.00 B/s    0.00 B/s  0.00 % 97.34 % redis-server /etc/redis/redis.conf
        1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    
    Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
      TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                                                                                                          
      260 be/4 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [md2_raid1]
     4053 be/4 mysql       0.00 B/s    0.00 B/s  0.00 % 99.99 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --~l/error.log --pid-file=/var/lib/mysql/server3.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
        1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
        2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
        3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    No smart errors or smart diags show any issues with any of the disks and kernel.log shows near nothing other than a process hang, 120 seconds about 5 days ago.
    Does anyone have any tips? Im open to reverting the FS back to ext3 if I really have to.
    Last edited by bottler; July 24th, 2011 at 06:32 PM.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •