I run a website that a very steady flow of traffic and Im seeing recent issues that I just dont like.
Server is 10.04.2 on a supermicro i7-950/6gb RAM with two 500gb Samsung F3 drives in a software RAID1 (1x5400, 1x7200) and for several weeks, its been running very well. Recently, Im seeing the server hang for 5-20 seconds. IOwait goes through the roof, nothing can write to the disk. Apache logs stop, redis fails to rebuild caches, mysql errors and then it continues and moves back to normal operation
/ is ext4, the kernel was 2.6.32-server-x64 but since updating to 2.6.38-server-x64, the issue has dropped from maybe once per 10 minutes to once per 15 minutes.
3 IOstat copy/pastes show this when it hangs.
Code:
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
11811 be/4 redis 0.00 B/s 0.00 B/s 0.00 % 99.99 % redis-server /etc/redis/redis.conf
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
273 be/3 root 0.00 B/s 0.00 B/s 0.00 % 97.39 % [jbd2/md2-8]
13149 be/4 redis 0.00 B/s 0.00 B/s 0.00 % 97.34 % redis-server /etc/redis/redis.conf
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
260 be/4 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % [md2_raid1]
4053 be/4 mysql 0.00 B/s 0.00 B/s 0.00 % 99.99 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --~l/error.log --pid-file=/var/lib/mysql/server3.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
No smart errors or smart diags show any issues with any of the disks and kernel.log shows near nothing other than a process hang, 120 seconds about 5 days ago.
Does anyone have any tips? Im open to reverting the FS back to ext3 if I really have to.
Bookmarks