theparanoidone
January 14th, 2009, 04:24 PM
I don't belive Ubuntu should compile the standard kernel with these options:
CONFIG_HZ_100=y
CONFIG_HZ=100
They SHOULD BE for any new server or servers no older than say 3 years old:
CONFIG_HZ_1000=y
CONFIG_HZ=1000
At the very least, Ubuntu should have a standard kernel package we can apt-get that has the clock set at 1000.
Here's why:
So, we have recently spent a very long time trying to track down problems with our MySQL server.
Our requirements are high concurrency, high load, and ACID transaction compliance.
We use InnoDB in MySQL for our database. To increase the safety of the data integrity, we turn "sync-binlog=1" in the MySQL my.cnf. We notice this basically killed all performance on our server... essentialy 98% performance hit on our ubuntu64 10GB Ram with Raid-1 sas with battery backed up cache.
We tried configuring MySQL 4.1, 5.0, 5.1, 6.0 and the config logs hundreds of times. Nothing we did would improve our performance.
We finally tried to run some tests with sysbench that are heavy write benchmarks. And on a whim... we decided to try another linux distro (CentOS) just to see if that would be a factor... We appear to have hit some sort of operating system specific wall.
Sysbench 50 TRHEADS
OS ubuntu32 ubuntu64 centos32
syncbinlog 0, WB 5600 12600 3800
syncbinlog 1, WB 348 294 1700
Sysbench 1 THREAD
OS ubuntu32 ubuntu64 centos32
syncbinlog 0, WB 4500 6000 2300
syncbinlog 1, WB 3200 4356 2300
Obviously... running just 1 thread is not acceptable for a database server.
The interesting thing to note is, the number of context switches in CENTOS will peak above 10,000 where UBUNTU will struggle to get past 3,000 context switches (doesn’t matter 32 bit or 64 bit).
We also notice that CENTOS has 1,000 intr/s while the system is idle where as ubuntu has 0.
Our theory is that CENTOS is polling more often to see if it needs to switch contexts (which under heavy load… it does), and therefore, CENTOS is handling operations more efficiently. The servers we are testing on are all WriteBack enabled raid servers of different processor speeds (close enough though to get a feel for what is working an what is not working)
We have recompiled the UBUNTU kernel with the following modifications:
CONFIG_HZ_1000=y
CONFIG_HZ=1000
(whereas previously ubuntu was set to a default of 100 hz).
After making these changes… the system behaves better (see the ubuntu64KRNPTCH column):
Sysbench 50 TRHEADS
OS ubuntu32 ubuntu64 centos32 ubuntu64KRNPTCH
syncbinlog 0, WB 5600 12600 3800 14000
syncbinlog 1, WB 348 294 1700 2100
Sysbench 1 THREAD
OS ubuntu32 ubuntu64 centos32 ubuntu64KRNPTCH
syncbinlog 0, WB 4500 6000 2300 5200
syncbinlog 1, WB 3200 4356 2300 2600
For now this appears to be the culprit (what a pain in the ass… I would never have dreamed in a million years the standard kernel configuration for ubuntu was the problem).
FYI, Here are our sysbench commands (be sure to change your mysql username and password):
./sysbench --num-threads=50 --test=oltp --oltp-test-mode=complex --oltp-table-size=100000 --oltp-distinct-ranges=0 --oltp-order-ranges=0 --oltp-sum-ranges=0 --oltp-simple-ranges=0 --oltp-point-selects=0 --oltp-range-size=0 --mysql-table-engine=innodb --mysql-host=127.0.0.1 --mysql-user=ROOT --mysql-password=PASSWORD prepare
./sysbench --num-threads=50 --test=oltp --oltp-test-mode=complex --oltp-table-size=100000 --oltp-distinct-ranges=0 --oltp-order-ranges=0 --oltp-sum-ranges=0 --oltp-simple-ranges=0 --oltp-point-selects=0 --oltp-range-size=0 --mysql-table-engine=innodb --mysql-host=127.0.0.1 --mysql-user=ROOT --mysql-password=PASSWORD run
Side Notes:
We are aware that Ubuntu has a rt (realtime kernel); however, the realtime rt kernel does not appear to work on our Dell 1950/2950's (we're investigating why). Even the most basic commands like "ls" and "su" don't work. Will post more feedback on the realtime kernel here soon.
CONFIG_HZ_100=y
CONFIG_HZ=100
They SHOULD BE for any new server or servers no older than say 3 years old:
CONFIG_HZ_1000=y
CONFIG_HZ=1000
At the very least, Ubuntu should have a standard kernel package we can apt-get that has the clock set at 1000.
Here's why:
So, we have recently spent a very long time trying to track down problems with our MySQL server.
Our requirements are high concurrency, high load, and ACID transaction compliance.
We use InnoDB in MySQL for our database. To increase the safety of the data integrity, we turn "sync-binlog=1" in the MySQL my.cnf. We notice this basically killed all performance on our server... essentialy 98% performance hit on our ubuntu64 10GB Ram with Raid-1 sas with battery backed up cache.
We tried configuring MySQL 4.1, 5.0, 5.1, 6.0 and the config logs hundreds of times. Nothing we did would improve our performance.
We finally tried to run some tests with sysbench that are heavy write benchmarks. And on a whim... we decided to try another linux distro (CentOS) just to see if that would be a factor... We appear to have hit some sort of operating system specific wall.
Sysbench 50 TRHEADS
OS ubuntu32 ubuntu64 centos32
syncbinlog 0, WB 5600 12600 3800
syncbinlog 1, WB 348 294 1700
Sysbench 1 THREAD
OS ubuntu32 ubuntu64 centos32
syncbinlog 0, WB 4500 6000 2300
syncbinlog 1, WB 3200 4356 2300
Obviously... running just 1 thread is not acceptable for a database server.
The interesting thing to note is, the number of context switches in CENTOS will peak above 10,000 where UBUNTU will struggle to get past 3,000 context switches (doesn’t matter 32 bit or 64 bit).
We also notice that CENTOS has 1,000 intr/s while the system is idle where as ubuntu has 0.
Our theory is that CENTOS is polling more often to see if it needs to switch contexts (which under heavy load… it does), and therefore, CENTOS is handling operations more efficiently. The servers we are testing on are all WriteBack enabled raid servers of different processor speeds (close enough though to get a feel for what is working an what is not working)
We have recompiled the UBUNTU kernel with the following modifications:
CONFIG_HZ_1000=y
CONFIG_HZ=1000
(whereas previously ubuntu was set to a default of 100 hz).
After making these changes… the system behaves better (see the ubuntu64KRNPTCH column):
Sysbench 50 TRHEADS
OS ubuntu32 ubuntu64 centos32 ubuntu64KRNPTCH
syncbinlog 0, WB 5600 12600 3800 14000
syncbinlog 1, WB 348 294 1700 2100
Sysbench 1 THREAD
OS ubuntu32 ubuntu64 centos32 ubuntu64KRNPTCH
syncbinlog 0, WB 4500 6000 2300 5200
syncbinlog 1, WB 3200 4356 2300 2600
For now this appears to be the culprit (what a pain in the ass… I would never have dreamed in a million years the standard kernel configuration for ubuntu was the problem).
FYI, Here are our sysbench commands (be sure to change your mysql username and password):
./sysbench --num-threads=50 --test=oltp --oltp-test-mode=complex --oltp-table-size=100000 --oltp-distinct-ranges=0 --oltp-order-ranges=0 --oltp-sum-ranges=0 --oltp-simple-ranges=0 --oltp-point-selects=0 --oltp-range-size=0 --mysql-table-engine=innodb --mysql-host=127.0.0.1 --mysql-user=ROOT --mysql-password=PASSWORD prepare
./sysbench --num-threads=50 --test=oltp --oltp-test-mode=complex --oltp-table-size=100000 --oltp-distinct-ranges=0 --oltp-order-ranges=0 --oltp-sum-ranges=0 --oltp-simple-ranges=0 --oltp-point-selects=0 --oltp-range-size=0 --mysql-table-engine=innodb --mysql-host=127.0.0.1 --mysql-user=ROOT --mysql-password=PASSWORD run
Side Notes:
We are aware that Ubuntu has a rt (realtime kernel); however, the realtime rt kernel does not appear to work on our Dell 1950/2950's (we're investigating why). Even the most basic commands like "ls" and "su" don't work. Will post more feedback on the realtime kernel here soon.