[ubuntu] PHP processes keep overwhelming my server, eating up the CPU and making it lock up :(

Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

Wow. I just checked my single core VPS and even though it is only running three low traffic sites, the stats in top were way more conservitive than the screenshot of the terminal posted.

Code:

top - 14:13:18 up 6 days, 21:31, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 33 total, 1 running, 32 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 524288k total, 104024k used, 420264k free, 0k buffers Swap: 524288k total, 12k used, 524276k free, 73356k cached

What has me concerned is they have 65 running processes at the moment and only 6.9% CPU is id (idle) and 92.3% CPU is st (stolen time).

It sounds like whatever host they have this VM on, it cannot handle the stress of running that many processes at once.

The number of processes sounds like a configuration thing to me, but I don't know much about php-cgi as I have only really used Apache with mod_php and Nginx with php-fpm.

Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

What is the hypervisor that is being used? It seems to be a hypervisor problem (possibly due to bad hardware?) if it only gives 8% of the CPU cycles to the virtual machine. The PHP issues may be due to lack of processing power, not RAM. What other VM's are running? What does their load look like?

Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

Out of interest, I tried to create a situation on my test server where I had a very high percentage of stolen CPU time. I was unsuccessful, achieving around 10% max.

I made a VM and loaded it down like crazy, then I also loaded down like crazy the host. The VM (The snapshot was not at the max stolen that I acheived):

Code:

top - 17:18:56 up 1:29, 2 users, load average: 68.48, 68.64, 68.41 Tasks: 161 total, 69 running, 92 sleeping, 0 stopped, 0 zombie Cpu(s): 93.9%us, 0.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 5.9%st Mem: 2050976k total, 220292k used, 1830684k free, 11468k buffers Swap: 2093052k total, 0k used, 2093052k free, 80556k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1709 doug 20 0 4164 88 0 R 6 0.0 4:41.65 waiter 1685 doug 20 0 4164 88 0 S 6 0.0 4:43.02 waiter 1686 doug 20 0 4164 88 0 R 6 0.0 4:43.76 waiter 1688 doug 20 0 4164 88 0 R 6 0.0 4:45.73 waiter 1689 doug 20 0 4164 88 0 R 6 0.0 4:43.25 waiter 1694 doug 20 0 4164 88 0 R 6 0.0 4:45.58 waiter 1696 doug 20 0 4164 88 0 R 6 0.0 4:43.98 waiter 1697 doug 20 0 4164 88 0 R 6 0.0 4:43.59 waiter

The host:

Code:

top - 17:19:48 up 10 days, 7:40, 5 users, load average: 72.54, 72.71, 72.45 Tasks: 213 total, 70 running, 143 sleeping, 0 stopped, 0 zombie Cpu(s):100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16004120k total, 13766388k used, 2237732k free, 1121260k buffers Swap: 8294396k total, 102792k used, 8191604k free, 10403068k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30938 libvirt- 20 0 6274m 1.2g 6944 S 378 8.1 345:10.46 kvm 32044 doug 20 0 4164 88 0 R 7 0.0 5:17.52 waiter 32072 doug 20 0 4164 88 0 R 7 0.0 5:15.29 waiter 32045 doug 20 0 4164 88 0 R 6 0.0 5:16.25 waiter 32047 doug 20 0 4164 88 0 R 6 0.0 5:15.97 waiter 32050 doug 20 0 4164 88 0 R 6 0.0 5:15.83 waiter

Ignore that the host has swap used, that is left over from some other test. I realize this doesn't help the main thread, but if I could have achieved over 90% stolen I was going to look for accouting issues in top.

Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

My friend who does this stuff for a living (I believe) thinks it's due to the size of the forum database. The MyBB database is about 350mb, and we're thinking that some MySQL queries are taking a long time and causing things to bottleneck.

We're looking at the slow queries logs and stuff at the moment... but I think we're close.

It does work on cPanel though so that's my last resort. I'll stick this out for a few more days though :)

Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

Nevermind. :)

Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

The size of the database matters less than whether it is well-indexed. I built an application for a client where the database had over eight million records. At first it took ages to run queries, but after I spent some time with tools like EXPLAIN, I created some additional indexes, and the queries then ran nearly instantaneously.

Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

Quote:

Originally Posted by hoppipolla

We're looking at the slow queries logs and stuff at the moment... but I think we're close.

It does work on cPanel though so that's my last resort. I'll stick this out for a few more days though :)

Now the question becomes: What is the difference between the cPanel setup and the way you have it setup now?

Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

Very good observations and comments guys!

I think you're both right there!

Hm.. I know little to nothing about indexing... I may have to learn about that!

And we do still need to look into what cPanel was doing differently.

But, at least we now know the problem! I locked the forum just temporarily to keep the site up and running until we sort this!! :)

Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

Suppose you have a table called "people" with names and addresses in it that you want to search by last name. Let's call that field "last_name". Assuming that last_name is not the primary key, which is unlikely since primary keys need to be unique, you need to add another index with the CREATE INDEX command like this:

Code:

CREATE INDEX people_last ON people (last_name);

If your query uses both first and last name, you'd need this instead:

Code:

CREATE INDEX people_name ON people (last_name, first_name);

That creates an index that uses the combination of those two fields.

The name of the index is arbitrary. I usually name them with the name of the table and some mnemonic device to remind me which fields were used.

You can see the existing indexes on a table with the SHOW INDEX command.

Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

In my previous post, it might have appeared as though I was going off on a tangent, since I couldn't get the "stolen" percentage high enough to prove what I suspected.
Now, I can and the point is that with a high percentage of "stolen" time, the accounting in "top" does not reflect reality. Rather, the accounting in "top" includes the stolen time.

So, and for example from that screen shot, for PID 1653 28% CPU time and 1:29.23 total CPU time at 92.3% "stolen". It is actually 2.2% CPU time and 6.87 seconds total CPU time. (assumes constant "stolen" time, which is probably incorrect)
I suppose even 6.87 seconds is still rather horrendous for one of those php-cgi scripts. I just wanted to understand the discrepencies we were seeing in the screen shot of "top".

Example from my computer (Where the sum of the %CPU per PID is about 400% (4 CPUs are used by this VM), but 98% of that isn't really there):

Code:

top - 14:40:28 up 45 min, 2 users, load average: 68.80, 68.60, 63.70 Tasks: 161 total, 70 running, 91 sleeping, 0 stopped, 0 zombie Cpu(s): 2.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 98.0%st Mem: 2050976k total, 223752k used, 1827224k free, 20256k buffers Swap: 2093052k total, 0k used, 2093052k free, 80696k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1697 doug 20 0 4164 88 0 R 11 0.0 2:16.70 waiter 1725 doug 20 0 4164 88 0 R 11 0.0 2:11.44 waiter 1732 doug 20 0 4164 88 0 R 11 0.0 2:12.57 waiter 1749 doug 20 0 4164 88 0 R 11 0.0 2:15.49 waiter 1708 doug 20 0 4164 88 0 R 10 0.0 2:14.86 waiter 1747 doug 20 0 4164 88 0 R 10 0.0 2:08.70 waiter 1722 doug 20 0 4164 88 0 R 9 0.0 2:11.07 waiter 1729 doug 20 0 4164 88 0 R 8 0.0 2:12.01 waiter 1757 doug 20 0 4164 88 0 R 8 0.0 2:10.13 waiter 1705 doug 20 0 4164 88 0 R 7 0.0 2:10.17 waiter 1698 doug 20 0 4164 88 0 R 7 0.0 2:17.56 waiter 1699 doug 20 0 4164 88 0 R 7 0.0 2:16.93 waiter 1710 doug 20 0 4164 88 0 R 7 0.0 2:14.10 waiter 1713 doug 20 0 4164 88 0 R 7 0.0 2:13.69 waiter 1718 doug 20 0 4164 88 0 R 7 0.0 2:14.67 waiter 1723 doug 20 0 4164 88 0 R 7 0.0 2:14.50 waiter 1735 doug 20 0 4164 88 0 R 7 0.0 2:09.35 waiter 1738 doug 20 0 4164 88 0 R 7 0.0 2:10.52 waiter 1741 doug 20 0 4164 88 0 R 7 0.0 2:09.20 waiter 1743 doug 20 0 4164 88 0 R 7 0.0 2:10.35 waiter 1748 doug 20 0 4164 88 0 R 7 0.0 2:10.09 waiter 1755 doug 20 0 4164 88 0 R 7 0.0 2:10.20 waiter 1758 doug 20 0 4164 88 0 R 7 0.0 2:08.76 waiter 1760 doug 20 0 4164 88 0 R 7 0.0 2:09.92 waiter 1763 doug 20 0 4164 88 0 R 7 0.0 2:08.86 waiter 1765 doug 20 0 4164 88 0 R 7 0.0 2:10.54 waiter 1736 doug 20 0 4164 88 0 R 6 0.0 2:09.31 waiter 1737 doug 20 0 4164 88 0 R 6 0.0 2:10.51 waiter

I suspect, but am not sure, that when things work when you use cPanel, you are actually just setting up the VM correctly to not have all of its processor time re-allocated elsewhere. (Also, I suspcect Seiji's ideas will help a lot also).