Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 29

Thread: PHP processes keep overwhelming my server, eating up the CPU and making it lock up :(

  1. #11
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    Wow. I just checked my single core VPS and even though it is only running three low traffic sites, the stats in top were way more conservitive than the screenshot of the terminal posted.

    Code:
    top - 14:13:18 up 6 days, 21:31,  1 user,  load average: 0.00, 0.00, 0.00
    Tasks:  33 total,   1 running,  32 sleeping,   0 stopped,   0 zombie
    Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:    524288k total,   104024k used,   420264k free,        0k buffers
    Swap:   524288k total,       12k used,   524276k free,    73356k cached
    What has me concerned is they have 65 running processes at the moment and only 6.9% CPU is id (idle) and 92.3% CPU is st (stolen time).

    It sounds like whatever host they have this VM on, it cannot handle the stress of running that many processes at once.

    The number of processes sounds like a configuration thing to me, but I don't know much about php-cgi as I have only really used Apache with mod_php and Nginx with php-fpm.
    Last edited by CharlesA; April 8th, 2013 at 07:20 PM.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  2. #12
    Join Date
    Feb 2007
    Location
    West Hills CA
    Beans
    8,088
    Distro
    Ubuntu 12.10 Quantal Quetzal

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    What is the hypervisor that is being used? It seems to be a hypervisor problem (possibly due to bad hardware?) if it only gives 8% of the CPU cycles to the virtual machine. The PHP issues may be due to lack of processing power, not RAM. What other VM's are running? What does their load look like?
    -------------------------------------
    Oooh Shiny: PopularPages

    Unumquodque potest reparantur. Patientia sit virtus.

  3. #13
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    1,452
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    Out of interest, I tried to create a situation on my test server where I had a very high percentage of stolen CPU time. I was unsuccessful, achieving around 10% max.

    I made a VM and loaded it down like crazy, then I also loaded down like crazy the host. The VM (The snapshot was not at the max stolen that I acheived):
    Code:
    top - 17:18:56 up  1:29,  2 users,  load average: 68.48, 68.64, 68.41
    Tasks: 161 total,  69 running,  92 sleeping,   0 stopped,   0 zombie
    Cpu(s): 93.9%us,  0.2%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  5.9%st
    Mem:   2050976k total,   220292k used,  1830684k free,    11468k buffers
    Swap:  2093052k total,        0k used,  2093052k free,    80556k cached
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     1709 doug      20   0  4164   88    0 R    6  0.0   4:41.65 waiter
     1685 doug      20   0  4164   88    0 S    6  0.0   4:43.02 waiter
     1686 doug      20   0  4164   88    0 R    6  0.0   4:43.76 waiter
     1688 doug      20   0  4164   88    0 R    6  0.0   4:45.73 waiter
     1689 doug      20   0  4164   88    0 R    6  0.0   4:43.25 waiter
     1694 doug      20   0  4164   88    0 R    6  0.0   4:45.58 waiter
     1696 doug      20   0  4164   88    0 R    6  0.0   4:43.98 waiter
     1697 doug      20   0  4164   88    0 R    6  0.0   4:43.59 waiter
    The host:
    Code:
    top - 17:19:48 up 10 days,  7:40,  5 users,  load average: 72.54, 72.71, 72.45
    Tasks: 213 total,  70 running, 143 sleeping,   0 stopped,   0 zombie
    Cpu(s):100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:  16004120k total, 13766388k used,  2237732k free,  1121260k buffers
    Swap:  8294396k total,   102792k used,  8191604k free, 10403068k cached
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    30938 libvirt-  20   0 6274m 1.2g 6944 S  378  8.1 345:10.46 kvm
    32044 doug      20   0  4164   88    0 R    7  0.0   5:17.52 waiter
    32072 doug      20   0  4164   88    0 R    7  0.0   5:15.29 waiter
    32045 doug      20   0  4164   88    0 R    6  0.0   5:16.25 waiter
    32047 doug      20   0  4164   88    0 R    6  0.0   5:15.97 waiter
    32050 doug      20   0  4164   88    0 R    6  0.0   5:15.83 waiter
    Ignore that the host has swap used, that is left over from some other test. I realize this doesn't help the main thread, but if I could have achieved over 90% stolen I was going to look for accouting issues in top.
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

  4. #14
    Join Date
    Jun 2007
    Location
    Kent, UK
    Beans
    214
    Distro
    Ubuntu

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    My friend who does this stuff for a living (I believe) thinks it's due to the size of the forum database. The MyBB database is about 350mb, and we're thinking that some MySQL queries are taking a long time and causing things to bottleneck.

    We're looking at the slow queries logs and stuff at the moment... but I think we're close.

    It does work on cPanel though so that's my last resort. I'll stick this out for a few more days though
    Registered Ubuntu user #28880

  5. #15
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    Nevermind.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  6. #16
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    8,835
    Distro
    Kubuntu 14.04 Trusty Tahr

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    The size of the database matters less than whether it is well-indexed. I built an application for a client where the database had over eight million records. At first it took ages to run queries, but after I spent some time with tools like EXPLAIN, I created some additional indexes, and the queries then ran nearly instantaneously.
    If you ask for help, please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

  7. #17
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    Quote Originally Posted by hoppipolla View Post
    We're looking at the slow queries logs and stuff at the moment... but I think we're close.

    It does work on cPanel though so that's my last resort. I'll stick this out for a few more days though
    Now the question becomes: What is the difference between the cPanel setup and the way you have it setup now?
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  8. #18
    Join Date
    Jun 2007
    Location
    Kent, UK
    Beans
    214
    Distro
    Ubuntu

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    Very good observations and comments guys!

    I think you're both right there!

    Hm.. I know little to nothing about indexing... I may have to learn about that!

    And we do still need to look into what cPanel was doing differently.

    But, at least we now know the problem! I locked the forum just temporarily to keep the site up and running until we sort this!!
    Registered Ubuntu user #28880

  9. #19
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    8,835
    Distro
    Kubuntu 14.04 Trusty Tahr

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    Suppose you have a table called "people" with names and addresses in it that you want to search by last name. Let's call that field "last_name". Assuming that last_name is not the primary key, which is unlikely since primary keys need to be unique, you need to add another index with the CREATE INDEX command like this:

    Code:
    CREATE INDEX people_last ON people (last_name);
    If your query uses both first and last name, you'd need this instead:

    Code:
    CREATE INDEX people_name ON people (last_name, first_name);
    That creates an index that uses the combination of those two fields.

    The name of the index is arbitrary. I usually name them with the name of the table and some mnemonic device to remind me which fields were used.

    You can see the existing indexes on a table with the SHOW INDEX command.
    Last edited by SeijiSensei; April 9th, 2013 at 07:22 PM.
    If you ask for help, please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

  10. #20
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    1,452
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    In my previous post, it might have appeared as though I was going off on a tangent, since I couldn't get the "stolen" percentage high enough to prove what I suspected.
    Now, I can and the point is that with a high percentage of "stolen" time, the accounting in "top" does not reflect reality. Rather, the accounting in "top" includes the stolen time.

    So, and for example from that screen shot, for PID 1653 28% CPU time and 1:29.23 total CPU time at 92.3% "stolen". It is actually 2.2% CPU time and 6.87 seconds total CPU time. (assumes constant "stolen" time, which is probably incorrect)
    I suppose even 6.87 seconds is still rather horrendous for one of those php-cgi scripts. I just wanted to understand the discrepencies we were seeing in the screen shot of "top".

    Example from my computer (Where the sum of the %CPU per PID is about 400% (4 CPUs are used by this VM), but 98% of that isn't really there):
    Code:
    top - 14:40:28 up 45 min,  2 users,  load average: 68.80, 68.60, 63.70
    Tasks: 161 total,  70 running,  91 sleeping,   0 stopped,   0 zombie
    Cpu(s):  2.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 98.0%st
    Mem:   2050976k total,   223752k used,  1827224k free,    20256k buffers
    Swap:  2093052k total,        0k used,  2093052k free,    80696k cached
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     1697 doug      20   0  4164   88    0 R   11  0.0   2:16.70 waiter
     1725 doug      20   0  4164   88    0 R   11  0.0   2:11.44 waiter
     1732 doug      20   0  4164   88    0 R   11  0.0   2:12.57 waiter
     1749 doug      20   0  4164   88    0 R   11  0.0   2:15.49 waiter
     1708 doug      20   0  4164   88    0 R   10  0.0   2:14.86 waiter
     1747 doug      20   0  4164   88    0 R   10  0.0   2:08.70 waiter
     1722 doug      20   0  4164   88    0 R    9  0.0   2:11.07 waiter
     1729 doug      20   0  4164   88    0 R    8  0.0   2:12.01 waiter
     1757 doug      20   0  4164   88    0 R    8  0.0   2:10.13 waiter
     1705 doug      20   0  4164   88    0 R    7  0.0   2:10.17 waiter
     1698 doug      20   0  4164   88    0 R    7  0.0   2:17.56 waiter
     1699 doug      20   0  4164   88    0 R    7  0.0   2:16.93 waiter
     1710 doug      20   0  4164   88    0 R    7  0.0   2:14.10 waiter
     1713 doug      20   0  4164   88    0 R    7  0.0   2:13.69 waiter
     1718 doug      20   0  4164   88    0 R    7  0.0   2:14.67 waiter
     1723 doug      20   0  4164   88    0 R    7  0.0   2:14.50 waiter
     1735 doug      20   0  4164   88    0 R    7  0.0   2:09.35 waiter
     1738 doug      20   0  4164   88    0 R    7  0.0   2:10.52 waiter
     1741 doug      20   0  4164   88    0 R    7  0.0   2:09.20 waiter
     1743 doug      20   0  4164   88    0 R    7  0.0   2:10.35 waiter
     1748 doug      20   0  4164   88    0 R    7  0.0   2:10.09 waiter
     1755 doug      20   0  4164   88    0 R    7  0.0   2:10.20 waiter
     1758 doug      20   0  4164   88    0 R    7  0.0   2:08.76 waiter
     1760 doug      20   0  4164   88    0 R    7  0.0   2:09.92 waiter
     1763 doug      20   0  4164   88    0 R    7  0.0   2:08.86 waiter
     1765 doug      20   0  4164   88    0 R    7  0.0   2:10.54 waiter
     1736 doug      20   0  4164   88    0 R    6  0.0   2:09.31 waiter
     1737 doug      20   0  4164   88    0 R    6  0.0   2:10.51 waiter
    I suspect, but am not sure, that when things work when you use cPanel, you are actually just setting up the VM correctly to not have all of its processor time re-allocated elsewhere. (Also, I suspcect Seiji's ideas will help a lot also).
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

Page 2 of 3 FirstFirst 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •