Page 1 of 3 123 LastLast
Results 1 to 10 of 29

Thread: PHP processes keep overwhelming my server, eating up the CPU and making it lock up :(

  1. #1
    Join Date
    Jun 2007
    Location
    Kent, UK
    Beans
    214
    Distro
    Ubuntu

    PHP processes keep overwhelming my server, eating up the CPU and making it lock up :(

    Heya guys.

    I set up a Debian Squeeze (I hope you don't mind me posting here - I love Ubuntu too and they are very similar, plus you have a great community here! I actually wish I used Ubuntu server instead now to be honest!) Lighttpd server with PHP and MySQL, and also with XCache and Varnish set up.

    I am quite new to this, but have tried as hard as I can to resolve this problem with no success.

    Whether I use Lighttpd or Nginx or XCache or APC the problem remains.

    Basically the server chugs along processing PHP perfectly fine, the RAM looks fine etc. Then suddenly it seems to grind to a halt and the PHP-CGI (or FPM) processes stack up and stack up, eating up all of my quad core CPU power and causing the whole server to become almost entirely non-responsive.

    I have 4 sites on my server, 3 of which are quite low traffic and demand, and one of which is quite intensive and busy. So it's almost definitely that site causing it. However we got that site working on this (VPS) server before with a cPanel setup (that I want to avoid now due to cost and preference of Debian, etc.

    Even when I remove all AJAX from the intensive site and lower traffic, it still does it.

    There are also no obvious errors reported in the PHP, MySQL or lighttpd logs (I got them all working and check them regularly).

    Here is the PHP part of my lighttpd.conf:

    http://pastebin.com/ThXAN7md

    (max requests set to 500 as recommended here, not 100% sure it's correct, but of course the problem happened in nginx too anyway!)

    and here is my php.ini:

    http://pastebin.com/8ijyY1d7

    and my MySQL my.cnf:

    http://pastebin.com/E9Jq02v5

    Any thoughts? XCache only has 64mb allocated to it but that's what they recommended to start with and it never seems to use it all in its admin CP (that I notice). Both Varnish and XCache are working fine I think and the sites respond VERY fast until this lock up event happens (which it predictably always does).


    Thanks very much guys

    Hoppi
    Registered Ubuntu user #28880

  2. #2
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    9,041
    Distro
    Kubuntu 14.04 Trusty Tahr

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    Sometimes it is contention over the database server that causes problems like these. As a PostgreSQL user, I cannot help with MySQL, but do a search for methods of tuning that program and see if you need to raise some of the defaults.

    Also how well indexed is this database? Have you tried running all the queries in the PHP scripts from the command prompt with the mysql command-line client? Are any of them comparatively slow? Perhaps some queries rely on non-indexed fields which can take a long time to process. Try using EXPLAIN to diagnose these problems.
    Last edited by SeijiSensei; April 7th, 2013 at 07:31 PM.
    If you ask for help, please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

  3. #3
    Join Date
    Jun 2007
    Location
    Kent, UK
    Beans
    214
    Distro
    Ubuntu

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    Here's what it looks like when it happens:

    http://snag.gy/VzRm8.jpg

    It usually stays like that for ages shuffling processes loads and the server may never recover until I reset it manually.

    There probably are quite a few PHP scripts I wouldn't know where to begin ._.

    I'll take a look but I'm feeling a tad out of my depth right now lol
    Registered Ubuntu user #28880

  4. #4
    Join Date
    Jun 2007
    Location
    Kent, UK
    Beans
    214
    Distro
    Ubuntu

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    I'm really close to just giving up and installing cPanel but I don't want to
    Registered Ubuntu user #28880

  5. #5
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    9,041
    Distro
    Kubuntu 14.04 Trusty Tahr

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    You managed to hit a long-term load average over 40 in just 45 minutes? I don't see how installing cpanel will fix that. I get worried when my server's load average gets above five for any extended period.

    Something is seriously wrong with the configuration of this server. I'm an Apache+mod_php user myself, so I cannot say what that might be. Spawning all those cgi-bin processes has to be part of the problem. Have you ever tried using Apache with mod_php instead of the more "exotic" options you describe?
    If you ask for help, please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

  6. #6
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    1,614
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    Isn't the problem less to do with the VM itself and more to do with the 92.3% stolen CPU time? (the right most number on the "top" cpu line.) For whatever reason, there isn't much CPU time being made available for this machine.

    Seiji: It probably isn't related at all, but on my i7 based 12.04.2 server, I have now had two occurances of a sudden instantanious jump in load average, such that when reverse calculated, knowing the IIR (Infinite Impulse Response) time constants would have been a one time load hit of around 1024. The second one I was trying to re-create, and thought I had the formula to do so, but no. However, I did catch the event, see plot below. More on this at some later date, if I am able to make progress. - Actualy the numbers suggest that hoppipolla's load issues are more sustained.
    Attached Images Attached Images
    Last edited by Doug S; April 8th, 2013 at 06:53 AM. Reason: had left and right reversed for top CPU
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

  7. #7
    Join Date
    Feb 2007
    Location
    West Hills CA
    Beans
    8,582
    Distro
    Ubuntu 12.10 Quantal Quetzal

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    Your PHP cache is set to the default 128MB which normally would be sufficient.

    ; Maximum amount of memory a script may consume (128MB)
    ; http://php.net/memory-limit
    memory_limit = 128M

    Try bumping to 256M and see if the behavior changes. How much RAM on this system? Are you hitting swap?

    Code:
    free
    Some other possible tweaks:


    [mysqlnd]
    ; Enable / Disable collection of general statistics by mysqlnd which can be
    ; used to tune and monitor MySQL operations.
    ; http://php.net/mysqlnd.collect_statistics
    mysqlnd.collect_statistics = On

    ; Enable / Disable collection of memory usage statistics by mysqlnd which can be
    ; used to tune and monitor MySQL operations.
    ; http://php.net/mysqlnd.collect_memory_statistics
    mysqlnd.collect_memory_statistics = Off

    ; Size of a pre-allocated buffer used when sending commands to MySQL in bytes.
    ; http://php.net/mysqlnd.net_cmd_buffer_size
    ;mysqlnd.net_cmd_buffer_size = 2048

    ; Size of a pre-allocated buffer used for reading data sent by the server in
    ; bytes.
    ; http://php.net/mysqlnd.net_read_buffer_size
    ;mysqlnd.net_read_buffer_size = 32768

    mysqlnd.net_read_timeout = 30
    Last edited by tgalati4; April 8th, 2013 at 03:13 AM.
    -------------------------------------
    Oooh Shiny: PopularPages

    Unumquodque potest reparantur. Patientia sit virtus.

  8. #8
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    9,041
    Distro
    Kubuntu 14.04 Trusty Tahr

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    The screenshot he posted above shows he has 2 GB of memory and 5 GB of swap, none of which is in use. He has almost a gigabyte devoted to disk caching. So I doubt he is running out of memory.

    Quote Originally Posted by Doug S
    Actualy the numbers suggest that hoppipolla's load issues are more sustained.
    The screenshot shows the machine has been up only 45 minutes, yet it is already entirely overwhelmed with an instantaneous load average in the 70's and the long-term figure around 45. As I said, I never run cgi-bin applications, but I wonder if they carry more overhead than running PHP as a module where a lot of memory is shared, and there is no need to bring up and tear down a separate application for each request.
    If you ask for help, please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

  9. #9
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    1,614
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    The screenshot shows the machine has been up only 45 minutes, yet it is already entirely overwhelmed with an instantaneous load average in the 70's and the long-term figure around 45.
    I agree, however the "st" number suggests it is not getting much CPU, so it is not surprising that it is falling further and further behind.
    hoppipolla: Is this a virtual computer? the top '"st" number suggests that it is.
    And
    with an instantaneous load average in the 70's
    That is actually the 1 minute load average, still heavily filtered. The 1 minute filter is roughly (with a once per 5 seconds sample rate)(it isn't actually calculated this way):
    Code:
    Load_n = Load * 0.08 + Load_old * 0.92
    And all I was trying say, rather poorly, with my example, is that I have cases where I don't believe the reported load average.
    Last edited by Doug S; April 8th, 2013 at 04:01 PM. Reason: added question
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

  10. #10
    Join Date
    Feb 2007
    Location
    West Hills CA
    Beans
    8,582
    Distro
    Ubuntu 12.10 Quantal Quetzal

    Re: PHP processes keep overwhelming my server, eating up the CPU and making it lock u

    How much RAM is allocated to the virtual machine that is running this instance?
    -------------------------------------
    Oooh Shiny: PopularPages

    Unumquodque potest reparantur. Patientia sit virtus.

Page 1 of 3 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •