Web server crashing every few days

[koenn]

Look at your load average, the 15 minute time constant one is 76!!!! Your swap space is full, and the memory is consumed. Your server appears to be very very severly overloaded. Of course, once it starts swapping a lot, things gets worse, as more and more time is spent just swapping.

Looks like all available swap space has been used up as well.
There's no way that system can stay up.

[lisati]

I had a similar out of memory problem a couple of months back. The solution I put in place, which might not suit someone with a busy site, was to look for a section in apache2.conf that began <IfModule mpm_prefork_module> and comment out the line MaxClients 150, and add a new line in that section:

MaxClients 50

[koenn]

booted it up again and it booted fine and is working for the time being but there was an error during boot up regarding Apache just before the log on screen:
[...]

I guess you'd have to look into this as well. A shipload of Apache processes after Apache announced "undefined results" (read; unpredictable behavior) - there might be a connection there.

Also, any reason why you have both mysql and postgres ? Unless there's a specific reason you need two different database engines, this (to me) suggests a certain sloppiness in the setup, which in turn hints at "all sorts of things could be going on there, it's gonna be hell to figure it all out"

[lisati]

Also, any reason why you have both mysql and postgres ? Unless there's a specific reason you need two different database engines, this (to me) suggests a certain sloppiness in the setup, which in turn hints at "all sorts of things could be going on there, it's gonna be hell to figure it all out"

Good catch: unless there's good reason to do otherwise, I'd recommend choosing between mysql and postgres. It might be coincidence, but having both can potentially introduce opportunities for conflicts.

[SeijiSensei]

I have often run both MySQL and PostgreSQL on servers. They listen on separate ports, have entirely separate data directories, and generate no conflicts that I can see. I prefer PostgreSQL but some web applications presume MySQL.

As I said before, a lot more physical memory and a lot more swap would do wonders.

[tgalati4]

I'm glad you were able to take a picture of the top screen as it crashed. You definitely need more RAM. You have a memory leak or runaway apache/postgres/mysql processes. The logs must show something--you have logs for apache, mysql, and postgres. What is generating all of the activity? Is it strictly traffic or an internal problem?

[koenn]

I have often run both MySQL and PostgreSQL on servers.

I'm not so questioning the possibility of having multiple DBMS and certainly not your ability to setup and run such a configuration.
I'm just wondering whether it, in the OP's case, has been done deliberately and correctly, or if we should treat it as an indication that the whole system might be an unpredictable mess.

As I said before, a lot more physical memory and a lot more swap would do wonders.

No doubt about it.
But the OP saying " it had been working fine for the last three years before any problems showed up" and " there is less activity now that there was two years ago" makes me wonder whether there could be more to it than just not enog RAm and swap. Denial of Servide ? Site harvesters ? configuration issues causing apache to spwan child processes in an endless loop ? ... just thinking out loud here.

[self-defence]

First thing is the traffic on the gateway doesn't support the activity supposed to be on the server.
There are a few websites running on the server but nothing that could have that kind of traffic. They mostly use MySql but there was one that used PostgreSql that's why it's there. But It could just as well be removed since it is no longer critical.

I checked the Apache access log and found something a little strange. There were a lot of requests from 127.0.0.1 entered as denied. And in the error log some requests to MySql that were not finding tables. I'm wondering if this could be the problem.

As far as physical memory is concerned unfortunately I don't have any top upgrade and as I said the server worked for years with almost no problems with the same stack of websites. The only thing that might be different a few WordPresses might have been upgraded to newer versions.

Thanks for the help guys. I'll try and investigate further the MySql error in the error log but I have no idea what the localhost access is all about.

[koenn]

you could post those log entries, see if someone here has an idea what they might mean

[self-defence]

Hi.

Well I was able to logon and get some logs form the server between crashes. It really does seem that the load seems to go out of control.

Here they are.

Access log (the last couple of lines):

27.0.0.1 - - [25/Mar/2013:14:48:32 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:00:13 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:18:34 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
220.181.108.122 - - [25/Mar/2013:15:30:15 +0100] "GET / HTTP/1.1" 200 16392 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.13 - - [25/Mar/2013:15:30:42 +0100] "GET / HTTP/1.1" 200 16392 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [25/Mar/2013:15:42:36 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:42:37 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:42:38 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:42:39 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:42:55 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:42:56 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:42:57 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:42:58 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:42:59 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:43:16 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:43:17 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:43:18 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:43:19 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [25/Mar/2013:15:43:21 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
199.30.16.63 - - [25/Mar/2013:16:28:32 +0100] "GET /robots.txt HTTP/1.1" 404 268 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
199.30.16.63 - - [25/Mar/2013:16:28:33 +0100] "GET /images/index_r1_c1.jpg HTTP/1.1" 404 280 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
157.56.93.38 - - [25/Mar/2013:17:28:23 +0100] "GET /robots.txt HTTP/1.1" 404 277 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.56.93.38 - - [25/Mar/2013:17:30:17 +0100] "GET /galerija/?p=311 HTTP/1.1" 404 276 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
202.46.60.63 - - [25/Mar/2013:17:39:01 +0100] "GET / HTTP/1.1" 200 16392 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
119.63.193.130 - - [25/Mar/2013:17:39:08 +0100] "GET / HTTP/1.1" 200 16392 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"

The localhost seems to be turning up the same way allover the place.

And the error log (the lats couple of lines but the log is filled with it):

[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('hits_month','1','2013','03','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('hits_week','8','2013','13','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('hits_day','10','2013','1364166000','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('hits_nobots','21','0','0','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('hits_year_nobots','25','2013','0','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('hits_month_nobots','99','2013','03','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('hits_week_nobots','23','2013','13','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('hits_day_nobots','19','2013','1364166000','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('visits','7','0','0','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('visits_year','2','2013','0','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('visits_month','3','2013','03','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('visits_week','9','2013','13','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('visits_day','11','2013','1364166000','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('visits_nobots','22','0','0','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('visits_year_nobots','26','2013','0','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('visits_month_nobots','98','2013','03','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('visits_week_nobots','24','2013','13','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS (name,type,val1,val2,val3) VALUES ('visits_day_nobots','17','2013','1364166000','1') made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS_RAW' is marked as crashed and should be repaired for query INSERT INTO wp_TABLE_STATISTICS_RAW\r\n\t\t\t\t\t(\r\n\t\t\t\t\t\tremote_addr,\r\n\t\t\t\t\t\thttp_user_agent,\r\n\t\t\t\t\t\thttp_accept_language,\r\n\t\t\t\t\t\tpage,\r\n\t\t\t\t\t\tpagetype,\r\n\t\t\t\t\t\tpageid,\r\n\t\t\t\t\t\tstamp,\r\n\t\t\t\t\t\tbrowser,\r\n\t\t\t\t\t\tbrowserversion,\r\n\t\t\t\t\t\tbrowsertype,\r\n\t\t\t\t\t\tos,\r\n\t\t\t\t\t\treferer,\r\n\t\t\t\t\t\tmethod,\r\n\t\t\t\t\t\tsearchstring,\r\n\t\t\t\t\t\tsearchstringtype,\r\n\t\t\t\t\t\treferertype,\r\n\t\t\t\t\t\tentrypage\r\n\t\t\t\t\t\t)\r\n\t\t\t\t\tVALUES\r\n\t\t\t\t\t (\r\n\t\t\t\t\t\t'-730782814',\r\n\t\t\t\t\t\t'',\r\n\t\t\t\t\t\t'',\r\n\t\t\t\t\t\t'/',\r\n\t\t\t\t\t\t'0',\r\n\t\t\t\t\t\t'1344',\r\n\t\t\t\t\t\t'1364233757',\r\n\t\t\t\t\t\t'100',\r\n\t\t\t\t\t\t'5.0',\r\n\t\t\t\t\t\t'0',\r\n\t\t\t\t\t\t'',\r\n\t\t\t\t\t\t'',\r\n\t\t\t\t\t\t'1',\r\n\t\t\t\t\t\t'',\r\n\t\t\t\t\t\t'0',\r\n\t\t\t\t\t\t'0',\r\n\t\t\t\t\t\t'1'\r\n\t\t\t\t\t\t) made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update
[Mon Mar 25 17:49:18 2013] [error] [client 212.113.35.162] WordPress database error Table './WEBSITE/wp_TABLE_STATISTICS' is marked as crashed and should be repaired for query SELECT count(type)\r\n                    FROM wp_TABLE_STATISTICS\r\n                    WHERE\r\n                        type=6 made by require, require_once, do_action, call_user_func_array, cystats_update_data, statistics->update

Could this WordPress be overloading the server?

Thanks for the help and best regards.