Web server crashing every few days

[self-defence]

Hi guys I've got an Ubuntu web server that keeps crashing on me every few days but don't even know where to start looking for the problem.
I've already updated and upgraded it but no luck.

When I took it out of the rack and hooked it up to a monitor and a keyboard it booted. So I stuck it back in and first time it booted ok but then a couple of days later went down and didn't want to boot up anymore. It has happened the same way a couple of times no.

So can anyone give me some advice how to start or what to look for?

Thanks and best regards

[tgalati4]

Look in /var/log and check every file for clues. If you don't find anything, then that usually points to a hardware problem--power supply or dying motherboard. While you have the rack out, reseat RAM, cables, and blow out the dust.

Post any messages that you don't understand, don't bother posting entire log files.

ssh in from another terminal and run top or htop and keep it running. If it freezes, you sometimes get a snapshot of what was running, take a picture with a digital camera, or cut and paste if possible.

[GordThompson]

I've got an Ubuntu web server that keeps crashing on me every few days
[...]
I stuck it back in and first time it booted ok but then a couple of days later went down and didn't want to boot up anymore. It has happened the same way a couple of times no.

So can anyone give me some advice how to start or what to look for?

I had a similar experience with a Ubuntu 8.x virtual server. It ran fine for a couple of years and then it started "winking out". A hardware problem was ruled out fairly quickly because the host machine was not having any problems at all.

It turns out that the client's "web guy" had just given the company's main website a makeover, changing it from (mostly) static web pages to a WordPress monstrosity with 40 or 50 plugins installed. Prior to that change an inconsiderate bot could establish multiple connections and "scarf down" the entire site without overwhelming the server. After the change one of those bots would come along and lock up the server so tight that it would not respond at all; the only thing I could do was reset (i.e., "power cycle") the VM.

So, +1 for starting by looking at the logs. That's how I came to the conclusion that those bots were killing us, and once I installed `mod_limitipconn` and capped the number of simultaneous connections from a given IP the server stopped tanking.

[hydn79]

There's a strong change its related to Apache. Can you run to command the press Shift + M and paste screenshot here?

What type of website(s) are on this server? Wordpress, Drupal, custom code?

[self-defence]

Sorry guys had some stuff and couldn't get to the server.

I'll be able to get it out tomorrow since it doesn't even boot after the last failure.

The websites running are mostly Wordpress and a couple of static ones and some very basic custom work.

You think it might be something to do with Apache?

Thanks for the help I'll check the logs and top. I've already tried to look at the logs but wasn't able to decode much.

[self-defence]

Hi guys.

Thanks for helping me so quickly but I was unable to access the server quicker because of my scheduled.

The server is a Dell PowerEdge 2400 if it makes any difference.

As always the server became unresponsive after a couple of days and this was the error on the screen when I turned on the monitor:


So I switched it off and booted it up again and it booted fine and is working for the time being but there was an error during boot up regarding Apache just before the log on screen:


I'm guessing there's something wrong in the Apache configuration file. Although I have no idea what since I used Webmin to configure everything.

But I don't know if the two errors are connected?

I logged in and left TOP running for when the server fails if something of significance can be caught there.

Thanks for the help and best regards.

[tgalati4]

Linux is unforgiving when it runs out of memory. Did you not see the "Out of memory . . ." message? Linux will start killing processes to survive, but that won't help with a remote server.

Instead of htop, run an ssh session and watch free. Do you have a large enough swap file? Do you have large enough memory footprint in php.ini? How much RAM in the machine? What is the output of free right after boot and starting all services.

[self-defence]

The sever has 512MB of RAM installed. But it had been working fine for the last three years before any problems showed up. I even ran Memtest to see if the memory was defective but it all passed.

It crashed again during the night.

Here is the free right after boot up:

free
             total       used       free     shared    buffers     cached
Mem:        515488     509140       6348          0       2468     324280
-/+ buffers/cache:     182392     333096
Swap:      1020116        896    1019220

And here's the top from when it crashed. It was still on the screen:


I don't think it has anything to do with increased activity on the web server if any there is less activity now that there was two years ago. So it should have been crashing then.

Or should I just download the whole web server and db content and do a fresh install?

Thanks for the help.

[Doug S]

Look at your load average, the 15 minute time constant one is 76!!!! Your swap space is full, and the memory is consumed. Your server appears to be very very severly overloaded. Of course, once it starts swapping a lot, things gets worse, as more and more time is spent just swapping.

[SeijiSensei]

I see so many apache and postgresql instances there that I have to wonder what this machine is doing. If it's a web server, and those are requests for pages that use the database, this machine is way underpowered for the amount of traffic it is handling. Increasing swap will help, but you need a lot more physical memory. I get worried when I have machines with load averages above 3-4. Values like 90 are off the charts. Upgrade to at least 2 GB of physical memory and a swap file of at least equal size.