8.04.3 apache high load, locks up server

[helfire]

Hi,

My server would lock up about every 2 weeks and I had no idea why. but i've figured it out that every time google indexes my server it takes the load up to 150+ and becomes unresponsive.

I've disabled all services except apache and mysql.

This has all updates. and has actually been dist-upgraded from 6.06

I can kill it consistently just by doing this from my home laptop:
while true
do
curl http://mysite.com > /dev/null
done

(it should handle more load than my dsl connection can throw at it...)

I'm not sure where to go from here, no errors in syslog, apache log, anywhere.

[mrsteveman1]

What's your maxclient setting in apache? Check your apache2.conf file if you don't know.

Are you using prefork or threaded mpm? If you don't know which one, type "apache2ctl -V" in a terminal.

Google shouldn't be able to kill a well tuned apache server :)

[helfire]

i agree, i've run apache on a PIII 450mhz with less ram and had hundreds of sites years ago.

maxclient was at 150, i put it down to 75 for now

$ apache2ctl -V
Server version: Apache/2.2.8 (Ubuntu)
Server built: Aug 18 2009 14:18:10
Server's Module Magic Number: 20051115:11
Server loaded: APR 1.2.11, APR-Util 1.2.12
Compiled using: APR 1.2.11, APR-Util 1.2.12
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT=""
-D SUEXEC_BIN="/usr/lib/apache2/suexec"
-D DEFAULT_PIDLOG="/var/run/apache2.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
-D SERVER_CONFIG_FILE="/etc/apache2/apache2.conf"

[mrsteveman1]

Do you have the ram for all those processes?

You should open top or htop and see if you can catch the load going up. I don't think the htop that ships with 8.04 can show you I/O per process but i bet your mysql process is doing a fair bit of disk access on every page request.

Are you doing any caching at all?

[helfire]

i watched while i did my curl test again (and it locked up again) my physical ram gets eaten up right away (but i have 2gb of swap free) my physical ram is 256mb (cheap dedicated server)

Is apache forking for every connection? could i set maxclients to like 10 or so and still get decent performance? (few small sites, nothing hits it much)

No caching, and no htop doesnt show io in 8.04

[helfire]

This started about the time i dist-upgraded from 7.10. When I did that I had no swap partition (because it changed to uuid's) but i re-added the swap.

I wonder if there's something messed up with the swap, as it seams to happen right when physical memory runs out.

edit:
swapon -s
Filename Type Size Used Priority
/dev/sda2 partition 1951888 82540 -1

Looks like it's on...

[mrsteveman1]

Yea prefork apache uses one process for each connection, and typically keeps a pool of processes alive so it doesn't get caught having to fork to handle a connection on the spot.

I didn't realize you were using swap, that's the problem :) Even though the better VPS providers use RAID10 or faster arrays, swapping on a server like that is so slow, that as soon as you start eating into it, your server will die because usually all the memory is 'in use' unlike a desktop. On the servers I manage, which tend to be similar VPS (Though usually with 512mb ram), i turn off swap completely. I'd rather let the kernel kill processes than have it swap active memory, and i'd rather it never even come to that.

In the short term, measure how much memory your apache pool is using and calculate how many processes you can handle, then set maxclients to that or a bit lower. Then when you get a chance turn off swap and reclaim the space for root or something. On a desktop it might be ok to have the kernel swapping out "unused" data, but on a server like this almost all the data is being used, and a page fault that requires swapping at all is going to lag client connections for 10s of seconds at a time before killing the whole server.

If you see performance problems even with maxclients set to 10-20 processes, you may need to adjust your keepalives and do caching.

[helfire]

ok, i put maxclients to 20 and thats just under my avalible memory, and no crashing :)

is mmprefork the right choice for a server with limited ram? It's been a long time since i've had to worry about apache :)

[mrsteveman1]

Threaded (worker) is more efficient, but to do that you have to use threadsafe components and php is generally considered not threadsafe.

What web applications are you running?

[helfire]

Mostly wordpress sites

[mrsteveman1]

Your best bet is to make sure you're using as few apache modules as possible (They get loaded into each process), install w3-total cache in wordpress and make sure you turn on gzipping and the options that slim down css and javascript files (in w3 total cache), then turn Apache keepalives down fairly low (3 seconds).

If you do that and cut maxclients down to only 10-12 you should be able to serve quite a bit of traffic per day without problems.

[helfire]

Thanks for the tips, server seems stable enough for now.

So i'm thinking of moving to a xen vps (linode or something), any tips on setting up a fresh 9.10-server with low physical memory?

When i've run without swap my server just locks and needs a hard reboot.

[mrsteveman1]

Linode is good i've never had a problem with them.

Just make sure you keep your application software processes in check and you should be fine. Don't let apache spawn more processes than you have memory for, and you might also want to tune mysql a bit, there are instructions for getting it to run efficiently on low memory on suns website somewhere but i can't find it at the moment.

Keep htop around so you can quickly see how things are running, and remove anything that doesn't need to be running from the runlevels (with update-rc.d).

[helfire]

I think i have apache down to a reasonable setting, but i have 2 areas that look similar, and kinda wondering if i need both or why i have both, and if the settings look ok.

I cant serve very many simultaneous clients though:

# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers 1
MinSpareServers 1
MaxSpareServers 5
MaxClients 10
MaxRequestsPerChild 5000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
StartServers 1
MaxClients 10
MinSpareThreads 10
MaxSpareThreads 25
ThreadsPerChild 25
MaxRequestsPerChild 5000
</IfModule>

Trying to tune mysql's memory now too.

[mrsteveman1]

Those sections aren't redundant, depending on which mpm you are using (prefork or worker) apache only pays attention to one or the other of those sections, but both remain in the config.

They look ok to me though :)

[Vegan]

I would suggest 512MB for a busy server or more. My P4-2GHz can manage with all the search engines gnawing away at once. So could the old P3-667Mhz board I recently retired.