Rather than waiting for the server to crash, monitor the swap file. When it passes some predetermined value (say 500 MB) then start pruning processes and examining log files.
Write a script that examines your swap file every 10 minutes. If it exceeds 500 MB then dump some vmstat data and send an email or wall message to take action.
cat /proc/swaps
man gawk
cat /proc/swaps | gawk '/sda3/ {print $4}'
If above expression is greater than 500 MB then do something. Your swap device will be different than mine (sda3).
echo "My server is about to crash!!" | wall
Bookmarks