- Given how web stack exploits work these days a compromise does not automagically mean a root compromise,
- suggesting a box is thoroughly compromised without actually investigating it is rather inefficient IMNSHO,
- suggesting a restore from backup without checking contents first or re-installing without investigating first may easily expose the same loophole all over again.
Please think about the consequences of "advice".
The main reason for web stack compromises is running outdated versions of software, third party plug-ins or not running software like it should be. The latter includes issues like running all web sites in the same shared hosting environment, web servers running default configurations, PHP configuration allowing dangerous HTTP methods, lax file access permissions, installers that aren't cleaned up, unrestricted access to management interfaces, leeched FTP credentials, compromised SSH accounts, no hardened PHP, web no application firewall, no testing whatsoever, etc, etc.
Originally Posted by weasle37
Most of the time you'll find a standardized kit containing a (PHP_based) backdoor, IRC bot and SSH scanner, rarely accompanied by a process hider, cronjob and other "goodies".
Best suggestion in short would be to run all (and I mean all) your logs through Logwatch as it is the quickest way I know to generate leads. Use the "--detail High --service All --range All --archives --numeric" args.
*While there's not much honor in "solving" these kinds of compromises, they do happen multiple times a day, I'm offering to help you with a more thorough investigation. You have to gather quite a lot of information, I won't guarantee any success but it'll be educating. If you're up for that please:
0. Read the CERT Intruder Detection Checklist. While it's old it can still serve as guideline in case you don't know which core items to check,
1. Replace "/path/to/data.txt" with a suitable location like /dev/shm or /var/tmp and save the following information (attach as plain text or pastebin it):
*Note you should mitigate the situation (raise firewall to deny incoming connections, stop services, kill rogue processes) only after you saved this information.
( \ps axfwwwe -opid,ppid,gid,uid,cmd 2>&1; \lsof -Pwln 2>&1; \netstat -anTpe 2>&1; \lastlog 2>&1; \last -wai 2>&1; \who -a 2>&1 ) > /path/to/data.txt
2. Please post the following nfo:
- which basic services the machine provides like telnet, Vino, SSH, et, etc and any LAMP ones including web-based management panels, statistics, web log, forum, shopping cart, plugins and other software running on top of your web server,
- if software was kept up to date or not,
- which logging, access restrictions are in place and what hardening was performed (if any),
- if there have been earlier breaches or anomalies,
- if you've killed any processes or deleted anything,
- if there are any foreign objects in directories the web server user or unprivileged users can write to like /var/www, home, /tmp, /var/tmp,
- anything "odd" in user shell histories,
- scan the directories mentioned above with LMD (http://www.rfxn.com/projects/linux-malware-detect/) or the databases from clamav.securiteinfo.com.
* Please be verbose when posting and add any nfo you think may be useful.