Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: my server is attempting to ssh brute force...

  1. #1
    Join Date
    Dec 2012
    Beans
    1

    Unhappy my server is attempting to ssh brute force...

    soooo.

    I just received notice from a sysadmin that my server was attempting some good ole brute force SSH attacks last night. My question to you is how do I start looking into this to see what was happening?

    I know how to check things out as the receiving end of one of these attacks, but never as the instigating machine. (obviously I was NOT attempting to brute force anyones network, this is malicious and done without my knowledge)

    can you point me to any log files or specefic entries that i should be looking for?

    Thanks,

    Weasle

  2. #2
    Join Date
    Nov 2007
    Location
    Newry, Northern Ireland
    Beans
    1,258

    Re: my server is attempting to ssh brute force...

    First thing is to drop the box off the network if possible, so the attacks stop.

    I guess you'll not like this, but if (and this seems likely) your box has been cracked, a re-install is the only way to be sure it is clean. Time to break out the backups of the data you have on it...
    Can't think of anything profound or witty.
    My Blog: http://gonzothegeek.blogspot.co.uk/

  3. #3
    Join Date
    Mar 2009
    Beans
    1,322

    Re: my server is attempting to ssh brute force...

    +1 on the reinstall.

    Frankly this sort of thing rarely corrupts data files, so if you pull your drive out and put it in another fully functional box as a second drive you can probably get immediate data backups.

    Usually what happens is either a rogue app is installed somewhere (including maybe your user directory) or an existing command is overwritten. So if you get your data to some system that has reliable apps you might be able to save your data.

    That said, you can never be 100% sure you got rid of the bad guy's code.

    Start reading here:
    https://wiki.ubuntu.com/BasicSecurity

    IMO it's a much better starting point than the official docs.

    Then get some really good passwords when you wipe your box clean, and don't let ANY password be trivial. And turn off useless services once you made sure they're useless.
    Help stamp out MBR partition tables. Use GPT instead!

  4. #4
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: my server is attempting to ssh brute force...

    +2 to wiping the box and starting over. If someone was able to gain access to the machine they could be running a script either remotely or via cron but the main thing with any compromise, is you do not necessarily know what all was done to the machine.

    A reinstall is the only way to be sure (unless you really need to nuke it from orbit...)
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  5. #5
    Join Date
    Sep 2011
    Beans
    1,531

    Re: my server is attempting to ssh brute force...

    Just to reiterate the obvious first step:

    Get the server off the network (unplug ethernet, kill wireless).

    You have two options:
    1. reimage it and move on with life, perhaps plan on reading the security stickies in this forum so that it doesn't happen again. (if you don't have backups now, make some with the server off the network)

    2. If you're curious to see what's happening, then you can do some investigation. Use the "Did I Just Get Owned" wiki to look at some logs, knowing that attackers often try to hide their activity by deleting/modifying logs. Look at netstat -antp to see where your server is trying to call. When you're done, reimage & restore the data from your backups.
    Last edited by Ms. Daisy; December 8th, 2012 at 03:30 AM.

  6. #6
    Join Date
    Aug 2009
    Beans
    Hidden!

    Re: my server is attempting to ssh brute force...

    - Given how web stack exploits work these days a compromise does not automagically mean a root compromise,
    - suggesting a box is thoroughly compromised without actually investigating it is rather inefficient IMNSHO,
    - suggesting a restore from backup without checking contents first or re-installing without investigating first may easily expose the same loophole all over again.
    Please think about the consequences of "advice".


    Quote Originally Posted by weasle37 View Post
    can you point me to any log files or specefic entries that i should be looking for?
    The main reason for web stack compromises is running outdated versions of software, third party plug-ins or not running software like it should be. The latter includes issues like running all web sites in the same shared hosting environment, web servers running default configurations, PHP configuration allowing dangerous HTTP methods, lax file access permissions, installers that aren't cleaned up, unrestricted access to management interfaces, leeched FTP credentials, compromised SSH accounts, no hardened PHP, web no application firewall, no testing whatsoever, etc, etc.

    Most of the time you'll find a standardized kit containing a (PHP_based) backdoor, IRC bot and SSH scanner, rarely accompanied by a process hider, cronjob and other "goodies".

    Best suggestion in short would be to run all (and I mean all) your logs through Logwatch as it is the quickest way I know to generate leads. Use the "--detail High --service All --range All --archives --numeric" args.


    *While there's not much honor in "solving" these kinds of compromises, they do happen multiple times a day, I'm offering to help you with a more thorough investigation. You have to gather quite a lot of information, I won't guarantee any success but it'll be educating. If you're up for that please:
    0. Read the CERT Intruder Detection Checklist. While it's old it can still serve as guideline in case you don't know which core items to check,
    1. Replace "/path/to/data.txt" with a suitable location like /dev/shm or /var/tmp and save the following information (attach as plain text or pastebin it):
    Code:
    ( \ps axfwwwe -opid,ppid,gid,uid,cmd 2>&1; \lsof -Pwln 2>&1; \netstat -anTpe 2>&1; \lastlog 2>&1; \last -wai 2>&1; \who -a 2>&1 ) > /path/to/data.txt
    *Note you should mitigate the situation (raise firewall to deny incoming connections, stop services, kill rogue processes) only after you saved this information.
    2. Please post the following nfo:
    - which basic services the machine provides like telnet, Vino, SSH, et, etc and any LAMP ones including web-based management panels, statistics, web log, forum, shopping cart, plugins and other software running on top of your web server,
    - if software was kept up to date or not,
    - which logging, access restrictions are in place and what hardening was performed (if any),
    - if there have been earlier breaches or anomalies,
    - if you've killed any processes or deleted anything,
    - if there are any foreign objects in directories the web server user or unprivileged users can write to like /var/www, home, /tmp, /var/tmp,
    - anything "odd" in user shell histories,
    - scan the directories mentioned above with LMD (http://www.rfxn.com/projects/linux-malware-detect/) or the databases from clamav.securiteinfo.com.
    * Please be verbose when posting and add any nfo you think may be useful.

  7. #7
    Join Date
    Sep 2011
    Beans
    1,531

    Re: my server is attempting to ssh brute force...

    Quote Originally Posted by unspawn View Post
    - Given how web stack exploits work these days a compromise does not automagically mean a root compromise,
    - suggesting a box is thoroughly compromised without actually investigating it is rather inefficient IMNSHO,
    - suggesting a restore from backup without checking contents first or re-installing without investigating first may easily expose the same loophole all over again.
    Please think about the consequences of "advice".
    Regardless of whether it's a root compromise (which no one suggested), something is amok on this server, that's indisputable.

    FWIW, I agree with you- if it were my server I would want to know what happened and how to harden my new installation. Perhaps I'm jaded but most people aren't interested in in-depth investigations, they just want it fixed quick, and that quick method is reinstallation. Users will only learn how to prevent it in the future if they care enough to take the time and learn. I have come to expect that they usually don't. Hopefully I'm wrong.

  8. #8
    Join Date
    Sep 2011
    Beans
    1,531

    Re: my server is attempting to ssh brute force...

    btw logwatch is very cool. I had never seen it before. I highly recommend it for all servers.

  9. #9
    cprofitt's Avatar
    cprofitt is offline νόησις νοήσεως - nóesis noéseos
    Join Date
    Oct 2006
    Location
    平静
    Beans
    1,445
    Distro
    Ubuntu Development Release

    Re: my server is attempting to ssh brute force...

    I agree with both 'camps' here.

    The only way to KNOW for sure that the box is clean is to reimage and restore the data from backup.

    I would also want to know WHAT happened so I could harden my server.

    -----
    If you have the ability isolate the 'compromised' server from the network and build a new one to replace it; if you do not have the extra server you could always take an image of the compromised server and then load that in a VM to investigate. If you are really serious you might want to take a forensic image of the compromised box.

    For forensics you would want to try Sluethkit and Autopsy.

    Best of luck.

  10. #10
    Join Date
    Aug 2006
    Beans
    1,374
    Distro
    Ubuntu 13.04 Raring Ringtail

    Re: my server is attempting to ssh brute force...

    I just received notice from a sysadmin that my server was attempting some good ole brute force SSH attacks last night. My question to you is how do I start looking into this to see what was happening?
    Whats the content of the message? What information is given to you from the sysadmin?
    I also would be interested in the information/anwers coming from the questions unspawn asked you.
    How did/do you a priori secure your server? (pro-active) How do you monitor your server activities? (Apart from the information given to you by the sysadmin?)

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •