Page 1 of 2 12 LastLast
Results 1 to 10 of 20

Thread: My Ubuntu server got hacked!

  1. #1
    Join Date
    Sep 2008
    Beans
    6

    Red face My Ubuntu server got hacked!

    I noticed today that my Ubuntu server got hacked. The server is located at university campus and some 200 people have access to it (via ssh). Server was running Ubuntu Hardy Heron (server edition) with all security patches applied (including latest kernel). I need your help because i have no idea how they compromised my box. I was browsing my file system (for fun) and i noticed a suspicious file (Z.tar.gz) lying in /var/tmp folder. I immediately listed the contents of Z.tar.gz and then extracted it to my home folder. There were many folders inside. In one of those folders was a modified OpenSSH server that logged passwords to /dev/.tmp so i listed the contents of /dev/.tmp and there were a lot of encrypted stuff inside (usernames,passwords and hosts i guess). Another folder in Z.tar.gz contained a source tree for private version of boxer /dev/mem rookit. I ran Rkhunter and it detected nothing suspicious (not a big surprise). I can't understand how the attacker elevated their privileges to root level. They must have used a private local root exploit because i'm the only user with admin rights and my password is VERY complex and i don't use that password anywhere else. I haven't found anything suspicious
    from log files and shell histories.

  2. #2
    Join Date
    Dec 2006
    Location
    Chicago
    Beans
    3,839

    Re: My Ubuntu server got hacked!

    Quote Originally Posted by gnucracker View Post
    I noticed today that my Ubuntu server got hacked. The server is located at university campus and some 200 people have access to it (via ssh). Server was running Ubuntu Hardy Heron (server edition) with all security patches applied (including latest kernel). I need your help because i have no idea how they compromised my box. I was browsing my file system (for fun) and i noticed a suspicious file (Z.tar.gz) lying in /var/tmp folder. I immediately listed the contents of Z.tar.gz and then extracted it to my home folder. There were many folders inside. In one of those folders was a modified OpenSSH server that logged passwords to /dev/.tmp so i listed the contents of /dev/.tmp and there were a lot of encrypted stuff inside (usernames,passwords and hosts i guess). Another folder in Z.tar.gz contained a source tree for private version of boxer /dev/mem rookit. I ran Rkhunter and it detected nothing suspicious (not a big surprise). I can't understand how the attacker elevated their privileges to root level. They must have used a private local root exploit because i'm the only user with admin rights and my password is VERY complex and i don't use that password anywhere else. I haven't found anything suspicious
    from log files and shell histories.
    Was the modified sshd installed?
    Code:
    md5sum /usr/sbin/sshd
    Did you enable the root account?
    Code:
    sudo getent shadow root|grep !
    Are you running any other servers?
    Code:
    sudo netstat -tulnp
    Have you had dictionary attacks on your ssh server lately?
    Code:
    grep "authentication failure" /var/log/auth.log|less
    Have you been using fail2ban or denyhosts?

    Giving users you can't trust shell access (even with limited privileges) is a little dangerous. I would use a more restrictive shell and/or chroot them.

  3. #3
    Join Date
    Sep 2008
    Beans
    6

    Re: My Ubuntu server got hacked!

    Was the modified sshd installed?

    1. Yes. (and modified ssh client was installed too) I already said that passwords were logged to /dev/.tmp

    Did you enable the root account?
    2.Root account was disabled. I was using sudo.

    Are you running any other servers?
    3.Only OpenSSH and Apache2 were running.

    Have you had dictionary attacks on your ssh server lately?
    4.Not lately.

    fail2ban was running and configured properly.
    Last edited by gnucracker; September 16th, 2008 at 07:29 PM.

  4. #4
    Join Date
    Dec 2006
    Location
    Chicago
    Beans
    3,839

    Re: My Ubuntu server got hacked!

    Quote Originally Posted by gnucracker View Post
    Was the modified sshd installed?

    1. Yes. (and modified ssh client was installed too) I already said that passwords were logged to /dev/.tmp

    Did you enable the root account?
    2.Root account was disabled. I was using sudo.

    Are you running any other servers?
    3.Only OpenSSH and Apache2 were running.

    Have you had dictionary attacks on your ssh server lately?
    4.Not lately.

    fail2ban was running and configured properly.
    Then I agree that they must have had local access through one of your 200 ssh accounts, then used a local exploit to escalate privileges. Local vulnerabilities are much more common than remote vulnerabilities. Either that, or they had physical access.

    It's hard to believe that they would be able to find a local vulnerability before a security update has been released for it, but wouldn't be smart enough to delete their hack tools to cover their tracks.

    Obviously, I would setup your server again from scratch (if you haven't already) since you don't know what else they did.

    I already said that passwords were logged to /dev/.tmp
    You said there was "encrypted stuff". You did not say where it came from, but apparently assumed it was from sshd. I guess they couldn't have created that file without root, so they probably could have and probably did install it.

  5. #5
    Join Date
    Aug 2006
    Beans
    841

    Re: My Ubuntu server got hacked!

    do you know how the approximate time of hacking? it appears it was done by one of the 200 account owners. otherwise fail2ban would have caught them. (remember there is a bug on the repo's fail2ban binary which will fail to start during boot if you dont create /var/run/fail2ban folder manually).

    one long shot is to scan the 200's users bash history files to find suspicious commands. if the attacker didnt cover his tracks well, he would have left that too.

  6. #6
    Join Date
    Sep 2008
    Beans
    6

    Re: My Ubuntu server got hacked!

    Quote Originally Posted by eldragon View Post
    do you know how the approximate time of hacking? it appears it was done by one of the 200 account owners. otherwise fail2ban would have caught them. (remember there is a bug on the repo's fail2ban binary which will fail to start during boot if you dont create /var/run/fail2ban folder manually).
    Intrusion happened two weeks ago according to timestamp of Z.tar.gz

    Quote Originally Posted by eldragon View Post
    one long shot is to scan the 200's users bash history files to find suspicious commands. if the attacker didnt cover his tracks well, he would have left that too.
    I think that the attacker was wise enough to type unset HISTFILE
    (I have already checked bash history files)
    Last edited by gnucracker; September 16th, 2008 at 08:43 PM.

  7. #7
    Join Date
    Sep 2008
    Beans
    6

    Re: My Ubuntu server got hacked!

    Quote Originally Posted by cdenley View Post
    Then I agree that they must have had local access through one of your 200 ssh accounts, then used a local exploit to escalate privileges. Local vulnerabilities are much more common than remote vulnerabilities. Either that, or they had physical access.

    It's hard to believe that they would be able to find a local vulnerability before a security update has been released for it, but wouldn't be smart enough to delete their hack tools to cover their tracks.

    Obviously, I would setup your server again from scratch (if you haven't already) since you don't know what else they did.
    It's not possible that they had physical access.. server was located in safe (locked) place. I think that the intruders were skilled hackers because i had latest security updates installed and they were still able to elevate their privileges. I guess that they just forgot Z.tar.gz to my hard drive.

    Quote Originally Posted by cdenley View Post
    You said there was "encrypted stuff". You did not say where it came from, but apparently assumed it was from sshd. I guess they couldn't have created that file without root, so they probably could have and probably did install it.
    Z.tar.gz contained full source tree for their ssh trojan and while examining the source i noticed that the trojan was using /dev/.tmp
    for storing passwords etc. I have already decrypted (by using their decrypter) the /dev/.tmp file and it contained usernames and passwords for over 70 users

  8. #8
    Join Date
    Aug 2005
    Location
    Boston, MA
    Beans
    Hidden!
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: My Ubuntu server got hacked!

    Apparently you have a serious problem since you are unable to determine how this crack happened. You need to clone the exisitng drive for further forensic analysis and disconnect this compromised server from the network.

    Leaving this connected to the network is a bad idea, no matter how bad the interruption to the user community is. This server will definitely need to be re-built, but not before you clone the drive to some other media.

    Keep us posted on anything you find.

  9. #9
    Join Date
    Sep 2006
    Beans
    3,713

    Re: My Ubuntu server got hacked!

    I'm going to regurgitate one of my prior posts because I'm a lowly post recycler:

    Bleh!

    I highly recommend reading the Securing Debian Manual. It's up to date and comprehensive, yet easy to follow.

    This article give some good info. For example, how to properly image the drive for forscenic analysis.
    Dead Linux Machines do Tell Tales [link to pdf]
    Corn? When did I eat corn?

  10. #10
    Join Date
    Nov 2005
    Location
    Nashville, TN
    Beans
    437
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: My Ubuntu server got hacked!

    Keep in mind that once you give people shell access on a box it's only a matter of time for a determined individual to root the box. It's a lot easier once you have a shell.

    Remember that not everyone who finds vulnerabilities publishes them so developers can write patches. Get all of the logs and use something like splunk to search for segfaults and such. The application that was exploited is of course going to be of interest to discover and patch.
    -Chayak

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •