"Jun" hack

[marsanyi]

I think I've been hacked.

Noticed today that the net was pokey, a tell-tale. System Status showed pretty constant uploading going on to somewhere, in bursts of several minutes, followed by periods of inactivity.

Last time I saw this sort of thing, I tracked it down to my cloud backup hogging the network. This time, however, turning off all the backup/syncing software didn't clean it up.

Ran nethogs, which told me that I had two processes running that seemed to be doing a lot of up/down: /usr/bin/java, loaded early in the sequence (process 1702) and something called "/root/jun". When the uploads kicked in, nethogs suddenly showed a couple of dozen connections from random ports on my machine to a selection of unknown IP addresses on their ports 80. Did some snooping; the IP addresses I found are in the Philippines (!), one in Taiwan.

Fired up ufw, and set it to deny all outgoing connections. The traffic immediately dropped to 0, but one of the CPUs on my machine jumped to 100% usage. I theorized that it was the firewall intercepting a few gazillion connect requests from something on my machine to the outside world. Maybe this "jun" thing?

I killed jun (with sudo killall). The CPU usage dropped back to normal. I went looking for "jun" and found it at /root/jun, just where nethogs said it was. Gingerly removed it, removed the firewall rule disallowing outbound connections, all is well. For a while. Then, same symptoms. Sure enough, "jun" was back in the process list, and back in the /root directory.

So there's some other rogue process that's reinstating "jun". I don't know what it is, so I'm turning off outbound connections until I get it sussed.

Anybody else seen this? Any suggestions as to finding the rogue process behind "jun"s creation? FYI: Ubuntu 12.04LTS, up-to-date (inc. all the bash updates that have been coming down the pipe the last few days). All help gratefully accepted.

[bashiergui]

Did you capture any packets before you took it offline?

You can poke around if you've got the time and desire. This might help (Pdf link)
http://www.sans.org/score/checklists/ID_Linux.pdf

I can't find anything legitimate relating to "jun" and java.

[John_Ten]

Well, it is probable you have a rootkit on your server, or an ssh backdoor or something. Run chkrootkit, rkhunter maybe they will find something. Inspect the logs, boot from an usb stick(if you can) and inspect the machine from there, if it's an LKM(loadable kernel module) rootkit it's pretty hard to find once it's loaded. Next time don't delete the jun program, try to examine it using "strings" or maybe run it in virtualbox or something if it's a binary to see what happens there if you wanna know what it is, but I think the jun program is the least of your problems, I think the big problem is someone compromised your machine and you need to find him.

[marsanyi]

Thanks for the pointer, I'll go try some of those tools.

I have a copy of "jun" quarantined. Hexdump on the binary didn't show anything I recognized (strings of text, for example).

[marsanyi]

I'll check out chkrootkit, thanks. I'm trying to figure out what's running that's behind all this and surgically exposing it. We'll see how far I get.

[bashiergui]

If you can zip/tar jun and post it somewhere like dropbox, I'd like to analyze it.

look at the pdf I linked you and post any weird stuff you find in your system. If you took the box offline, then there's no danger in inspecting the machine.

[marsanyi]

I can do that, tar.gz'd the binary. The raw binary is 1.22Mb, the compressed is 468K. With whom should I share the link?

[marsanyi]

So ps -aux proves more useful than ps -A. The /usr/bin/java job running low in the list is indeed my cloud backup, CrashPlan. I see an entry for root, job 4794 (fairly late loaded) running /usr/bin/.sshd. I expected one for /usr/sbin/sshd, as I have another Ubuntu machine running that. Looking in /usr/bin for .sshd, I see a file with exactly the same filesize as "jun".

[marsanyi]

sudo lsof -p 4794 (.sshd) shows:

COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
.sshd   4794 root  cwd    DIR    8,1     4096       2 /
.sshd   4794 root  rtd    DIR    8,1     4096       2 /
.sshd   4794 root  txt    REG    8,1  1223123 8655397 /usr/bin/.sshd
.sshd   4794 root    0u   CHR    1,3      0t0    1029 /dev/null
.sshd   4794 root    1u   CHR    1,3      0t0    1029 /dev/null
.sshd   4794 root    2u   CHR    1,3      0t0    1029 /dev/null
.sshd   4794 root    3uW  REG    8,1        4  262150 /tmp/moni.lod

sudo lsof -p 4735 (jun) shows:
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
jun     4735 root  cwd    DIR    8,1     4096      2 /
jun     4735 root  rtd    DIR    8,1     4096      2 /
jun     4735 root  txt    REG    8,1  1223123 143352 /root/jun
jun     4735 root    0u   CHR    1,3      0t0   1029 /dev/null
jun     4735 root    1u   CHR    1,3      0t0   1029 /dev/null
jun     4735 root    2u   CHR    1,3      0t0   1029 /dev/null
jun     4735 root    3uW  REG    8,1        4 262147 /tmp/gates.lod
jun     4735 root    4u  IPv4 410946      0t0    TCP 192.168.2.200:39003->118.123.19.72:25000 (SYN_SENT)

[Habitual]

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
.sshd 4794 root txt REG 8,1 1223123 8655397 /usr/bin/.sshd

sudo lsof -p 4735 (jun) shows:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
jjun 4735 root txt REG 8,1 1223123 143352 /root/jun

Sure looks like you've been rooted.
I''d start by firewalling the Chinese IP of 118.123.19.72

and check the crons on the system for method of re-starting:

for i in /var/spool/cron/*; do echo $; sed 's/^//' $i; echo; done | grep -v "#"

If there is anything suspect, use comments (insert an "#" in front of them) on any suspicious crons and restart crond (however that's done on Ubuntu these days)
Also, if any of those cron entries are suspicious, after commenting them out, look at the files they are using in the cron.

and investigate the directories they are being called from.

Install and run rkhunter and carefully review /var/log/rkhunter.log when it's done.
Clean /tmp

It's not clear to me if you should change your root password with this infection on the system until after it's been cleaned.