Being hacked? Lost 3 days of information and no log files

[seacyd]

I encounter a strange situation: Log information on a computer is missing, sent emails are missing (3 days) and in one folder information is even back to an older state (possibly 2 month or more).

Has the computer being compromised overnight?

Chronology of events:
Wednesday -1 week - everything worked fine
Sunday - everything worked fine
Monday - everything worked fine
Tuesday - everything worked fine

Today it was discovered that the database is at an old state (about 2 month old), sent emails are lost of last 3 days and no log files can be found for last 3 days (Sunday to Tuesday).


Kindly let me know your ideas and the best way to address this situation.


--
seacyd

[cdenley]

Maybe your hard drive failed, but worked fine when you rebooted. I've seen systems run for days with a failed hard drive. Everything seems to work fine as far as users can tell, but written data is only cached in memory and never makes it to the hard drive. When the write cache fills all your memory, then you notice the problems. I would expect I/O errors to appear on the console, though.

[seacyd]

Ok, thanks for sharing your idea about this issue.

What I did not tell you was that the computer was completely shutdown and restarted every new day. So I would guess the cache should have been written back to disk every day on shutdown.

What is strange is that I see in the directory /var/log the current log files as well as archieved log files. But there is the gap between 16th and 23rd (even though from 20th to 22nd there was activity)

(...)
-rw-r--r-- 1 root       root  78314 2009-09-16 11:48 Xorg.0.log.old
-rw-r--r-- 1 root       root 178235 2009-09-23 07:09 udev
(...)

There is only one user accessing the system regularly (family business). What I do not understand is, if somebody compromised the system, would this person not just modify the log files instead of deleting the log files completely?

If the system is compromised, are there possibilities to analyse it without having an intrusion detection system installed?

[cdenley]

They could have simply deleted the log files since it is easier than editing them. It would prevent you from determining where they are connecting from. I'm not sure why an attacker would delete logs such as Xorg.0.log within a certain date range, e-mail, or recent database records, though. If they were trying to be destructive, I would expect something like "rm -rf /", assuming they got root, or at least delete all e-mail and database records.

I think all you can do is inspect any logs that are still there, or search for rootkits or anything that may have been left behind. If you suspect your system is compromised, you can never be 100% sure it is safe to use until you inspect and backup important files then reinstall ubuntu.

What kind of servers do you use? Have you set a root password?

sudo netstat -tlnp

[seacyd]

Only CUPS, privoxy and tor are running:

Aktive Internetverbindungen (Nur Server)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8118          0.0.0.0:*               LISTEN      5246/privoxy    
tcp        0      0 0.0.0.0:631             0.0.0.0:*               LISTEN      2891/cupsd      
tcp        0      0 127.0.0.1:9050          0.0.0.0:*               LISTEN      2560/tor        
tcp6       0      0 :::631                  :::*                    LISTEN      2891/cupsd

Additional information:

Thanks for the rootkit tip. I installed rkhunter but got only few warnings, which do not appear to be dangerous to me?:

   /usr/sbin/unhide                                         [ Warning ]
(...)
    /usr/sbin/unhide-linux26                                 [ Warning ]

[Press <ENTER> to continue]
 Performing filesystem checks
    Checking /dev for suspicious file types                  [ Warning ]

and in the log file this information is provided:
[17:12:07] Warning: Suspicious file types found in /dev:
[17:12:07]          /dev/shm/pulse-shm-190707849: data

[cdenley]

Is there a reason cups is listening for remote connections? Is your server a print server? Is it firewalled?

[seacyd]

No - actually I thought it was listening to the local 192.168 network only. I need to change this.

Is there any known exploit through CUPSD?

[cdenley]

Yes, depending on the version you're using, but AppArmor should prevent an attacker from deleting stuff.
http://www.ubuntu.com/usn/usn-760-1
Sevices can only bind to interfaces, not subnets. Generally, it either listens on the local loopback interface, or any interface.

[seacyd]

Thanks - I added the Listen lines (127.0.0.1:631 and not disclosed for 192.168.???.???) and Allow 192.168.???.0/24 within the various brackets at cupsd.conf)

I executed the following apparmor command:
--
APPARMOR
# sudo apparmor_status
apparmor module is loaded.
8 profiles are loaded.
8 profiles are in enforce mode.
/usr/lib/connman/scripts/dhclient-script
/usr/share/gdm/guest-session/Xsession
/usr/sbin/tcpdump
/usr/lib/cups/backend/cups-pdf
/sbin/dhclient3
/usr/sbin/cupsd
/sbin/dhclient-script
/usr/lib/NetworkManager/nm-dhcp-client.action
0 profiles are in complain mode.
1 processes have profiles defined.
1 processes are in enforce mode :
/usr/sbin/cupsd (30466)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
---

I further checked with "tiger"
wajig install tiger

sudo tiger

Out of the lengthy report under /var/log/tiger I copy here some lines what I consider to be the most interesting:


# Checking listening processes
--WARN-- [lin003w] The process `avahi-daemon' is listening on socket 36925
(UDP on every interface) is run by avahi.
--WARN-- [lin003w] The process `avahi-daemon' is listening on socket 5353 (UDP
on every interface) is run by avahi.
--WARN-- [lin002i] The process `cupsd' is listening on socket 631 (UDP) on
every interface.
--WARN-- [lin003w] The process `ntpd' is listening on socket 123 (UDP on every
interface) is run by ntp.
--WARN-- [lin003w] The process `ntpd' is listening on socket 123 (UDP on
192.168.178.6 interface) is run by ntp.

# Looking for unusual device files...
--ALERT-- [fsys006a] Unexpected device files found:
crw------- 1 root root 5, 1 Jul 28 07:00 /lib/udev/devices/console
crw-r----- 1 root kmem 1, 2 Jul 28 07:00 /lib/udev/devices/kmem
brw------- 1 root root 7, 0 Jul 28 07:00 /lib/udev/devices/loop0
crw------- 1 root root 10, 200 Jul 28 07:00 /lib/udev/devices/net/tun
crw------- 1 root root 1, 3 Jul 28 07:00 /lib/udev/devices/null
crw------- 1 root root 108, 0 Jul 28 07:00 /lib/udev/devices/ppp
lrwxrwxrwx 1 root root 15 Jul 28 07:00 /lib/udev/devices/stderr -> /proc/self/fd/2


--
I further issued the command chkrootkit:

sudo chkrootkit

Nothing found.

[openfly]

Nothing is DIRECTLY indicative of an exploit.

You might want to checksum your kernel and core binaries and compare them to a stock build. Do this with a livecd to be certain.