[all variants] Intrusion Detection

[bodhi.zazen]

Ubuntu Intrusion Detection

Quote :

“Paranoia will get you through times of no enemies better than enemies will get you through times of no paranoia”

~ Pete Granger

Contents

1. Introduction ~ post #1
2. Install Snort ~ post #2
3. Configure snort ~ post #3
4. Install base ~ post #4
5. Using snort / base ~ post #5
6. Install ossec-hids ~ post #6
7. Install ossec-hids web interface ~ post #7
8. Using ossec-hids ~ post #8

Introduction

This how to was written as an extension to Ubuntu Security and is intended as an introduction to intrusion detection, Ubuntu Style.

This post is quite long, and for what I hope is greater readability, I have broken it into separate posts.

Here is a very nice link that reviews IDS :

Security Focus ~ An Introduction to Intrusion Detection Systems

And for the impatient, the readers digest version :

There are two "arms" of intrusion detection: HIDS and NIDS.

HIDS = Host-based Intrusion Detection System.
NIDS = Network-based Intrusion Detection System.

In a nut-shell, HIDS monitors you system files for unauthorized changes. Examples of this type of monitoring methodology might include techniques such as scanning for viruses, tripwire, Tiger, rkhunter, and chkrootkit.

Similarly, NIDS monitors your network traffic for DOS attacks, port scans, or other suspicious network activity. Examples include watching your firewall in Windows for alerts, snort, or Wireshark.

Although there are other options, both for applications and configuration, in this tutorial I will show you how to install ossec-hids and snort:

NIDS = snort
HIDS = ossec

Snort :

Snort will monitor your network traffic by checking packets against "rules". We will configure snort to log "alerts" to a mysql database. We will then use base to display this information in a web browser (Firefox). Although seemingly foreign at first, base is a very nice web based gui front end for snort. Base is basically point and click and contains numerous links to help interpret alerts.

SNORT® is an open source network intrusion prevention and detection system utilizing a rule-driven language, which combines the benefits of signature, protocol and anomaly based inspection methods. With millions of downloads to date, Snort is the most widely deployed intrusion detection and prevention technology worldwide and has become the de facto standard for the industry.

• Note : Snort will not work with wireless interfaces, you need to use airsnort instead.

OSSEC-HIDS :

OSSEC-HIDS will monitor your log files, monitor the integrity of system files, check for root kits, and perform active response. Active response means ossec will blacklist (block connections) from potential crackers "automagically".

OSSEC is an Open Source Host-based Intrusion Detection System. It performs log analysis, integrity checking, Windows registry monitoring, rootkit detection, real-time alerting and active response.

OSSEC will, amongst other things, monitor snort and blacklist offending ip addresses.

Note : There are of course other options for HIDS, NIDS, as well as alternate configuration options for both snort and ossec.


You should be familiar with :

1. Installing from source (don't worry I will walk you through it).

2. Your ip address, both on your private LAN and public IP address.

3. Your netmask

• You can show your netmask with

  sudo ifconfig | grep --color=always -e Mask -e 255

4. If you wish to access base and the OSSEC web interface outside your LAN you will need to know how to configure your router (you do have a router don't you?). In addition be sure to understand the security implications of running LAMP. In addition you may wish to use ,htaccess or ssl.

Reference: community/ApacheMySQLPHP

5. Installing and configuring snort will take some time, give yourself a few hours.

We will be running all commands in this tutorial as root

So either add "sudo" in front of these commands or open a terminal and obtain a root shell:

sudo -i

[bodhi.zazen]

How to install snort + mysql + base

As has been pointed out by Sarmacid you can install snort from the Ubuntu repositories. In the repos snort is on version 2.7 where from source we are on snort 2.8. To use the repos use "sudo apt-get install snort-mysql".

• See post 20 & 21 for (brief) discussion.

You will need to download a set of rules for snort. The downloads page is here :

http://www.snort.org/pub-bin/downloads.cgi


1. prep ~ Install the various tools and dependencies for Snort and OSSEC.

You wee need the Universe repository enabled.

If you need assistance enabling your repositories, see : /community/Repositories/Ubuntu

apt-get -y install build-essential libpcap0.8-dev libmysqlclient15-dev mysql-client-5.0 mysql-server-5.0 bison flex apache2 libapache2-mod-php5 php5-gd php5-mysql libphp-adodb php-pear libc6-dev g++ gcc pcregrep libpcre3-dev

Note: This will install mysql and apache. Please be sure you understand the implications of this.

For reference : /community/Repositories/ApacheMySQLPHP

During the installation of these applications, make note of (write down) your mysql root password.

2. Obtain snort source code ~ be sure to check the snort home page for updated versions of snort.

Although snort is in the repositories you will need to compile snort yourself. This is because the binary in Ubuntu does not have support for snort logging to a mysql database enabled.

This procedure has been tested (and is working) on both 32 bit 64 bit arch.

cd /usr/src
wget http://www.snort.org/dl/current/snort-2.8.3.tar.gz
tar zxvf snort-2.8.3.tar.gz

3. Obtain a set of rules. Snort uses rules to examine packets and report suspicious activity to your logs and mysql.

In order to get a set of rules you have a set of options listed on the snort rules page

The "Community Rules", at the bottom of the page, are available without any further registration. For a more "up to date" set of rules you must either register or subscribe.

(continuing in the /usr/src directory)

wget http://www.snort.org/the_rules_you_wish_to_use
cd snort-2.8.3
tar zxvf ../snortrules*

In addition you may be interested in obtaining a copy of "bleeding" rules from here :
Bleeding Edge Threats

The snort rule sets are here : http://doc.bleedingthreats.net/bin/v...in/AllRulesets

I downloaded the bleeding-all.rules

You may also be interested in : http://www.emergingthreats.net

You can keep your rules up to date with oinkmaster. Oinkmaster is in the reops.

If you use oinkmaster, be sure to READ THE DOCUMENTATION.

cd /usr/src/snort-2.8.3/rules
wget http://www.bleedingthreats.net/rules/bleeding-all.rules

4. Compile snort :

cd /usr/src/snort-2.8.3
./configure -enable-dynamicplugin --with-mysql
make
make install

Snort *should* compile and install without errors.

If, however, you do get errors when compiling snort, see This thread

5. You can remove snort with :

make uninstall

Back to top

[bodhi.zazen]

Configure snort

Configure mysql

Next we need to configure a mysql database for snort to use for alerts.

mysql -u root -p

Enter your mysql password for root (you did write it down didn't you ?)

You will get a mysql prompt "mysql>". I will use this prompt to indicate commands entered in mysql (as opposed to the command line) you do not need to enter the "mysql >".

mysql> create database snort; mysql> grant all privileges on snort.* to 'snort'@'localhost' identified by 'snort_password'; mysql> exit

• Consider changing the name of the database to something other than "snort".
• Consider changing the name of mysql user to something other then "snort" (in 'snort'@'localhost').
• Change the password to something other then "snort_password".

Now, back at the command line, import the snort database scheme

mysql -D snort -u snort -p < /usr/src/snort-2.8.3/schemas/create_mysql

Configure snort

We need to configure snort and write a start script, and add a cron job.

If things are too quiet, and snort goes a few hours without logging an alert to mysql, snort loses the connection with mysql. You then need to restart snort to re-establish a connection with the mysql database.

First lets create a user for snort. Again change the user name if you wish.

adduser snort

Enter a password (it does not matter, we will be locking the account anyways)

chsh snort

Enter a shell of "/bin/true" (without quotes).

Last, lock the account.

passwd snort -l

Next configure snort :

cd /usr/src/snort-2.8.3
mkdir -p /etc/snort/rules /var/log/snort
chown -R root.snort /var/log/snort
chmod -R 770 /var/log/snort
cp etc/* /etc/snort/
cp rules/* /etc/snort/rules

We next need to make a few edits to /etc/snort/snort.conf :

Using any editor, open /etc/snort/snort.conf and make the following changes :

• In nano you can search using ctrl-W
• In vim you can search using /
• Search for "HOME_NET" , "EXTERNAL_NET", then mysql (without quotes).

1. Change "var HOME_NET any" to "var HOME_NET 192.168.0.0/16" (use your netmask here).
2. Change "var EXTERNAL_NET any" to "var EXTERNAL_NET !$HOME_NET". This sets the external variable to everything other then your network.
3. Change "var RULE_PATH ../rules" to "var RULE_PATH /etc/snort/rules". This tells snort where to find the rule set.
4. Search for "mysql" or scroll down the list to the section with "# output database: log, mysql, user= ...", remove the "#" at the front of this line and change the syntax to :

   output database: log, mysql, user=snort password=snort_password dbname=snort host=localhost

Write a script to start snort :

The only "problem" with installing snort from source is that we now need a script to start snort. The other issue is that if there are no alerts, snort will lose it's connection with the mysql database.

To solve this, I wrote a script to start / restart snort.

The script is attached to this post and is called "ubuntu.snort.init.txt"

Copy this file to your computer and copy/move it to /etc/init.d/snort

Now lets look at the code. You need to look at two lines.

1. The first is your interface. The default is eth0. If you wish to use snort on an alternate interface, such as eth1, you will need to edit the line IFACE="eth0" and change "eth0" to "eth1"
   • Note : Snort will not work with wireless interfaces, you need to use airsnort instead.
2. The second option is to whitelist ip addresses. I advise you do this with caution, but you *may* wish to white IP addresses such as your router and your public ip address.
   
   To white list an IP , add it to the line WHITELIST='' (note that is two single quotes, ' ' and not a double quote " ) , one ip at a time, separated by a space, like this :
   
   

   WHITELIST='127.0.0.1 192.168.1.1'

Now that you are done editing the file, set ownership and permissions :

chown root.root /etc/init.d/snort
chmod 500 /etc/init.d/snort

Starting snort on boot

My script has a 20 second sleep built in (sometimes when you start snort it will fail after a 10-15 second delay). To avoid adding a 20 second longer boot time, use the "boot" option.

With this factoid in mind, edit /etc/rc.local and add :

exec /etc/init.d/snort boot

Add this single line above "exit 0" if your have an exit 0 in the file


Restarting snort with a cron job

Did I mention, Snort may lose the connection to the mysql data base if no alerts are received for several hours (which can happen once we eliminate false positives and install OSSEC-HIDS)? In addition if you clear your data in base you may need to re-start snort.

To restart snort with my script :

/etc/init.d/snort restart

The script will use zenity (a gui interface) if you have it installed (zenity is included in a default Ubuntu or Xubuntu desktop installation, but you will need to add it if you are running Kubuntu). On servers, without X, the script will run without zenity (the script runs either with or without X). In addition, if you run the script as a user you will need to be in the admin group and will be prompted for your password (unless you are in the 15 minute grace period for sudo/gksu).

To restart snort every 6 hours, use crontab (as root)

crontab -e

Add a line for snort :

0 0,6,12,18 * * * /etc/init.d/snort restart >/dev/null 2>&1

Congratulations !! Snort is now configured.


Back to top

[bodhi.zazen]

Install base

Base is a web interface for snort and the snort alerts. See the "using base" section for a brief introduction.

cd
wget http://easynews.dl.sourceforge.net/sourceforge/secureideas/base-1.3.9.tar.gz

Note : Later versions of base do not work (with Ubuntu at least).

cd /var/www
tar zvxf ~/base-1.3.9.tar.gz
mv base-1.3.9 base
cd base
cp -R /usr/src/snort-2.8.3/doc/signatures .
cd ..
chown -R www-data.www-data base

Install a few Pear modules:

pear install Image_Color Image_Canvas-alpha Image_Graph-alpha

Configure apache to use php5, use any editor (nano)

nano /etc/apache2/apache2.conf

At the very bottom of the file add :

DirectoryIndex index.html index.cgi index.pl index.php index.xhtml

Save your changes and re-start apache:

/etc/init.d/apache2 restart

Open a web browser and navigate to http://your_ip_address/base

You must accept cookies from base

Click continue on the first page.

Step 1 of 5: Enter the path to ADODB.
This is /usr/share/php/adodb.

• Sometimes when setting up base , after this first step I get a white page, just repeat step 1

Step 2 of 5:
Database type = MySQL, Database name = snort, Database Host = localhost, Database username = snort, Database Password = snort_password

• leave default port blank.

Step 3 of 5: If you want to use authentication (used to log into the web interface) enter a username and password here and check the box.

Step 4 of 5: Click on Create BASE AG.

Step 5 of 5: once step 4 is done at the bottom click on Now continue to step 5 and log in.

Congratulations ! You should now see something that looks like this :

Click to enlarge picture

You can password protect the base directory with .htaccess and/or use ssl.

Back to top

[bodhi.zazen]

Using snort / base

Although seemingly foreign, everything in base is point and click. Click on your alerts for example and they will be listed. Click on various links and you will be brought to web pages(s) where the alert is explained in more detail.

For example, here is a screen shot showing us some alerts.

Click to enlarge picture

Clilck on the blue "19" by "Total Number of Alerts" and base will show you ...

Click to enlarge picture

Alerts :

#0-(72-1) [nessus] [local] [snort] WEB-MISC robots.txt access 2008-09-07 08:38:48 74.6.17.188:34357 192.168.1.3:80 TCP

click on the [nessus], [local], or [snort] takes you to a web page explaining the alert.

click on the ip address (74.6.17.188) to take you to a page where you can select a link to look up the offending ip.



On this second page, click on ARIN (or any other) which will take us to a page where we can see this ip address belongs to Yahoo.com

===============

Another example, from a local port scan :

#18-(72-19) [snort] (http_inspect) NON-RFC DEFINED CHAR 2008-09-11 16:49:57 192.168.1.5:52093 192.168.1.3:80 TCP

Here we only have the option [snort] which takes us to :

http://www.snort.org/pub-bin/sigs.cgi?sid=119:14

This alert was generated by my portscan to show the active response of ossec (see below).


Basic alert management


First, when you first install snort, you will likely get a large number of alerts. Most of these are legitimate traffic (false positives).

YOU WILL NEED TO RESEARCH EACH ALERT AND DETERMINE IF YOU ARE VULNERABLE. IF SO, FIX YOUR VULNERABILITY.

For "false positives, once you have confirmed an alert is indeed either a false positive or legitimate traffic, either modify or comment out the rule (writing snort rules is beyond this tutorial, see How to snort rules).

For the example here, robots.txt :

Note: There are better ways of managing robots.txt, see the snort links and apache documentation, I am using this only as an example of editing snort rules.

grep is our friend here, so find the alert with :

grep robots.txt /etc/snort/rules/*

returns :

/etc/snort/rules/web-misc.rules:# NOTES: this signature looks for someone accessing the file "robots.txt" via /etc/snort/rules/web-misc.rules:# engines) more efficient. robots.txt is often used to inform a web spider /etc/snort/rules/web-misc.rules:# Verify that the robots.txt does not include any sensitive information. /etc/snort/rules/web-misc.rules:alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"WEB-MISC robots.txt access"; flow:to_server,established; uricontent:"/robots.txt"; nocase; metadata:service http; reference:nessus,10302; classtype:web-application-activity; sid:1852; rev:4;)

so now open /etc/snort/rules/web-misc and comment out the line:

sudo nano -w /etc/snort/rules/web-misc.rules

Hit Ctrl-W to search, search for "robots.txt" (without quotes). Keep hitting Crtl-W to go to the next robots.txt.

When you find the appropriate line (the one that starts with a "alert"), add a # to the front of the line.

Re-start snort.

Once you have managed the false positives, watch for repeat offenders. If I see an IP address persistently triggering snort, I black list it in iptables.

If you do not know how to do this, see here : Uncomplicated_Firewall_ufw

Specifically : https://help.ubuntu.com/community/Un...ocking%20Rules

Hint: EDIT /etc/ufw/before.rules


Back to top

[bodhi.zazen]

OSSEC-HIDS

OSSEC-HIDS is much easier to install. Basically it is downloading and then running a script.

Go to the ossec download page and download the most recent version :

http://www.ossec.net/main/downloads/

wget http://www.ossec.net/files/ossec-hids-1.6.tar.gz
tar xzvf ossec-hids-1.6.tar.gz

Now run the installation script :

cd ossec-hids-1.6
./install.sh

You will be asked a series of questions. Basically select your language, use a "local" installation, and enter an e-mail address. Otherwise go with the defaults.

When you get to the question :

- Do you want to add more IPs to the white list? (y/n)? [n]:

Answer y and add additional IP if you wish to white list them.

There is a very nice post on the Ubuntu forums here :

Howto setup OSSEC-HIDS on your ubuntu box

~ Thanks RShadow

The only "problem" is that the post is a little outdated. The information about running the install script is accurate, but you DO NOT need to write an init script. ossec 1.6 will install a script for you into /etc/init.d/ossec

Start / Stop OSSEC with :

sudo /etc/init.d/ossec start|stop

Configure OSSEC

Not much needs to be done. HOWEVER I would caution you that OSSEC has an active response to threats. If OSSEC detects a bad ip address, it will block that ip address using iptables. This means that if your snort rules are giving you false alerts legitimate traffic to your server will be blocked.

This also means you can lose access to your server as well.

Fortunately this is a temporary ban. It is more than sufficient to deter the script kiddies and if your access is blocked, access is restored in a few minutes.

This means , however, you need to monitor snort (base) and fine tune your rules so you are not blocking legitimate traffic.

Again, if there are repeat offending IP addresses, black list them in iptables (See the using snort/base post for how to do this).

Additional configuration of OSSEC is is done by editing /var/ossec/etc/ossec.conf

This configuration file is well commented and you will see a white list section where you may white list additional ip addresses if needed.


Back to top

[bodhi.zazen]

Web access to ossec

Download the web interface from http://www.ossec.net/main/downloads/

cd
wget http://www.ossec.net/files/ui/ossec-wui-0.3.tar.gz
cd /var/www
tar xzvf ~/ossec-wui-0.3.tar.gz
mv ossec-wui* ossec

cd ossec
./setup.sh

During the setup you will be asked for a user name and password. You will use this user name and password to access the web interface.

When the script is finished running, change ownership of the directory and add www-data to the ossec group

cd /var/www
chown -R www-data.www-data ossec
usermod -G ossec -a www-data

Restart apache

/etc/init.d/apache2 restart

Log in at http://your_ip_address/ossec

From the web interface you can see any changes to system files and alerts.


Back to top

[bodhi.zazen]

Using ossec :

Once you log into the web interface you will have a number of tabs.

Main ~ This is where you will see alerts.

Integrity checking ~ Will show you recent changes to system files.




Understanding and modifying rules

Listing of rules (it is incomplete): http://www.ossec.net/wiki/index.php/Rule

Modifying rules : http://www.ossec.net/wiki/index.php/...w:Ignore_Rules

I did find this wiki page on integrating base + ossec, but I have not tried it.

ossec + base : http://www.ossec.net/wiki/index.php/OSSEC_&_BASE


Example of ossec active response :

# Start by pinging the server: root@hardy:~#ping 192.168.0.3 PING 192.168.1.3 (192.168.0.3) 56(84) bytes of data. 64 bytes from 192.168.0.3: icmp_seq=1 ttl=64 time=0.378 ms 64 bytes from 192.168.0.3: icmp_seq=2 ttl=64 time=0.377 ms 64 bytes from 192.168.0.3: icmp_seq=3 ttl=64 time=0.359 ms --- 192.168.0.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 0.359/0.371/0.378/0.018 ms Portscan the server : root@hardy:~#nmap -sS -sV -O -PI -PT 192.168.0.3 Starting Nmap 4.53 ( http://insecure.org ) at 2008-09-11 17:27 MDT <-- Notice how Nmap hangs ? --> # Now ping the server again: root@hardy:~#ping 192.168.0.3 PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data. <-- Notice how your pings are blocked ? --> --- 192.168.0.3 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 2999ms

Back to top

[Rocket2DMn]

The Ubuntu Guru strikes again! Thanks bodhi, this thread is great, I think it's your best so far.

Now for a short little addendum to the main guide, for those interested in low system impact.

Intrusion Detection using a Virtual Machine

If you would like to take this guide for a run without heavily messing with their current system configuration, you can still get the full effect by setting up Ubuntu in a Virtual Machine (VM) and following the guide from there. The concepts here apply to whatever virtualization software you prefer, I just use vbox as an example since that is what I used.

The only work that needs to be done on your host machine (not the VM) is to setup a network bridge so that your VM will have an IP assigned by your network's DHCP server (in a home network, this is usually the central router). You can also opt set a static IP that is recognized by the rest of the network. With this network IP, you can then actually see what is happening on the network, unlike with the default private IP that VMs normally get assigned.
Example:

Normal Private IP: 10.0.2.15
Network IP: 192.168.1.101

(Note: Yes, yes, 192.168.xxx.yyy is also in the private IP range, but this is what is generally seen on a home network, and I therefore refer to as the network IP.)

General Directions for using VirtualBox:

Setup your virtual machine with an Ubuntu installation - there are many guides out there on how to do this (ex: community/VirtualBox). Don't forget to install LAMP and the build-essential metapackage which are needed for the above tutorial.

Now create a network bridge on your host machine by following the directions at community/VirtualBox#Networking. Our guru, bodhi.zazen, also suggested the following link which will also work for vbox - community/KVM#Creating a network bridge on the host. I found it helps to first set your host to DHCP temporarily if you are using a static IP so that you can easily configure the bridge. Then you can setup your static IP after your bridge is successfully created and tested. The VM won't know the difference, other than it will get an IP immediately compatible with your network. Also, don't forget to create the scripts to bring the bridge up and down as described in the VirtualBox wiki link.
Here is my /etc/network/interfaces file, with static IP on the host, for your reference (yours will vary). I have edited out my username, and I am using a Linksys WRT54G router:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

auto eth0
iface eth0 inet manual
address 0.0.0.0

auto br0
iface br0 inet static
   bridge_ports eth0
address 192.168.1.201
netmask 255.255.255.0
gateway 192.168.1.1

auto lo
iface lo inet loopback

auto tap1
iface tap1 inet manual
up ifconfig $iface 0.0.0.0 up
down ifconfig $iface down
tunctl_user username

where username is your username which is used in /etc/vbox/interfaces (see the Network portion of vbox guide linked above).

If you're in to using static IPs, you can now set a static IP on your VM as well. This makes it convenient to access the web service interfaces for BASE and OSSEC from another system (like the host).

Enjoy!

[Vivaldi Gloria]

Thanks mate. This thread is indeed great.