I would recommend reading this first (about intrusion detection):

http://ubuntuforums.org/showthread.php?p=5787017&mode=threaded&highlight=iptables#post5787017

and this ...
...