Howto spam filtering in Evolution with bogofilter and spamassassin

[samiaji]

After some research and adaptation I came up with personal spam filter using bogofilter and spamassassin with Evolution.

Disclaimer:
I just compiled and adapted what others have written about filtering spam, I included the reference web site at the endo of this post.
I assumed you use standard installation so every directory will be as standard. Customise to your own situation
Make backup -- I can't strees this enough, proper backup have saved me many times.

Background:
I used Faculty of IT at UTS procmail spam filtering at server side. However since I will leaving soon I better "train" my laptop some spam filtering

Ok here is my spec
Dell Inspiron 5100 laptop wiht P4 2.8 GHz, 512 Mb RAM, 32 MB ATI 7500, 30 GB HDD
Ubuntu Breezy 5.10, Evolution 2.4.1, Spamassassin 3.0.4-2, Bogofilter 0.95-2
I used everything standard in Breezy (either cd or repo)

First install spamassassin and bogofilter if you have not done so with whatever methods you like. This is what I did

sudo apt-get install spamassassin bogofilter

After installation
First step enable Junk mail filter in Evolution
Edit->Preferences->Mail Preferences->Junk->Check incoming mail for junk and remote test

Second create two message filters in Evolution to utilise bogofilter
The first filter is to teach bogofilter to recognise spam email (either from marked automatically or manually by user).

Give the rule any name mine is Bogofilter teach spam
Set Status is Junk ThenPipe to program /usr/bin/bogofilter -s + Stop Processing
It looks like this




The second filter is the actual bogofilter action which (hopefully) remove spam

Give the rule any name mine is Bogofilter check spam
Set Pipe to program /usr/bin/bogofilter -u Then Set Status Junk + Stop Processing
It looks like this



You can stop here if you satisfied with bogofilter alone.

Then you have to train bogofilter using the first message filter (that is why the bogofilter teach is in the first position).

I have a collection of spam (700+) in spam folder (under evolution) and I used this for training both spamassassin and bogofilter.

Every spam email will be marked Junk and moved to Evolution Junk folder. You can marked every spam email you received and then it will be moved to Junk folder. I marked all the email in my Spam folder as junk.
Then I moved to Junk folder and select all message. To train bogofilter you just need to apply filter

Message -> Apply Filters or Ctrl+Y

It took about 5 minutes for bogofilter to learn 700+ spam email

You also can train bogofilter to identify spam and non spam (ham) using this command in CLI

bogofilter -s < /home/user/.evolution/mail/local/spam for spam message

bogofilter -n </home/user/.evolution/mail/local/Inbox for nonspam message and replace Inbox with whatever non spam folder you have in evolution

REMEMBER your non spam folder have to be free from spam. (thx dcstar for your comment).

If you want to add another layer to that (using spamassassin) then here it is

First generate the spamassassin configuration using this website if you like me (a.k.a.lazy)

http://www.yrex.com/spam/spamconfig.php

Then create a third message filter (after the first two for bogofilter above)

Give the rule any name mine is Spamassassin 1
Set Pipe to program spamassassin -e Does not return 0 Then Move to Folder Junk/Spam + Set Status Read + Stop Processing
It looks like this



Again you have to train your spamassassin. I used (again) the spam folder I collected.

To train spamassassin use this command in CLI

sa-learn --spam --mbox /home/user/.evolution/mail/local/spam


The other feature of spamassassin is you could also teach them to recognise non spam (ham) as in bogofilter. I use this command in CLI

sa-learn --ham --mbox /home/user/.evolution/mail/local/Inbox

Replace Inbox with whatever non spam mail folder in Evolution.

After the three filters then you can create any filter to sort your email to any folder you like.
All incoming mail will be filtered by those 3 filters before any other filters.

I still train both my bogofilter and spamassassin, but the number of spam I received and appeared in Inbox have been reduced considerably. For illustration I received about 500-750 emails per day.

Regards
Samiaji
Reference
Bogofilter
http://bogofilter.sourceforge.net/
SpamAssassin
http://spamassassin.apache.org/
Using bogofilter in evolution
http://johnleach.co.uk/words/archives/2005/09/15/180/
Spamassassin in evolution
http://software.newsforge.com/softwa....shtml?tid=130
Spamassassin configuration generator
http://www.yrex.com/spam/spamconfig.php

[dcstar]

And to add in a few more handy hints from the other thread on this useful HOWTO:

(More useful stuff from samiaji):
If you use spamassassin as your second layer spam filter (after the bogofilter) you can utilise dcc and razor network. follow the instruction on this URL
http://www.howtoforge.com/virtual_po...ota_courier_p4

One thing to add to it for all of us not using IPv6 is:

sudo cdcc 'ipv6 off'

To prevent it from trying to use the stuff........

Another hint for all the Evolution users using this solution is to regularly "clean out" your Junk folder.

I am in the habit now of checking it every couple of days to ensure that there aren't any messages that have been mistakenly identified as Spam (and I did find some bulk e-mails from an airline that I actually wanted - so I them marked those as "Not Junk").

Once you are sure that the contents of the Junk folder are junk, delete them and do a "Empty Trash" to purge them from your files.

Setting up my Evolution with Spamassassin (with the extra dcc and Razor rules) has cut ~99% of my spam out with only a few "false positives" (from one airline site only).

Once you have your Evolution set up and working with spam filtering, you may then want to try incoming mail virus scanning with something like ClamAV:

http://ubuntuforums.org/showthread.php?t=84423

[pinguinus]

Set Pipe to program spamassassin -e Does not return 0 Then Move to Folder Junk/Spam + Set Status Read + Stop Processing

It is not necessary to choose a folder where SpamAssassin filters spam in Evolution. Actually just using:
Set Pipe to program spamassassin -e Does not return 0
and:
Stop Processing
(maybe with:
Set status: Junk
before the last Stop processing rule if you want but even that is not necessary)
is enough because SpamAssassin then knows automatically to move spam to Evolution's default Junk folder.

More information about this and other Evolution spam filtering techniques (like finetuning spamfilters) in this Newsforge article: http://software.newsforge.com/softwa....shtml?tid=130

[dcstar]

Originally Posted by pinguinus It is not necessary to choose a folder where SpamAssassin filters spam in Evolution. Actually just using: .......

Just ticking the "Check incoming mail for junk" option in the Breezy version of Evolution will automatically use Spamassassin, so no rule is needed at all!

[btdown]

Just ticking the "Check incoming mail for junk" option in the Breezy version of Evolution will automatically use Spamassassin, so no rule is needed at all!

I dont think that is correct. Ticking the "Check incoming mail for junk" does absolutely nothing without the rules piping it to spamassassin or bogofilter. I trained/deleted for 2 weeks and Evolution never filtered a single spam.

[kaplanmyrth]

Originally Posted by btdown I dont think that is correct. Ticking the "Check incoming mail for junk" does absolutely nothing without the rules piping it to spamassassin or bogofilter. I trained/deleted for 2 weeks and Evolution never filtered a single spam.

I can only say what my own experience has been with respect to this, and a filter is definitely not required in order to get Evolution to learn and filter junk mail.

I have had Evolution installed for months and wondered why the junk mail filtering didn't catch anything. Finally read these forums and installed Spamassassin (and not bogofilter). I also installed razor, dcc and pyzor, and customised the /etc/default/spamassassin file to turn on spamd and use razor, dcc and pyzor. Made sure Evolution had the junk mail filtering check boxes checked, and restarted it.

Training was easy -- since I've been marking mail as Junk all along, I had a Junk folder full of mail. I just moved it all to my Inbox, selected it, and marked it as junk. Now, for the first time, the status bar indicated it was "Learning junk".

And finally, without adding any filters, it is checking incoming mail for junk.

[ubuntubrian]

This is great but to me is useless. A "How To" that actually includes the steps and commands to get filtering to work would be very helpful. I can check filtering in Evolution but getting it to work means I need toknow how to pipe, etc. Can anyone make this sensible to a newbie?
Thanks

[magnetism]

Give the rule any name mine is Bogofilter check spam Set Pipe to program /usr/bin/bogofilter -u Then Set Status Junk + Stop Processing It looks like this

The pictures are missing
What should the return value be?

[ubuntubrian]

I went to a couple of the links supplied in this thread and got the filters installed. I don't get that much spam, 10-20 a day, but am using the training method and anxiously waiting for bogofilter to take over.
Interestingly, before I upgraded to Dapper Evolution had a small icon below the mail list that had SpamAssassin's perception of the likelyhood that a particular message was spam. As I marked spam as junk, spamassassin seemed to be getting the messages more and more correct. Then I upgraded and spamassassin disappeared. I had never done anything other than check the boxes in mail preferences-as it should be.

[ubuntubrian]

I don't know if this thread died yet but.. I have tried Samiaji's method to no avail at all. I checked out a couple of sites re: this issue and one of them uses the same method but says to NOT check the "check incoming mail for junk" boxes in preferences. Could this be the reason my filter set up doesn't filter? I've stored up to 500 junk messages in junk and then applied the filters and deleted, as in the how to.
Any help before I switch to Thunderbird?

Thanks