Bug 69418 - filters don't work anymore as in 3.1.4
Summary: filters don't work anymore as in 3.1.4
Status: RESOLVED FIXED
Alias: None
Product: kmail
Classification: Applications
Component: general (show other bugs)
Version: 1.5.94
Platform: unspecified Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2003-12-01 09:33 UTC by Martin Koller
Modified: 2007-09-14 12:17 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In:


Attachments
kmail.log.gz (2.68 KB, application/x-gzip)
2003-12-31 15:14 UTC, Martin Koller
Details
kmail.filters.gz (1.04 KB, application/x-gzip)
2003-12-31 15:14 UTC, Martin Koller
Details
kmail.log.goe0.gz (2.25 KB, application/x-gzip)
2004-01-01 13:47 UTC, Martin Koller
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Koller 2003-12-01 09:33:07 UTC
Version:           1.5.94 (using KDE 3.1.94 (3.2 Beta 2), compiled sources)
Compiler:          gcc version 2.96 20000731 (Red Hat Linux 7.2 2.96-112.7.2)
OS:          Linux (i686) release 2.4.22

I was using KDE-3.1.4 before with a spam filter set up as:
Name: SpamOracle_marked?
Match any of the following:
X-Spam doesn't contain yes
X-Spam doesn't contain no
Action:
remove header X-Spam
pipe through spamoracle mark
to incoming + manual filtering

Name: SpamOracle_isSpam
match all of the following
X-Spam contains yes
Action:
remove header X-Spam
file into folder Junk-mail


This worked, but I have now upgraded to 3.2, and the mails are no more filtered.
Can it be, that if an incoming mail does not contain the X-Spam header at all, the first rules are not matched any more ?
Comment 1 Martin Koller 2003-12-01 09:34:37 UTC
Another hint: If I mark the junkmails in the inbox and press CTRL-J (apply filters), that still works and then moves the mails to my junk-mail folder.
Comment 2 Martin Koller 2003-12-31 14:06:00 UTC
This is the same bug as #67078. I have updated from CVS now and I can still reproduce this. Do you want me to test something ?
Comment 3 Andreas Gungl 2003-12-31 14:40:27 UTC
This works here like a charm. What I don't understand is the following:

Match any of the following: 
 X-Spam doesn't contain yes 
 X-Spam doesn't contain no

So the filter should get triggered in any case, the test evaluates to true no matter what X-Spam contains. You could use as well size >= 0.

For me it makes no difference whether a X-Spam header is in the message or not. The filter is triggered. You write, you've updated from cvs. Can you start KMail in a console from command line and watch the output. You should be able to see lines like this:

kmail: ####### KMFilter::process: going to apply action file into folder "drafts"

If you see differences between filtering on incoming messages and manual filtering then I were interested in the output as well as in the filter definitions in your /home/<user>/.kde/share/config/kmailrc file.
Comment 4 Martin Koller 2003-12-31 15:14:20 UTC
Subject: Re:  filters don't work anymore as in 3.1.4

On Wednesday 31 December 2003 14:40, Andreas Gungl wrote:
> ------- Additional Comments From a.gungl@gmx.de  2003-12-31 14:40 -------
> This works here like a charm. What I don't understand is the following:
>
> Match any of the following:
>  X-Spam doesn't contain yes
>  X-Spam doesn't contain no
>
> So the filter should get triggered in any case, the test evaluates to true
> no matter what X-Spam contains. You could use as well size >= 0.

I think I found this on some spamorcale/kmail HOWTO page when I set it up.
I already modified the filter to use size>0

>
> For me it makes no difference whether a X-Spam header is in the message or
> not. The filter is triggered. You write, you've updated from cvs. Can you
> start KMail in a console from command line and watch the output. You should
> be able to see lines like this:
>
> kmail: ####### KMFilter::process: going to apply action file into folder
> "drafts"

I did that and attached the log file. You only find the  ... action pipe 
through "spamoracle mark" ... twice - this was when I did CTRL-J on two 
different mails _after_ they have bee received _without_ being sent through 
the filter.

>
> If you see differences between filtering on incoming messages and manual
> filtering then I were interested in the output as well as in the filter
> definitions in your /home/<user>/.kde/share/config/kmailrc file.

also attached.



Created an attachment (id=3899)
kmail.log.gz

Created an attachment (id=3900)
kmail.filters.gz
Comment 5 Andreas Gungl 2003-12-31 22:15:55 UTC
Martin, in your (second, extended) log I've found:

kmail: ####### KMFilterMgr::process: going to be apply filter rule Match any of the following:
"<size>" <greater> "0"
kmail: 

Can you countercheck with:
- Match ALL of the following
- "<size>" <greaterOrEqual> "0" (I mean size >= 0, don't know the exact phrase in the moment.)

If that doesn't shed some light on the problem, then it's getting really difficult. I think, we'll have to add more debug output. At least the rule is used and somehow it's not considered to match the message.
I hope, we can solve the issue before 3.2 is tagged.

Comment 6 Martin Koller 2004-01-01 13:47:28 UTC
Subject: Re:  filters don't work anymore as in 3.1.4

On Wednesday 31 December 2003 22:15, Andreas Gungl wrote:
> ------- Additional Comments From a.gungl@gmx.de  2003-12-31 22:15 -------
> Martin, in your (second, extended) log I've found:
>
> kmail: ####### KMFilterMgr::process: going to be apply filter rule Match
> any of the following: "<size>" <greater> "0"
> kmail:
>
> Can you countercheck with:
> - Match ALL of the following
> - "<size>" <greaterOrEqual> "0" (I mean size >= 0, don't know the exact
> phrase in the moment.)
>
> If that doesn't shed some light on the problem, then it's getting really
> difficult. I think, we'll have to add more debug output. At least the rule
> is used and somehow it's not considered to match the message. I hope, we
> can solve the issue before 3.2 is tagged.

First of all - A happy new year!

I changed now the rule to >=0, I have still the same problem but some changes 
also:

The interesting sections are:

kmail: ####### KMFilterMgr::process: going to be apply filter rule Match all 
of the following:
"<size>" <greater-or-equal> "0"
kmail: 
kmail: ####### KMFilterMgr::process: filter rule did match
kmail: ####### KMFilter::process: going to apply action remove header "X-Spam"
kmail: ####### KMFilter::process: going to apply action pipe through 
"spamoracle mark"
kmail: ####### KMFilterMgr::process: going to be apply filter rule Match all 
of the following:
"X-Spam" <contains> "yes"
"Subject" <contains-not> "[Bug"
kmail: 
kmail: ####### KMFilterMgr::process: going to be apply filter rule Match all 
of the following:
"

Interesting here is, that on incoming mail the filter gets now applied, but it 
seems that kmail did not recognize the added X-Spam header afterwards (in the 
rule after this one).

But more important, it seems that what comes out of spamoracle with that 
filter is NOT the mail which is then stored in kmail, because if I look at 
the mail-source (pressing "V"), I don't find the X-Spam header!
So maybe that is the source of the whole problem.

Was there a change in the handling of receiving mails from 3.1.4 to 3.2, so 
that the received mail is somehow only temporary and only this one is 
modified by spamorcale ?

Now look what happend when I pressed CTRL-J:

...
kmail: 
kmail: ####### KMFilterMgr::process: going to be apply filter rule Match all 
of the following:
"<size>" <greater-or-equal> "0"
kmail: 
kmail: ####### KMFilterMgr::process: filter rule did match
kmail: ####### KMFilter::process: going to apply action remove header "X-Spam"
kmail: ####### KMFilter::process: going to apply action pipe through 
"spamoracle mark"
kmail: ####### KMFilterMgr::process: going to be apply filter rule Match all 
of the following:
"X-Spam" <contains> "yes"
"Subject" <contains-not> "[Bug"
kmail: 
kmail: ####### KMFilterMgr::process: filter rule did match
kmail: ####### KMFilter::process: going to apply action remove header "X-Spam"
kmail: ####### KMFilter::process: going to apply action file into folder 
"JUNK-mail"
kmail: ####### KMFilter::process: going to apply action mark as "U"
k

So it seems by applying the filters manually, the mail is then really modified 
and therefore also used by the next filter.

I attach the full log.


Created an attachment (id=3912)
kmail.log.goe0.gz
Comment 7 Andreas Gungl 2004-01-01 14:28:10 UTC
Subject: Re:  filters don't work anymore as in 3.1.4

This looks like a bug which has been considered to be fixed.

############### (commited 22.12.2003)
CVS commit by kloecker: 

Fix bug 69860 (output of pipe through sometimes not used). This also fixes
ad hoc filters. Based on patch by Andreas Gungl. Reviewed by Don.
###############

I need some more information to debug that problem. First I need to know 
what change triggered the rule to use spamoracle. Was it "size >= 0" or 
"Match all ...". Second I would like to know how the message looks like 
after spamoracle was applied. Can you create a small shell script 
(one-liner: spamoracle mark | tee ~/filter result ) and use that in the 
Pipe Through action.

On Donnerstag, 1. Januar 2004 13:47, Martin Koller wrote:
> On Wednesday 31 December 2003 22:15, Andreas Gungl wrote:
> > ------- Additional Comments From a.gungl@gmx.de  2003-12-31 22:15
> > ------- Martin, in your (second, extended) log I've found:
> >
> > kmail: ####### KMFilterMgr::process: going to be apply filter rule
> > Match any of the following: "<size>" <greater> "0"
> > kmail:
> >
> > Can you countercheck with:
> > - Match ALL of the following
> > - "<size>" <greaterOrEqual> "0" (I mean size >= 0, don't know the exact
> > phrase in the moment.)
> >
> > If that doesn't shed some light on the problem, then it's getting
> > really difficult. I think, we'll have to add more debug output. At
> > least the rule is used and somehow it's not considered to match the
> > message. I hope, we can solve the issue before 3.2 is tagged.
>
> First of all - A happy new year!
>
> I changed now the rule to >=0, I have still the same problem but some
> changes also:
>
> The interesting sections are:
>
> kmail: ####### KMFilterMgr::process: going to be apply filter rule Match
> all of the following:
> "<size>" <greater-or-equal> "0"
> kmail:
> kmail: ####### KMFilterMgr::process: filter rule did match
> kmail: ####### KMFilter::process: going to apply action remove header
> "X-Spam" kmail: ####### KMFilter::process: going to apply action pipe
> through "spamoracle mark"
> kmail: ####### KMFilterMgr::process: going to be apply filter rule Match
> all of the following:
> "X-Spam" <contains> "yes"
> "Subject" <contains-not> "[Bug"
> kmail:
> kmail: ####### KMFilterMgr::process: going to be apply filter rule Match
> all of the following:
> "
>
> Interesting here is, that on incoming mail the filter gets now applied,
> but it seems that kmail did not recognize the added X-Spam header
> afterwards (in the rule after this one).
>
> But more important, it seems that what comes out of spamoracle with that
> filter is NOT the mail which is then stored in kmail, because if I look
> at the mail-source (pressing "V"), I don't find the X-Spam header!
> So maybe that is the source of the whole problem.
>
> Was there a change in the handling of receiving mails from 3.1.4 to 3.2,
> so that the received mail is somehow only temporary and only this one is
> modified by spamorcale ?
>
> Now look what happend when I pressed CTRL-J:
>
> ...
> kmail:
> kmail: ####### KMFilterMgr::process: going to be apply filter rule Match
> all of the following:
> "<size>" <greater-or-equal> "0"
> kmail:
> kmail: ####### KMFilterMgr::process: filter rule did match
> kmail: ####### KMFilter::process: going to apply action remove header
> "X-Spam" kmail: ####### KMFilter::process: going to apply action pipe
> through "spamoracle mark"
> kmail: ####### KMFilterMgr::process: going to be apply filter rule Match
> all of the following:
> "X-Spam" <contains> "yes"
> "Subject" <contains-not> "[Bug"
> kmail:
> kmail: ####### KMFilterMgr::process: filter rule did match
> kmail: ####### KMFilter::process: going to apply action remove header
> "X-Spam" kmail: ####### KMFilter::process: going to apply action file
> into folder "JUNK-mail"
> kmail: ####### KMFilter::process: going to apply action mark as "U"
> k
>
> So it seems by applying the filters manually, the mail is then really
> modified and therefore also used by the next filter.
>
> I attach the full log.
>
>
> Created an attachment (id=3912)
>  --> (http://bugs.kde.org/attachment.cgi?id=3912&action=view)
> kmail.log.goe0.gz

Comment 8 Martin Koller 2004-01-01 15:03:47 UTC
Subject: Re:  filters don't work anymore as in 3.1.4

On Thursday 01 January 2004 14:28, Andreas Gungl wrote:
> This looks like a bug which has been considered to be fixed.
>
> ############### (commited 22.12.2003)
> CVS commit by kloecker:
>
> Fix bug 69860 (output of pipe through sometimes not used). This also fixes
> ad hoc filters. Based on patch by Andreas Gungl. Reviewed by Don.
> ###############
>
> I need some more information to debug that problem. First I need to know
> what change triggered the rule to use spamoracle. Was it "size >= 0" or
> "Match all ...". Second I would like to know how the message looks like
> after spamoracle was applied. Can you create a small shell script
> (one-liner: spamoracle mark | tee ~/filter result ) and use that in the
> Pipe Through action.

OK, I think we've got it:
The filter on incoming message does pass spamoracle without modification - 
that is due to the fact, that spamoracle does not recognize the input as a 
mail I think.
I checked why, and have seen that the incoming message does not have the very 
first line

- From dermalptch1zx@rogers.com Thu Jan  1 14:45:45 2004

The input to spamoracle starts with the

Return-Path: <dermalptch1zx@rogers.com>

Header.

When I save the received mail as plain text, I see this first "From" line, and 
when I feed this by hand through spamoracle, it then does get the X-Spam 
header!!

The second issue regarding the ">=0" match:
I changed it back to match "any", and it still works (matches).
I double checked and changed also back to ">0", which does _not_ match.

So this seems to be a second bug.

- --
Best regards/Sch
Comment 9 Andreas Gungl 2004-01-01 20:04:48 UTC
Martin, can you check that KMail from KDE 3.1.x sent the complete mail to spamoracle (including the From line)?
I can confirm that the From line in the current version is not sent to the pipe.

Reading your last comment I'm not sure whether you piped the messages manually (outside KMail) through spamoracle. If so, that could have helped finding the problems.

I'll try to fix the size issue until the weekend, if it's not possible I'll file a new report.
Comment 10 Andreas Gungl 2004-01-01 21:46:13 UTC
I've filed a new report for the size problem (#71614).

Regarding the "From line" I've found that the line is only a delimiter in mbox files. It doesn't really belong to the message which becomes obvious when you look at messages stored in maildir format. So this is IMO a problem of spamoracle. You may consider to write a wrapper script which provides a dummy "From line" to make spamoracle work. Or you may consider to switch to another tool like bogofilter or spamassassin.

Is it okay for you when I close the report with respect to the new one for the size problem? Anyway, thanks for your patience and your help on tracking it down.
Comment 11 Martin Koller 2004-01-02 11:49:03 UTC
Subject: Re:  filters don't work anymore as in 3.1.4

On Thursday 01 January 2004 21:46, Andreas Gungl wrote:
>
> Regarding the "From line" I've found that the line is only a delimiter in
> mbox files. It doesn't really belong to the message which becomes obvious
> when you look at messages stored in maildir format. So this is IMO a
> problem of spamoracle. You may consider to write a wrapper script which
> provides a dummy "From line" to make spamoracle work. Or you may consider
> to switch to another tool like bogofilter or spamassassin.

Andreas,

I found now that in fact I was wrong with my assumption.
It is not the From line what puzzles spamoracle - it is the line delimiter!

The mail which did not work had DOS delimiters.
If I save the mail, which was not correctly parsed by spamoracle, as UNIX 
fileformat, it then works.

I also checked with exactly the same mail
1) kmail from KDE 3.1.3 => piped to spamoracle with UNIX delimiters
2) kmail from KDE 3.2 CVS => piped to spamoracle with DOS delimiters

???
Why that change ?
Can this be reverted ?

>
> Is it okay for you when I close the report with respect to the new one for
> the size problem?

No, see above.

> Anyway, thanks for your patience and your help on 
> tracking it down.

You're welcome. I really want to solve that problem, as it worked perfectly 
with KDE 3.1.x, and I think this bug will puzzle a lot of other people.

- --
Best regards/Sch
Comment 12 Ingo Klöcker 2004-01-03 17:41:17 UTC
Do I understand correctly that filtering incoming messages (from a POP server?) doesn't work, but that filtering messages with Ctrl+J does work?

If yes, then I think I know what the problem is and how to fix it.
Comment 13 Martin Koller 2004-01-04 11:37:03 UTC
Subject: Re:  filters don't work anymore as in 3.1.4

On Saturday 03 January 2004 17:41, Ingo Kl
Comment 14 Andreas Gungl 2004-01-21 23:55:07 UTC
The fix didn't make it into KDE 3.2 but it's possible it will be ported back to 3.2.1 - in any case it will be included in the next KDE PIM release.