Bug 182208

I've detected that index.ids has nothing to do here: Now these messages does not appear anymore, but messages still are piped but not updated.

It is a hell that all the spam does not get detected!

I see the same behavior on CTRL-j, Status Line says "n messages are waiting to be filtered" and nothing updates.

But when enabling "Add this filter to the Apply Filter menu" in "'Configure Filters->Advanced" and then applying only this (=the "Pipe Through") filter, it works as expected. I'm still investigating, how this could relate.


Do you see the same behavior?

I'm using KMail 1.11.1 with IMAP and local folders.

Further investigation shows that the filters are not executed, if ANY filter "Move Into" an IMAP folder. I tried with just the four rules generated for bogofilter.
Even when changing the "Move Into" target of "Classify as Spam" rule (which is _not_ even executed on CTRL-J) to an IMAP folder, no filter rules seem to be executed at all on CTRL-J and no X-Bogosity line is added to header.
Changing the "Move Into" target back to an local directory fixes the problem and the filter rules work as expected (X-Bogosity line is added and them message conditionally moved to local Spam folder).

Confirmed, when a filter has a "Move Into" imap folder, no action is performed by any filter.
svn trunk r939757 (kde 4.2.66)

If someone could point me into the right direction, I would try to look into the code, debug it and write a patch against it.

And: Not being able to filter spam these days deserves a higher bug-severity than "normal"...

> If someone could point me into the right direction, I would try to look into the code, debug it and write a patch against it.

Ah, the endless quest with broken IMAP filters returns...
IIRC, KMail has to filtering systems (for historical reasons): one called ActionScheduler and one called KMFilterManager. (Again, IIRC) KMail uses the ActionScheduler as soon as there is at least on online IMAP target folder, because the KMFilterManager can not deal with that. Seems that there is some bug in the ActionScheduler.

Did this always not work or just regressed recently? If it regressed recently, a wild guess of mine would be that this is because of r908961, see http://websvn.kde.org/?view=rev&revision=908961.

Another interesting ActionScheduler commit is r833475, see http://websvn.kde.org/?view=rev&revision=833475. And possibly r816901.

Matthias, I hope these pointers are already of some use for you. If you need more info, just ask. I really appreciate that you want to help out here.

Created attachment 32276 [details]
Patch to preserve X-UID header when "Pipe Through" with ActionScheduler

I finally got it working with my local changes to kmail trunk.
But I do not want to introduce any further hacks to the code, so I need some assistance concerning the creation of the patches.
These are the problems I discovered:

1. kmfilteraction.cpp:1721
PipeJob *job = new PipeJob(0, aMsg, commandLine, tempFileName);
QObject::connect ( job, SIGNAL( done() ),
                     handler, SLOT( actionMessage() ) );
This fails to connect and so actionMessage is never called after a PipeJob.
I manually added a call to actionMessage() at the end of PipeJob::run(), but this is not how it should be solved. Should be pretty obvious to fix, but unfortunatly I'm no Qt guy, so could someone have a look at it?
This fixes the "After 'Pipe Througth' on IMAP, filterings stalls"

2.
--- kmcommands.cpp	(revision 941185)
+++ kmcommands.cpp	(working copy)
@@ -1739,9 +1739,19 @@
  void KMMetaFilterActionCommand::start()
 {
-  if ( ActionScheduler::isEnabled() && false ) { // don't use it for now
+  if ( ActionScheduler::isEnabled() 
+    || kmkernel->filterMgr()->atLeastOneOnlineImapFolderTarget() ) {
This function is executed when the user chooses to run only one filter (from the menu or toolbar). My patch makes it use the ActionScheduler when we are working on IMAP, exactly
as the Ctrl-J behavior currently is.

3. Later in the same function
KMFilterMgr::FilterSet set = KMFilterMgr::All;
QList<KMFilter*> filters;
filters.append( mFilter );
ActionScheduler *scheduler = new ActionScheduler( set, filters );
In trunk, filters, that are not executed on incoming mail, not on outgoing mail and not on manual filtering (ctrl+j), are not executed on explicit user wish. They are compared to the FilterSet mask (All = Inbound|Outbound|Explicit) and this test fails.
So I tried to use KMFilterMgr::FilterSet set = KMFilterMgr::NoSet
and in ActionScheduler::filterMessage():
--- actionscheduler.cpp	(revision 941185)
+++ actionscheduler.cpp	(working copy)
@@ -624,7 +624,8 @@
       ((mSet & KMFilterMgr::Inbound) && (*mFilterIt)->applyOnInbound() &&
        (!mAccount ||
         (mAccount && (*mFilterIt)->applyOnAccount(mAccountId)))) ||
-      ((mSet & KMFilterMgr::Explicit) && (*mFilterIt)->applyOnExplicit())) {
+      ((mSet & KMFilterMgr::Explicit) && (*mFilterIt)->applyOnExplicit()) ||
+       (mSet == KMFilterMgr::All)) {
 
       // filter is applicable
     if ( FilterLog::instance()->isLogging() ) {
But here I am stuck: After setting set = KMFilterMgr::NoSet,
ActionScheduler::filterMessage is not even executed. Maybe this one is obvious, too, to someone more confident with the codebase.

4. A Patch to keep the X-UID field after using "Pipe Through" with ActionScheduler (see above)
I copied the code from the corresponding kmfilter function. This one can be applied as is.

Please excuse if I did something obviously wrong - its my first time with the kmail sources.

Hi Matthias,

great work figuring those things out! The filter stuff certainly belongs to the more complicated parts of KMail.
I'll attach a patch which contains all your points (your patch only had one point in it).

Please test it on your side and see if everything works as expected. Once done, send it to reviewboard.kde.org, see http://techbase.kde.org/index.php?title=Contribute/Send_Patches&action=submit. (other devs will probably also want to have a look at the patch)

To 1: Good catch. I guess this is a result of Qt3 -> Qt4 porting. For some reason, the PipeJob redeclared done() as a public virtual method, while in the base class (ThreadWeaver::Job), it was a non-virtual signal. Strange. I've fixed that, see the attached patch.

To 2: I'm a bit concerned that this might introduce regressions for POP3 users, since they'll suddenly switch from KMFilterMgr to ActionScheduler. But we have to fix those regressions then, so the fix is OK.

To 3: Added a new property to the ActionScheduler: ignoreFilterSet. That does the trick.

To 4: Interesting. Which spam filter actually removes that header?

Created attachment 32287 [details]
Patch proposal

You'll need to revert your local changes, then apply this patch with patch -p0 < filter.diff

Matthias: ping

Thomas: Pong
I'm still there and working on this issue. My patch seems not to fix the whole problem (but I'm sure that it did at some time...?). 

Current status is that piping through Bogofilter does update the message, but the next filter comparing the X-Bogosity does not match, even if it should. Applying it by Ctrl+J works as it is supposed to.

I'm have some other work going on, but I'm definitly not dropping this thing until everything is fixed!

Thanks Matthias for the update. I mainly wanted to know what you think about the changes I proposed in the patch I attached.

Your patch is doing nicely what it should, but I'm still looking into the remaining issues.

I just upgraded to 1.11.2 (kde 4.2.2) and I'm still seeing this IMAP filtering bug.  Very happy to see some activity on this important issue (spam filtering is critical).

I finally fixed all issues and the patch is now reviewable under http://reviewboard.kde.org/r/369/

SVN commit 951033 by tmcguire:

Fix online IMAP filters not working in various situations.

Patch by Matthias Gehre <M.Gehre@gmx.de>, thanks very much!

BUG: 171061
BUG: 174430
BUG: 182208
BUG: 186264


 M  +15 -11    actionscheduler.cpp  
 M  +17 -5     actionscheduler.h  
 M  +4 -1      kmcommands.cpp  
 M  +22 -6     kmfilteraction.cpp  
 M  +1 -1      managesievescriptsdialog.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=951033

SVN commit 960606 by tmcguire:

Backport r951033 by tmcguire from trunk to the 4.2 branch:

Fix online IMAP filters not working in various situations.

Patch by Matthias Gehre <M.Gehre@gmx.de>, thanks very much!

CCBUG: 171061
CCBUG: 174430
CCBUG: 182208
CCBUG: 186264



 M  +15 -11    actionscheduler.cpp  
 M  +17 -5     actionscheduler.h  
 M  +4 -1      kmcommands.cpp  
 M  +22 -6     kmfilteraction.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=960606