Bug 182208 - messages passed through a pipe does not get updated in the main window
Summary: messages passed through a pipe does not get updated in the main window
Status: RESOLVED FIXED
Alias: None
Product: kmail
Classification: Applications
Component: filtering (show other bugs)
Version: SVN trunk (KDE 4)
Platform: Debian testing Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-01-28 22:35 UTC by envite
Modified: 2009-04-28 20:15 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Patch to preserve X-UID header when "Pipe Through" with ActionScheduler (1.24 KB, patch)
2009-03-20 01:52 UTC, Matthias Gehre
Details
Patch proposal (6.23 KB, patch)
2009-03-20 15:23 UTC, Thomas McGuire
Details

Note You need to log in before you can comment on or make changes to this bug.
Description envite 2009-01-28 22:35:58 UTC
Version:            (using KDE 4.2.0)
OS:                Linux
Installed from:    Debian testing/unstable Packages

I've detected that trying to pass a message through a pipe in a filter causes this message to appear in konsole:

kmail(29697) KMFolderIndex::updateIndexStreamPtr: utime( "/home/envite/.kde4/share/apps/kmail/filter/.7.index.ids" , 0) failed (KMMsgDict::getFolderIdsLocation( *this ))
kmail(29697) KMFolderIndex::updateIndexStreamPtr: utime( "/home/envite/.kde4/share/apps/kmail/filter/.7.index.ids" , 0) failed (KMMsgDict::getFolderIdsLocation( *this ))
kmail(29697) KMFolderIndex::updateIndexStreamPtr: utime( "/home/envite/.kde4/share/apps/kmail/filter/.7.index.ids" , 0) failed (KMMsgDict::getFolderIdsLocation( *this ))

index.ids number changes each time, and the message appears three times as shown. In the konsole I correctly see the output of the pipe, but it does not get updated into the main window nor in the IMAP server.

Trying to ctrl+j the message again show another three fails, with a new index.ids number, a new correct pipe (spamassasin -L) output, and nothing to happen in the main window.
Comment 1 envite 2009-02-01 04:00:43 UTC
I've detected that index.ids has nothing to do here: Now these messages does not appear anymore, but messages still are piped but not updated.

It is a hell that all the spam does not get detected!
Comment 2 Matthias Gehre 2009-03-17 15:52:51 UTC
I see the same behavior on CTRL-j, Status Line says "n messages are waiting to be filtered" and nothing updates.

But when enabling "Add this filter to the Apply Filter menu" in "'Configure Filters->Advanced" and then applying only this (=the "Pipe Through") filter, it works as expected. I'm still investigating, how this could relate.


Do you see the same behavior?

I'm using KMail 1.11.1 with IMAP and local folders.
Comment 3 Matthias Gehre 2009-03-18 03:29:41 UTC
Further investigation shows that the filters are not executed, if ANY filter "Move Into" an IMAP folder. I tried with just the four rules generated for bogofilter.
Even when changing the "Move Into" target of "Classify as Spam" rule (which is _not_ even executed on CTRL-J) to an IMAP folder, no filter rules seem to be executed at all on CTRL-J and no X-Bogosity line is added to header.
Changing the "Move Into" target back to an local directory fixes the problem and the filter rules work as expected (X-Bogosity line is added and them message conditionally moved to local Spam folder).
Comment 4 Jaime Torres 2009-03-18 12:13:25 UTC
Confirmed, when a filter has a "Move Into" imap folder, no action is performed by any filter.
svn trunk r939757 (kde 4.2.66)
Comment 5 Matthias Gehre 2009-03-18 14:57:37 UTC
If someone could point me into the right direction, I would try to look into the code, debug it and write a patch against it.

And: Not being able to filter spam these days deserves a higher bug-severity than "normal"...
Comment 6 Thomas McGuire 2009-03-18 22:01:31 UTC
> If someone could point me into the right direction, I would try to look into the code, debug it and write a patch against it.

Ah, the endless quest with broken IMAP filters returns...
IIRC, KMail has to filtering systems (for historical reasons): one called ActionScheduler and one called KMFilterManager. (Again, IIRC) KMail uses the ActionScheduler as soon as there is at least on online IMAP target folder, because the KMFilterManager can not deal with that. Seems that there is some bug in the ActionScheduler.

Did this always not work or just regressed recently? If it regressed recently, a wild guess of mine would be that this is because of r908961, see http://websvn.kde.org/?view=rev&revision=908961.

Another interesting ActionScheduler commit is r833475, see http://websvn.kde.org/?view=rev&revision=833475. And possibly r816901.

Matthias, I hope these pointers are already of some use for you. If you need more info, just ask. I really appreciate that you want to help out here.
Comment 7 Matthias Gehre 2009-03-20 01:52:17 UTC
Created attachment 32276 [details]
Patch to preserve X-UID header when "Pipe Through" with ActionScheduler
Comment 8 Matthias Gehre 2009-03-20 01:55:37 UTC
I finally got it working with my local changes to kmail trunk.
But I do not want to introduce any further hacks to the code, so I need some assistance concerning the creation of the patches.
These are the problems I discovered:

1. kmfilteraction.cpp:1721
PipeJob *job = new PipeJob(0, aMsg, commandLine, tempFileName);
QObject::connect ( job, SIGNAL( done() ),
                     handler, SLOT( actionMessage() ) );
This fails to connect and so actionMessage is never called after a PipeJob.
I manually added a call to actionMessage() at the end of PipeJob::run(), but this is not how it should be solved. Should be pretty obvious to fix, but unfortunatly I'm no Qt guy, so could someone have a look at it?
This fixes the "After 'Pipe Througth' on IMAP, filterings stalls"

2.
--- kmcommands.cpp	(revision 941185)
+++ kmcommands.cpp	(working copy)
@@ -1739,9 +1739,19 @@
  void KMMetaFilterActionCommand::start()
 {
-  if ( ActionScheduler::isEnabled() && false ) { // don't use it for now
+  if ( ActionScheduler::isEnabled() 
+    || kmkernel->filterMgr()->atLeastOneOnlineImapFolderTarget() ) {
This function is executed when the user chooses to run only one filter (from the menu or toolbar). My patch makes it use the ActionScheduler when we are working on IMAP, exactly
as the Ctrl-J behavior currently is.

3. Later in the same function
KMFilterMgr::FilterSet set = KMFilterMgr::All;
QList<KMFilter*> filters;
filters.append( mFilter );
ActionScheduler *scheduler = new ActionScheduler( set, filters );
In trunk, filters, that are not executed on incoming mail, not on outgoing mail and not on manual filtering (ctrl+j), are not executed on explicit user wish. They are compared to the FilterSet mask (All = Inbound|Outbound|Explicit) and this test fails.
So I tried to use KMFilterMgr::FilterSet set = KMFilterMgr::NoSet
and in ActionScheduler::filterMessage():
--- actionscheduler.cpp	(revision 941185)
+++ actionscheduler.cpp	(working copy)
@@ -624,7 +624,8 @@
       ((mSet & KMFilterMgr::Inbound) && (*mFilterIt)->applyOnInbound() &&
        (!mAccount ||
         (mAccount && (*mFilterIt)->applyOnAccount(mAccountId)))) ||
-      ((mSet & KMFilterMgr::Explicit) && (*mFilterIt)->applyOnExplicit())) {
+      ((mSet & KMFilterMgr::Explicit) && (*mFilterIt)->applyOnExplicit()) ||
+       (mSet == KMFilterMgr::All)) {
 
       // filter is applicable
     if ( FilterLog::instance()->isLogging() ) {
But here I am stuck: After setting set = KMFilterMgr::NoSet,
ActionScheduler::filterMessage is not even executed. Maybe this one is obvious, too, to someone more confident with the codebase.

4. A Patch to keep the X-UID field after using "Pipe Through" with ActionScheduler (see above)
I copied the code from the corresponding kmfilter function. This one can be applied as is.

Please excuse if I did something obviously wrong - its my first time with the kmail sources.
Comment 9 Thomas McGuire 2009-03-20 15:21:52 UTC
Hi Matthias,

great work figuring those things out! The filter stuff certainly belongs to the more complicated parts of KMail.
I'll attach a patch which contains all your points (your patch only had one point in it).

Please test it on your side and see if everything works as expected. Once done, send it to reviewboard.kde.org, see http://techbase.kde.org/index.php?title=Contribute/Send_Patches&action=submit. (other devs will probably also want to have a look at the patch)

To 1: Good catch. I guess this is a result of Qt3 -> Qt4 porting. For some reason, the PipeJob redeclared done() as a public virtual method, while in the base class (ThreadWeaver::Job), it was a non-virtual signal. Strange. I've fixed that, see the attached patch.

To 2: I'm a bit concerned that this might introduce regressions for POP3 users, since they'll suddenly switch from KMFilterMgr to ActionScheduler. But we have to fix those regressions then, so the fix is OK.

To 3: Added a new property to the ActionScheduler: ignoreFilterSet. That does the trick.

To 4: Interesting. Which spam filter actually removes that header?
Comment 10 Thomas McGuire 2009-03-20 15:23:14 UTC
Created attachment 32287 [details]
Patch proposal

You'll need to revert your local changes, then apply this patch with patch -p0 < filter.diff
Comment 11 Thomas McGuire 2009-03-25 00:32:54 UTC
Matthias: ping
Comment 12 Matthias Gehre 2009-03-25 01:03:14 UTC
Thomas: Pong
I'm still there and working on this issue. My patch seems not to fix the whole problem (but I'm sure that it did at some time...?). 

Current status is that piping through Bogofilter does update the message, but the next filter comparing the X-Bogosity does not match, even if it should. Applying it by Ctrl+J works as it is supposed to.

I'm have some other work going on, but I'm definitly not dropping this thing until everything is fixed!
Comment 13 Thomas McGuire 2009-03-25 17:39:02 UTC
Thanks Matthias for the update. I mainly wanted to know what you think about the changes I proposed in the patch I attached.
Comment 14 Matthias Gehre 2009-03-26 12:05:59 UTC
Your patch is doing nicely what it should, but I'm still looking into the remaining issues.
Comment 15 CW West 2009-04-02 23:34:32 UTC
I just upgraded to 1.11.2 (kde 4.2.2) and I'm still seeing this IMAP filtering bug.  Very happy to see some activity on this important issue (spam filtering is critical).
Comment 16 Matthias Gehre 2009-04-03 01:14:44 UTC
I finally fixed all issues and the patch is now reviewable under http://reviewboard.kde.org/r/369/
Comment 17 Thomas McGuire 2009-04-08 15:54:02 UTC
SVN commit 951033 by tmcguire:

Fix online IMAP filters not working in various situations.

Patch by Matthias Gehre <M.Gehre@gmx.de>, thanks very much!

BUG: 171061
BUG: 174430
BUG: 182208
BUG: 186264


 M  +15 -11    actionscheduler.cpp  
 M  +17 -5     actionscheduler.h  
 M  +4 -1      kmcommands.cpp  
 M  +22 -6     kmfilteraction.cpp  
 M  +1 -1      managesievescriptsdialog.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=951033
Comment 18 Thomas McGuire 2009-04-28 20:15:08 UTC
SVN commit 960606 by tmcguire:

Backport r951033 by tmcguire from trunk to the 4.2 branch:

Fix online IMAP filters not working in various situations.

Patch by Matthias Gehre <M.Gehre@gmx.de>, thanks very much!

CCBUG: 171061
CCBUG: 174430
CCBUG: 182208
CCBUG: 186264



 M  +15 -11    actionscheduler.cpp  
 M  +17 -5     actionscheduler.h  
 M  +4 -1      kmcommands.cpp  
 M  +22 -6     kmfilteraction.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=960606