| Summary: | Filters don't work because headers are not downloaded | ||
|---|---|---|---|
| Product: | [Frameworks and Libraries] Akonadi | Reporter: | S. Burmeister <sven.burmeister> |
| Component: | Mail Filter Agent | Assignee: | kdepim bugs <kdepim-bugs> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | ahartmetz, amantia, bluelightning |
| Priority: | NOR | ||
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Platform: | openSUSE | ||
| OS: | Linux | ||
| Latest Commit: | http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69 | Version Fixed In: | 4.9 |
| Sentry Crash Report: | |||
Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia. Committed on 04/07/2012 at 17:35. Pushed by amantia into branch 'master'. A biggish refactoring of the filter functionality: 1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only). This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list) Might fix 293768. 2) Download only what is really needed for the filter and add safety checks that the mail is not modified if the requested part is missing. So far the distinction was only betwee the body and rest. This should speed up filtering. Also related to 287752, but should fix also 292283, 288109. 3) Rename qDebug->kDebug + fix some includes 4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates with empty content), eg. 275233 All bug reporters: in case you still see the bug, reopen the corresponding one. For the reports that were CC'd, but not closed, please try to reproduce and tell the result. So far the bug is fixed on master, but might be backported later to 4.9. Related: bug 287752, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682 M +7 -11 kmail/kmcommands.cpp M +10 -7 kmail/kmcommands.h M +1 -1 kmail/kmmainwidget.cpp M +1 -5 mailcommon/filter/filteraction.cpp M +6 -5 mailcommon/filter/filteraction.h M +6 -0 mailcommon/filter/filteractionaddheader.cpp M +2 -0 mailcommon/filter/filteractionaddheader.h M +2 -2 mailcommon/filter/filteractionaddtag.cpp M +1 -1 mailcommon/filter/filteractionaddtag.h M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h M +7 -0 mailcommon/filter/filteractionbeep.cpp M +1 -0 mailcommon/filter/filteractionbeep.h M +18 -1 mailcommon/filter/filteractioncopy.cpp M +7 -0 mailcommon/filter/filteractioncopy.h M +6 -0 mailcommon/filter/filteractiondelete.cpp M +1 -0 mailcommon/filter/filteractiondelete.h M +5 -0 mailcommon/filter/filteractionexec.cpp M +1 -0 mailcommon/filter/filteractionexec.h M +5 -0 mailcommon/filter/filteractionforward.cpp M +1 -0 mailcommon/filter/filteractionforward.h M +6 -0 mailcommon/filter/filteractionmove.cpp M +2 -0 mailcommon/filter/filteractionmove.h M +6 -0 mailcommon/filter/filteractionpipethrough.cpp M +1 -0 mailcommon/filter/filteractionpipethrough.h M +6 -0 mailcommon/filter/filteractionplaysound.cpp M +2 -0 mailcommon/filter/filteractionplaysound.h M +5 -0 mailcommon/filter/filteractionredirect.cpp M +1 -0 mailcommon/filter/filteractionredirect.h M +6 -1 mailcommon/filter/filteractionremoveheader.cpp M +1 -0 mailcommon/filter/filteractionremoveheader.h M +6 -0 mailcommon/filter/filteractionreplyto.cpp M +1 -0 mailcommon/filter/filteractionreplyto.h M +6 -0 mailcommon/filter/filteractionrewriteheader.cpp M +1 -0 mailcommon/filter/filteractionrewriteheader.h M +6 -0 mailcommon/filter/filteractionsendfakedisposition.cpp M +1 -0 mailcommon/filter/filteractionsendfakedisposition.h M +8 -2 mailcommon/filter/filteractionsendreceipt.cpp M +1 -0 mailcommon/filter/filteractionsendreceipt.h M +6 -0 mailcommon/filter/filteractionsetidentity.cpp M +1 -0 mailcommon/filter/filteractionsetidentity.h M +6 -0 mailcommon/filter/filteractionsetstatus.cpp M +1 -0 mailcommon/filter/filteractionsetstatus.h M +6 -0 mailcommon/filter/filteractionsettransport.cpp M +1 -0 mailcommon/filter/filteractionsettransport.h M +5 -0 mailcommon/filter/filteractionstatus.cpp M +2 -0 mailcommon/filter/filteractionstatus.h M +5 -2 mailcommon/filter/filteractionunsetstatus.cpp M +2 -1 mailcommon/filter/filteractionunsetstatus.h M +4 -4 mailcommon/filter/filterimporter/filterimporterevolution.cpp M +1 -1 mailcommon/filter/filterimporter/filterimporterprocmail.cpp M +1 -1 mailcommon/filter/filterimporter/filterimportersylpheed.cpp M +2 -2 mailcommon/filter/filtermanager.cpp M +2 -8 mailcommon/filter/filtermanager.h M +8 -2 mailcommon/filter/itemcontext.cpp M +8 -1 mailcommon/filter/itemcontext.h M +10 -17 mailcommon/filter/kmfilterdialog.cpp M +2 -1 mailcommon/filter/kmfilterdialog.h M +25 -8 mailcommon/filter/mailfilter.cpp M +4 -3 mailcommon/filter/mailfilter.h M +48 -20 mailcommon/searchpattern.cpp M +34 -11 mailcommon/searchpattern.h M +1 -1 mailcommon/tests/searchpatterntest.cpp M +78 -47 mailfilteragent/filtermanager.cpp M +10 -8 mailfilteragent/filtermanager.h M +8 -6 mailfilteragent/mailfilteragent.cpp M +3 -0 mailfilteragent/mailfilteragent.h http://commits.kde.org/kdepim/d64d61b470c56b92294f6adee6d74305a217628d Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia. Committed on 04/07/2012 at 17:35. Pushed by amantia into branch 'KDE/4.9'. (Backport filter fix from master) A biggish refactoring of the filter functionality: 1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only). This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list) Might fix 293768. 2) Download only what is really needed for the filter and add safety checks that the mail is not modified if the requested part is missing. So far the distinction was only betwee the body and rest. This should speed up filtering. Also related to 287752, but should fix also 292283, 288109. 3) Rename qDebug->kDebug + fix some includes 4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates with empty content), eg. 275233 All bug reporters: in case you still see the bug, reopen the corresponding one. For the reports that were CC'd, but not closed, please try to reproduce and tell the result. So far the bug is fixed on master, but might be backported later to 4.9. Related: bug 287752, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682 (cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d) FIXED-IN: 4.9 M +7 -11 kmail/kmcommands.cpp M +10 -7 kmail/kmcommands.h M +1 -1 kmail/kmmainwidget.cpp M +1 -5 mailcommon/filter/filteraction.cpp M +6 -5 mailcommon/filter/filteraction.h M +6 -0 mailcommon/filter/filteractionaddheader.cpp M +2 -0 mailcommon/filter/filteractionaddheader.h M +2 -2 mailcommon/filter/filteractionaddtag.cpp M +1 -1 mailcommon/filter/filteractionaddtag.h M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h M +7 -0 mailcommon/filter/filteractionbeep.cpp M +1 -0 mailcommon/filter/filteractionbeep.h M +18 -1 mailcommon/filter/filteractioncopy.cpp M +7 -0 mailcommon/filter/filteractioncopy.h M +6 -0 mailcommon/filter/filteractiondelete.cpp M +1 -0 mailcommon/filter/filteractiondelete.h M +5 -0 mailcommon/filter/filteractionexec.cpp M +1 -0 mailcommon/filter/filteractionexec.h M +5 -0 mailcommon/filter/filteractionforward.cpp M +1 -0 mailcommon/filter/filteractionforward.h M +6 -0 mailcommon/filter/filteractionmove.cpp M +2 -0 mailcommon/filter/filteractionmove.h M +6 -0 mailcommon/filter/filteractionpipethrough.cpp M +1 -0 mailcommon/filter/filteractionpipethrough.h M +6 -0 mailcommon/filter/filteractionplaysound.cpp M +2 -0 mailcommon/filter/filteractionplaysound.h M +5 -0 mailcommon/filter/filteractionredirect.cpp M +1 -0 mailcommon/filter/filteractionredirect.h M +6 -1 mailcommon/filter/filteractionremoveheader.cpp M +1 -0 mailcommon/filter/filteractionremoveheader.h M +6 -0 mailcommon/filter/filteractionreplyto.cpp M +1 -0 mailcommon/filter/filteractionreplyto.h M +6 -0 mailcommon/filter/filteractionrewriteheader.cpp M +1 -0 mailcommon/filter/filteractionrewriteheader.h M +6 -0 mailcommon/filter/filteractionsendfakedisposition.cpp M +1 -0 mailcommon/filter/filteractionsendfakedisposition.h M +8 -2 mailcommon/filter/filteractionsendreceipt.cpp M +1 -0 mailcommon/filter/filteractionsendreceipt.h M +6 -0 mailcommon/filter/filteractionsetidentity.cpp M +1 -0 mailcommon/filter/filteractionsetidentity.h M +6 -0 mailcommon/filter/filteractionsetstatus.cpp M +1 -0 mailcommon/filter/filteractionsetstatus.h M +6 -0 mailcommon/filter/filteractionsettransport.cpp M +1 -0 mailcommon/filter/filteractionsettransport.h M +5 -0 mailcommon/filter/filteractionstatus.cpp M +2 -0 mailcommon/filter/filteractionstatus.h M +5 -2 mailcommon/filter/filteractionunsetstatus.cpp M +2 -1 mailcommon/filter/filteractionunsetstatus.h M +4 -4 mailcommon/filter/filterimporter/filterimporterevolution.cpp M +1 -1 mailcommon/filter/filterimporter/filterimporterprocmail.cpp M +1 -1 mailcommon/filter/filterimporter/filterimportersylpheed.cpp M +2 -2 mailcommon/filter/filtermanager.cpp M +2 -8 mailcommon/filter/filtermanager.h M +8 -2 mailcommon/filter/itemcontext.cpp M +8 -1 mailcommon/filter/itemcontext.h M +10 -17 mailcommon/filter/kmfilterdialog.cpp M +2 -1 mailcommon/filter/kmfilterdialog.h M +25 -8 mailcommon/filter/mailfilter.cpp M +4 -3 mailcommon/filter/mailfilter.h M +48 -20 mailcommon/searchpattern.cpp M +34 -11 mailcommon/searchpattern.h M +1 -1 mailcommon/tests/searchpatterntest.cpp M +78 -47 mailfilteragent/filtermanager.cpp M +10 -8 mailfilteragent/filtermanager.h M +8 -6 mailfilteragent/mailfilteragent.cpp M +3 -0 mailfilteragent/mailfilteragent.h http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69 Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia. Committed on 06/07/2012 at 11:12. Pushed by amantia into branch 'KDE/4.8'. Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport. An unwanted side-effect might be that more data is downloaded than before, based on the type of filters used. FIXED-IN: 4.8.5 Related: bug 287752, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682 M +0 -1 mailfilteragent/mailfilteragent.cpp http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec |
Version: unspecified (using Devel) OS: Linux If emails arrive in the imap inbox they are filtered. If they are filtered for criteria other than the subject, i.e. things that have been downloaded for the message list already, filters fail. Filter log, notice that () is empty although the emails contains: List-Id: german translators <kde-i18n-de.kde.org>: [18:38:20] Evaluating filter rules: (match any of the following) "List-Id" <contains> "<kde-i18n-doc.kde.org>" "List-Id" <contains> "<kde-i18n-de.kde.org>" [18:38:20] 0 = "List-Id" <contains> "<kde-i18n-doc.kde.org>" () [18:38:20] 0 = "List-Id" <contains> "<kde-i18n-de.kde.org>" () but filtering for subject rules show that some email content is available for filters: [18:38:20] Evaluating filter rules: (match any of the following) "Subject" <contains> "[Bug " "Subject" <contains> "[Issue " "Subject" <contains> "Bugs" [18:38:20] 0 = "Subject" <contains> "[Bug " (Re: Hilfe, meine Netzwerkschnittstellen wollen "Nach oben" ) [18:38:20] 0 = "Subject" <contains> "[Issue " (Re: Hilfe, meine Netzwerkschnittstellen wollen "Nach oben" ) [18:38:20] 0 = "Subject" <contains> "Bugs" (Re: Hilfe, meine Netzwerkschnittstellen wollen "Nach oben" ) Akonadiconsole shows: KMailFilterKernel: * 35 33 (NAME "INBOX" MIMETYPE (message/rfc822 inode/directory) REMOTEID "/INBOX" REMOTEREVISION "" RESOURCE "akonadi_imap_resource_2" MESSAGES 1233 UNSEEN 145 SIZE 34112236 CACHEPOLICY (INHERIT true INTERVAL 5 CACHETIMEOUT 60 SYNCONDEMAND true LOCALPARTS (ENVELOPE HEAD)) ANCESTORS ((33 "imap://my_imap_account/") (0 "")) ENTITYDISPLAY "(\"Posteingang\" \"mail-folder-inbox\" \"\" ())" uidnext "350076" collectionflags "\\Answered \\Flagged \\Draft \\Deleted \\Seen KMAILFORWARDED KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $TODO $WATCHED $IGNORED $SIGNED $ATTACHMENT $REPLIED $JUNK $NOTJUNK $ERROR $QUEUED \\*" uidvalidity "1064465519" collectionquota "59271168 104858624" imapacl "imap_account lrswipcda %% " imapquota "user/imap_account %%%% STORAGE % 102401 %%%% STORAGE % 57882" AccessRights "a" INDEXPOLICY "(ENABLE false)" collectionannotations "/vendor/cmu/cyrus-imapd/sharedseen false" timestamp "1327318937") 950 OK List completed 951 STATUS 35 (MESSAGES UNSEEN SIZE) * STATUS "RWTH IMAP/INBOX" (MESSAGES 1233 UNSEEN 145 SIZE 34112236) 951 OK STATUS completed akonadi_mailfilter_agent: 2643 OK UID FETCH completed 2644 UID STORE 70989 REV 0 (REMOTEID "350075" PLD:HEAD[1] {447} + Ready for literal data (expecting 447 bytes) + Ready for literal data (expecting 348 bytes) * 70989 FETCH (REV 1) 2644 OK DATETIME "23-Jan-2012 17:38:20 +0000" STORE completed 2646 LSUB 35 0 () () * 35 33 (NAME "INBOX" MIMETYPE (message/rfc822 inode/directory) REMOTEID "/INBOX" REMOTEREVISION "" RESOURCE "akonadi_imap_resource_2" CACHEPOLICY (INHERIT true INTERVAL 5 CACHETIMEOUT 60 SYNCONDEMAND true LOCALPARTS (ENVELOPE HEAD)) ENTITYDISPLAY "(\"Posteingang\" \"mail-folder-inbox\" \"\" ())" uidnext "350076" collectionflags "\\Answered \\Flagged \\Draft \\Deleted \\Seen KMAILFORWARDED KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $TODO $WATCHED $IGNORED $SIGNED $ATTACHMENT $REPLIED $JUNK $NOTJUNK $ERROR $QUEUED \\*" uidvalidity "1064465519" collectionquota "59271168 104858624" imapacl "imap_account lrswipcda %% " imapquota "user/imap_account %%%% STORAGE % 102401 %%%% STORAGE % 57882" AccessRights "a" INDEXPOLICY "(ENABLE false)" collectionannotations "/vendor/cmu/cyrus-imapd/sharedseen false" timestamp "1327318937") 2646 OK List completed KMAil Kernel: * 70989 FETCH (UID 70989 REV 0 REMOTEID "350075" MIMETYPE "message/rfc822" COLLECTIONID 35 SIZE 5092 DATETIME "23-Jan-2012 18:38:20 +0000" FLAGS () PLD:ENVELOPE[1] {348} ("Mon, 23 Jan 2012 18:37:02 +0100" "Re: Hilfe, meine Netzwerkschnittstellen wollen \"Nach oben\"\r" (("Christian Gebler" NIL "geblerchristian" "googlemail.com")) NIL NIL (("german translators" NIL "kde-i18n-de" "kde.org")) NIL NIL "<201201231832.06323.lueck@hube-lueck.de>" "<4F1D9ABE.6080504@gmail.com>" "<201201231832.06323.lueck@hube-lueck.de>")) 3434 OK UID FETCH completed 3435 UID FETCH 70989 EXTERNALPAYLOAD (UID REMOTEID REMOTEREVISION COLLECTIONID FLAGS SIZE DATETIME PLD:ENVELOPE) * 70989 FETCH (UID 70989 REV 1 REMOTEID "350075" MIMETYPE "message/rfc822" COLLECTIONID 35 SIZE 795 DATETIME "23-Jan-2012 17:38:20 +0000" FLAGS () PLD:ENVELOPE[1] {348} ("Mon, 23 Jan 2012 18:37:02 +0100" "Re: Hilfe, meine Netzwerkschnittstellen wollen \"Nach oben\"\r" (("Christian Gebler" NIL "geblerchristian" "googlemail.com")) NIL NIL (("german translators" NIL "kde-i18n-de" "kde.org")) NIL NIL "<201201231832.06323.lueck@hube-lueck.de>" "<4F1D9ABE.6080504@gmail.com>" "<201201231832.06323.lueck@hube-lueck.de>")) 3435 OK UID FETCH completed 3436 LSUB 35 0 () (STATISTICS true ANCESTORS INF) * 35 33 (NAME "INBOX" MIMETYPE (message/rfc822 inode/directory) REMOTEID "/INBOX" REMOTEREVISION "" RESOURCE "akonadi_imap_resource_2" MESSAGES 1233 UNSEEN 145 SIZE 34112236 CACHEPOLICY (INHERIT true INTERVAL 5 CACHETIMEOUT 60 SYNCONDEMAND true LOCALPARTS (ENVELOPE HEAD)) ANCESTORS ((33 "imap://my_imap_account/") (0 "")) ENTITYDISPLAY "(\"Posteingang\" \"mail-folder-inbox\" \"\" ())" uidnext "350076" collectionflags "\\Answered \\Flagged \\Draft \\Deleted \\Seen KMAILFORWARDED KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $TODO $WATCHED $IGNORED $SIGNED $ATTACHMENT $REPLIED $JUNK $NOTJUNK $ERROR $QUEUED \\*" uidvalidity "1064465519" collectionquota "59271168 104858624" imapacl "imap_account lrswipcda %% " imapquota "user/imap_account %%%% STORAGE % 102401 %%%% STORAGE % 57882" AccessRights "a" INDEXPOLICY "(ENABLE false)" collectionannotations "/vendor/cmu/cyrus-imapd/sharedseen false" timestamp "1327318937") 3436 OK List completed 3437 STATUS 35 (MESSAGES UNSEEN SIZE) * STATUS "RWTH IMAP/INBOX" (MESSAGES 1233 UNSEEN 145 SIZE 34112236) 3437 OK STATUS completed If one selects the emails all of its content is downloaded and applying manual filters works. Reproducible: Always Steps to Reproduce: Create some filter criteria based on List-ID or List-Post headers Actual Results: Emails not filtered when they arrive in the inbox Expected Results: All needed information for filtering downloaded before filtering.