Bug 299968 - Filters aren't applied on incoming mails
Summary: Filters aren't applied on incoming mails
Status: RESOLVED FIXED
Alias: None
Product: Akonadi
Classification: Unclassified
Component: Mail Filter Agent (show other bugs)
Version: 4.9
Platform: Ubuntu Packages Linux
: NOR major with 26 votes (vote)
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-05-13 23:09 UTC by JaromirD
Modified: 2013-03-08 01:22 UTC (History)
10 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.10.1


Attachments
Screen shot showing inbox after receiving mail (345.16 KB, image/png)
2012-05-15 18:41 UTC, Thomas Arend
Details
Screen shot showing cron folder after receiving mail (271.15 KB, image/png)
2012-05-15 18:43 UTC, Thomas Arend
Details
kmail filter log after receiving messages (183.71 KB, text/html)
2012-05-15 18:45 UTC, Thomas Arend
Details
A filtered mail (1.73 KB, application/mbox)
2012-05-15 19:00 UTC, Thomas Arend
Details
This mail was not filtered (1.17 KB, application/mbox)
2012-05-15 19:02 UTC, Thomas Arend
Details
Test folder after rreiving 100 test messages (110.81 KB, application/mbox)
2012-05-18 22:45 UTC, Thomas Arend
Details
Test folder some minutes later (109.24 KB, application/mbox)
2012-05-18 22:52 UTC, Thomas Arend
Details
Three messages not moved to folder Test and not filtered in any way. (3.00 KB, application/mbox)
2012-05-18 22:54 UTC, Thomas Arend
Details

Note You need to log in before you can comment on or make changes to this bug.
Description JaromirD 2012-05-13 23:09:22 UTC
Neither spam filtering nor sorting the incoming mail (pop3 account) doesn't work. Nor it did in previous version of kmail. Manually applied filtering seems to work ok.

Reproducible: Always

Steps to Reproduce:
1. create any filter on pop3 account (may be bogofilter or filtering on senders)
2. wait for the incoming mail (no mail is sorted/filtered)
3. manually apply filters 
Actual Results:  
pt. 2 - no mail is sorted/filtered
pt. 3 -  filtering works as expected

Expected Results:  
mail should be filtered/sorted automatically when the kmail fetches it from the server. There should be no need for manual filterng (if the filter rules matches, of course)

Kubuntu Precise Pangolin, fully updated
Comment 1 Thomas Arend 2012-05-14 16:44:10 UTC
I am using 4.8.3 now and can confirm this nasty bug back to 4.8.0. Some of the incoming mails are filtered some are not. (They are cron job reports or positive spam and the filter criteria fits to all.)

When I select all messages in the message list, the filter is not working. When I filter the messages with after  "CROM" or "***SPAM" and apply the appropriate filter or all filters its working fine. 

But it was not my intend to do this every day multiple times by hand :-(
If its not fixed soon I will switch to another mail client.
Comment 2 Laurent Montel 2012-05-14 20:04:32 UTC
"If its not fixed soon I will switch to another mail client." so bye !
I use kmail all the time. It works fine.
So ok perhaps there is a bug but without a test case I can't fix.
What is your filter which doesn't work ?
What do you have in "Filter log viewer" ? 
etc.
Comment 3 Thomas Arend 2012-05-14 21:50:30 UTC
Hello, the filter criteria has not changed since years and is very simple.

If "X-Cron-Env" contains "<SHELL" then move to Folder "Cron". Which is the case in all CRON-Jobs.
Spamassassin Filter: If "X-Spam-Flag" = "Yes" then move to Folder "Spam".

I get ~20 messages per hour. When I receive them from the mailbox it may be 200 or 400 messages. Sometime 20 oder 30 are not filtered sometimes more sometimes less.

Yes I have a Log-Viewer put it is not always willing to show up. In the moment it is very unwilling and not cooperative. :-(  Upps now it is up. You have to be pationed. 8 GB and six cores are not so much power. It can take some minutes to start. I would like to send you the log file. But the "Save as", "Clear" or "Abort" buttons are also not so cooperative. Maybe they will do something in ten minutes, an hour or two. Or maybe next year? Can you wait? Are the scoll bars are on holiday?

The filter works on some incoming messages and on some not. It works by hand under special circumstances. That is selecting not all massages - which I do by searching for "CRON". Applying all filters to all messages Control-A Control-J does nothing but reports that "n of m discussion groups are updated. (I am using German version so English message may be a bit different.) A lot reports are "0 von 0 Diskussiongruppen wurden aktualisiert". In some cases n and m are increased and m is running ahead of n. Why does kamail2 not know at start how many messages are to filter. I had not figured out, what "Diskussionsgruppen" mean. Never defined such things.

Wow. After several minutes. The log-Viewer is running really fast and it seems that "save as" works now. You have to be patient. But it hangs in the file selection dialog. As I mentioned above me PC is not so powerful. Uh, not the file dialog has closed.But the log viewer is a little bit sleepy.

It had saved the log in kmail-filter.html (1 MByte) - big deal - but now it is a bit stressed after the work. I thing it has retired. - Uh, it wants to save the file again but the confirmation window for overwriting the old file is empty and doesn't react. Upps - after some minutes it shows its question.

Oh! How wonderful now the CRON-Jobs an other messages are moved to the right folder. After nearly six hours all messages are sorted! 

Never saw such a poor program. How was the trick done?

Have a good night.
Comment 4 Thomas Arend 2012-05-15 18:41:55 UTC
Created attachment 71111 [details]
Screen shot showing inbox after receiving mail

This Screen shot shows the inbox with some not filtered messages.
Comment 5 Thomas Arend 2012-05-15 18:43:50 UTC
Created attachment 71112 [details]
Screen shot showing cron folder after receiving mail

This screen shot shows the cron folder afer receiving the messages. A lot messages are in the folders but some are stil in the inbox.
Comment 6 Thomas Arend 2012-05-15 18:45:10 UTC
Created attachment 71113 [details]
kmail filter log after receiving messages

This is the filter log after receiving messsages.
Comment 7 Thomas Arend 2012-05-15 19:00:46 UTC
Created attachment 71114 [details]
A filtered mail

This mail was correctly filtered to cron folder.
Comment 8 Thomas Arend 2012-05-15 19:02:00 UTC
Created attachment 71115 [details]
This mail was not filtered

This mail stayed in the inbox and was not filtered and moved th cron folder.
Comment 9 Thomas Arend 2012-05-18 22:37:05 UTC
(In reply to comment #2)
> "If its not fixed soon I will switch to another mail client." so bye !
> I use kmail all the time. It works fine.
> So ok perhaps there is a bug but without a test case I can't fix.
> What is your filter which doesn't work ?
> What do you have in "Filter log viewer" ? 
> etc.

I created a test case. I send 100 messages with a delay of to seconds to my account.

for ((i=0; i<100 ; i++)) ; do echo "Test #`printf %4.4d $i`" | mail -s "Filter Test #`printf %4.4d $i`" thomas ; sleep 2 ; done

I created two filters. One to add a header field with a time stamp and one to move incoming messages to a folder named "Test".

Three messages 0056, 0053, 0003 where not filtered. Some messages showed up several times in the Test folder (#0038). Some were filtered more than once (#0038 3 times) and showed multiple times in the list. See mbox attached.

Some messages were missing in the inbox and the folder "Test" (#0096 .. #0099) but appeared 
later. Some duplicates (see #0038) were later cone and only three stayed in the folder "Test". 

Maybe this is a bug and not a feature. If it's a feature I don't need it. Please switch it off!
Comment 10 Thomas Arend 2012-05-18 22:45:44 UTC
Created attachment 71204 [details]
Test folder after rreiving 100 test messages

This is the mbox export of the Test folder after receiving the messages. Some messages are filtered more than once, three - not in the export - are not filtered and some (at least 4) are missing but show up later.
Comment 11 Thomas Arend 2012-05-18 22:52:55 UTC
Created attachment 71205 [details]
Test folder some minutes later

This is the Test folder some minutes later. Some duplicates are gone, some missed messages show up.
Comment 12 Thomas Arend 2012-05-18 22:54:41 UTC
Created attachment 71206 [details]
Three messages not moved to folder Test and not filtered in any way.

This is the mbox export of three unfiltered messages.
Comment 13 Thomas Arend 2012-05-18 22:59:58 UTC
After nearly an hour there is only one copy of message #0038 left in the list. But this message passed the filter three times as the three entries "X-Filter-Test" prove.. 

From thomas@arend-rhb.de  Sat May 19 00: 12:57 2012
Return-Path: <thomas@arend-rhb.de>
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on k2.arend.tksd
X-Spam-Status: No, score=-1.8 required=4.5 tests=AWL,BAYES_00,NO_RELAYS autolearn=ham version=3.3.1
X-Spam-Spammy: 0.955-5711--1994h-14398s--0d--HContent-Transfer-Encoding:7bit, 0.917-1648--1625h-6115s--0d--H*c:us-ascii
X-Spam-Hammy: 0.000-1990--32152h-1s--0d--H*p:D*arend-rhb.de, 0.000-5--75h-0s--0d--H*UA:mailx
X-Original-To: thomas
Delivered-To: thomas@arend-rhb.de
Received: by k2.arend.tksd (Postfix, from userid 1000) id B5C6428A17; Sat, 19 May 2012 00:08:45 +0200 (CEST)
Date: Sat, 19 May 2012 00:08:45 +0200
To: thomas@arend-rhb.de
Subject: Filter Test #0038
User-Agent: Heirloom mailx 12.2 01/07/07
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-ID: <20120518220845.B5C6428A17@k2.arend.tksd>
From: Thomas Arend <thomas@arend-rhb.de>
X-UIDL: 7@<"!D\P!!p1#"!d*6"!
X-Filter-Test: Sat, 19 May 2012 00:12:57 +0200
X-Filter-Test: Sat, 19 May 2012 00:12:57 +0200
X-Filter-Test: Sat, 19 May 2012 00:12:58 +0200

Test #0038
Comment 14 Thomas Arend 2012-09-12 19:21:44 UTC
With KDE 4.9.1 no action is carried out on the Filter. The log reports that the rule matched but the mail resides in the inbox.

-----  

[21:18:54] Applying filter action: Move Into Folder ""
[21:18:54] Evaluating filter rules: (match all of the following) "X-Spam-Flag" <contains> "YES"
[21:18:55] 1 = "X-Spam-Flag" <contains> "YES" (YES)
[21:18:55] Filter rules have matched.
------------------------------
[21:18:55] Applying filter action: Move Into Folder ""
[21:18:55] Evaluating filter rules: (match all of the following) "X-Spam-Flag" <contains> "YES"
[21:18:55] 1 = "X-Spam-Flag" <contains> "YES" (YES)
[21:18:55] Filter rules have matched.
------------------------------
[21:18:55] Applying filter action: Move Into Folder ""
Comment 15 Christophe Giboudeaux 2012-09-13 13:30:23 UTC
> [21:18:55] Applying filter action: Move Into Folder ""

Check the action in your filters settings. The field right to "Move into folder" is empty ?
Comment 16 András Manţia 2012-10-13 18:37:03 UTC
This looks like filters are misconfigured (could happen if the akonadi db was recreated or if migrating failed).
Comment 17 Thomas Arend 2012-10-13 19:18:13 UTC
@ András Manţia
How can a filter be misconfigured when it was newly created?
Why does it work on some messages and on some not? All messages are from the same type.
Comment 18 András Manţia 2012-10-13 19:27:29 UTC
 Applying filter action: Move Into Folder "" suggest that. Please copy the failing filter setting from $HOME/.kde[4]/share/config/akonadi_mailfilter_agentrc
Comment 19 Thomas Arend 2012-10-13 20:08:09 UTC
Here are my most common filters.

Some messages are moved some not.

The filter protocol is useless, because it does not log an identifier for the message. You can't match a message to a log entry. 

So I don't no what happens:
Case 1: Filters are not applied.
Case 2: Filter rules fit, but check fails. Bug in the filter test.  
Case 3: Filter is applied, check positive, but command is not executed.

Without a working logging you can also ask a crystal ball. 

Thomas

[Filter #0]
Applicability=0
AutomaticName=false
ConfigureShortcut=true
ConfigureToolbar=true
Enabled=true
Icon=/usr/share/pixmaps/gnome-spider.png
StopProcessingHere=true
ToolbarName=Mail ist SPAM
action-args-0=632
action-name-0=transfer
actions=1
apply-on=check-mail,manual-filtering
contentsA=YES
fieldA=X-Spam-Flag
funcA=contains
identifier=QHIAWqUq7GpbNJLG
name=Mail ist SPAM
operator=and
rules=1

[Filter #1]
Applicability=0
AutomaticName=false
ConfigureShortcut=true
ConfigureToolbar=false
Enabled=true
Icon=system-run
StopProcessingHere=true
ToolbarName=CRON-Jobs
action-args-0=588
action-name-0=transfer
actions=1
apply-on=check-mail,manual-filtering
contentsA=<SHELL
fieldA=X-Cron-Env
funcA=contains
identifier=uuXAxfNMdyY0bg1k
name=CRON-Jobs
operator=and
rules=1
Comment 20 Ondrej Holecek 2013-01-22 18:13:36 UTC
In KMail shipped with 4.9.98 (4.10 RC3, openSUSE packaged) incoming IMAP but already read messages are not filtered at all.
Use case e.g. when someone read mail using both webmail interface and KMail, when message is marked as read in webmail, it stays unfiltered in inbox. When unread, message is filtered.

I can reproduce it 100% time so far.
Comment 21 András Manţia 2013-02-28 08:27:00 UTC
Fixed in KDE 4.10.1 and later by Wolfgang Rohdewald.
http://commits.kde.org/kdepim/185c572e079ef40bee2225e582c11dbdbb35840f
Comment 22 Joshua J. Kugler 2013-03-08 01:22:45 UTC
I have upgraded to 4.10.1. Filters are working much better! I still have one account, however, where the incoming messages are are not filtered until I do a manual filtering.

This is the same account that is not getting it's identity set properly. I have a filter for this account to set the identity, and a couple upgrades ago, the filter broke, setting it to a different identity in my config.  What information do you need from me to trouble shoot this?