Bug 255388

Same here, I upgraded to KDE 4.6 b1 this morning and got a beta of kmail 2, and the mail in my inbox was never moved out of it to the other folders by my filters.

Same on my openSuSE box, after I upgraded to KMail 2.0.89 on KDE 4.5.80 (4.6 beta 1). I also confirm that manually applying filters on the message from it's contextual menu works.

Confirm same bug on Gentoo amd64 and KDE 4.6 beta 1.

Hej guys,

can you please post the exact configuration of your filters, e.g. what you have configured as inbox folder to filter on, on which resource the inbox folder is located etc.

It works fine for me if the filter points to my standard LocalFolders/inbox and pop3 resource feeds new mails into this folder, but it seems some slightly different configurations don't work. So I need the exact information to reproduce/fix it.
Thanks in advance!

Ciao,
Tobias

No problem, I'll attach my config file now.

Created attachment 54457 [details]
boudewijn's kmail config files (~/.kde/share/config/kmail*)

Created attachment 54465 [details]
Kmail2rc

Created attachment 54470 [details]
kmail*rc files

As written in bug 256389, which is maybe a duplicate of this one, I found the following facts which make filtering of incoming massages not work:
a) If the filter is set to apply only for a selected (pop3) account filtering does work on kmail start up or when applied manually via Ctrl+J. It does _not_ work when fetching new mail with a running kmail. It works if the filter is set to apply for all accounts.
b) Have your account configured to have the 'target folder' different from the standard inbox
=> filter do not work for incoming mail (neither on kmail start up nor when fetching new mail with running kmail), only when applied manually (Ctrl+J)

*** Bug 259687 has been marked as a duplicate of this bug. ***

I have used the filter log viewer which is part of Kmail2. The filter viewer does not log anything on incoming mails, but it writes logs wenn I manualy kick in the filter.

Since update to Kontact 4.6 beta3, filter mail is working in my case.

I wish I could say the same. I have also upgraded, but filtering is not working.

Created attachment 55361 [details]
KMail2 configuration files (AvL)

Today I installed a fresh Kubuntu 10.10 and upgraded immediately to Kontact 4.6 beta 3. After that upgrade I used Kontact for the first time and specified my filters. My filters operate on the Inbox of an IMAP account. I have three filters which filter on the List-Id header and the X-Bugzilla-URL header and move the relevant messages to different subdirectories of the Inbox of that IMAP-account. In the 'Advanced' section of the 'Configure Filters...' dialog I've set the options for all three filters to apply them to incoming messages and to only apply filters to the specific IMAP-account which is receiving those messages (I have another IMAP-account as well which doesn't need filtering). Like the other commenter, manually applying the filter works fine, but KMail doesn't do it automatically on incoming messages. So beta 3 doesn't fix it for me either.

Problem still there in OpenSuse 11.3 and Kubuntu 10.10 both with Kontact 4.6 / Kmail 2.0.89:

Filter are not applied on incoming mails, but work fine when applied manually.

I'm experiencing this same problem, but I can't figure out a pattern.  I have a number of filters (about 12).  All of them seem to work properly EXCEPT my spam/ham moving filters.  The first few filters work correctly (including classifying the mail as ham/span), but the next three (move to one place if it's spam, to another if it's ham, to a third if it's unsure) do NOT run automatically, but DO work if applied manually (as indicated above).  The filters AFTER these three DO work properly automatically.

I've been unable to find *any* difference between those three (which don't run automatically) and the others.

My filters also don't work after migration, and also if starting fresh and importing a previously exported from KMail1 filters list.  Creating a new filter in KMail2 processes OK. In KMail2 the filter actions show as "Unknown Folder" and the relevant line from from the export file is:

    action-args-0=.Computers.directory/.KDE.directory/kde-pim-meeting

This compares to an export of a new filter from KMail2 which shows:

    action-args-0=105

Looks like KMail2 uses a very different means of referring to folders.

Note that my mail folders are in ~/.Mail, but that shouldn't affect things.

Oh, and contents of the kmail2rc filter after the migration showing migrated account but unmigrated folder:

[Filter #0]
Applicability=0
AutomaticName=false
ConfigureShortcut=false
ConfigureToolbar=false
Icon=
StopProcessingHere=true
ToolbarName=<List-Id>:<kde-pim-meeting.kde.org>
accounts-set=akonadi_mixedmaildir_resource_0
action-args-0=.Computers.directory/.KDE.directory/kde-pim-meeting
action-name-0=transfer
actions=1
apply-on=check-mail,manual-filtering
contentsA=<kde-pim-meeting.kde.org>
fieldA=List-Id
funcA=contains
name=<List-Id>:<kde-pim-meeting.kde.org>
operator=and
rules=1

I've tried wiping out ALL of my filters and restarting.  Now I'm having the larger problem that my bogofilter filtering won't work.  It's set up to pipe the email through bogofilter and play a chime (so I could tell if it was running).
When the filter runs, the chime sounds, but the message is NOT being piped through the filter.

[Filter #0]
Applicability=2
AutomaticName=false
ConfigureShortcut=true
ConfigureToolbar=false
Icon=
StopProcessingHere=false
ToolbarName=Bogofilter
accounts-set=akonadi_imap_resource_0,akonadi_pop3_resource_0
action-args-0=/usr/bin/bogofilter -pev -c /home/cfigura/.bogofilter/.bogofilter.cf
action-args-1=/usr/share/sounds/Kopete_Received.ogg
action-name-0=filter app
action-name-1=play sound
actions=2
apply-on=check-mail,manual-filtering
contentsA=0
fieldA=<size>
funcA=greater
name=Bogofilter
operator=or
rules=1

[General]
filters=1

Info was supplied so the status should be changed to new or assigned.

One additional point: Filters only work when your default inbox is called "inbox". If you create a new user with e.g. German localization where it is called "Posteingang" filters do not work on incoming mail.

I can confirm this on Gentoo amd64, KDE 4.6.2, KDEPIM 4.6. (incl. KMail) pulled from Git on 2011-04-11, Akonadi 1.5.2.

At least in my case I've noticed this:
* *no* filters apply automatically to e-mail that is pulled by IMAP
* filters *do* automatically apply for e-mail that is pulled by POP
* all filters work just fine if triggered by hand

From this I suspect that the problem is not in particular filters, but just that incoming IMAP messages do not trigger the switch that applies the filters to them.

Before anyone asks the obvious: filters have been migrated from KMail1 and are set (applied by hand again) to automatically apply to incoming from all resources.

I am in the exact same situation as Matija, only on Gentoo x86, 4.6.2, kdepim 4.5.95. Automatic filters work for POP. For IMAP they only work when triggered manually.

I have described the Spamfilter problem in detail in a bug of my own (#266154).

(In reply to comment #22)
> I am in the exact same situation as Matija, only on Gentoo x86, 4.6.2, kdepim
> 4.5.95. Automatic filters work for POP. For IMAP they only work when triggered
> manually.
> 
> I have described the Spamfilter problem in detail in a bug of my own (266154).

If your spam filter is applied as well, when you manually trigger the filters, bug #266154 may well be a duplicate of this one.

Is this still not enough info?
If so, I'd be happy to provide it (e.g. my kmail2rc) if someone told me what parts, if any, I should blank out for security and privacy reasons.

When I click “Mark as Spam”, the filter action is carried out (move to Spam folder).

But even if the source of our two problems is the same (different handling of folders), I think my spam isssue and this one still constitute two different problems. This bug is about filters not getting triggered for IMAP mails, and mine is about mails not getting the Spam header from Bogofilter.
The Spamfilter is triggered alright here. I tested it by changing the pipe through command from “bogofilter -p -e” to “tee /some/file | bogofilter -p -e” and sent me a mail. Once I received it, /some/file was there with the content of the mail.

*** This bug has been confirmed by popular vote. ***

I too am experiencing the same symptoms under KDE 4.6.3. Under the Kmail Filter Rules, I have certain filters set to "Apply this filter to incoming messages >> from all but online IMAP accounts".  However, the flag that tells Kmail not to apply the filter to IMAP Emails is ignored and all Email from all accounts is filtered using the filter no matter the source (POP or IMAP accounts).

And exactly same here.

Newest KDEPIM (4.5.96 I believe), KDE 4.6.3 on Gentoo.

I use only IMAP and have quite a few filters triggered for specific accounts.

None of them gets applied - I have to constantly ctrl+J them.

Filtering works for me, i.e. moving to local folders.

Bogofilter does not work because the headers are not modified but that's bug 266154.

Forgot to add: My kmail version is KMail 2.0.89 and the incoming mails are in Posteingang (German for inbox) and not some other folder.

It only works once though, i.e. if they are retrieved. If they have been retrieved they are not treated as incoming and thus filters not applied.

Gentoo, amd64, KDE 4.6.3, KDE PIM 2.0.96

Filters works ok, with one exception.
Rule "From is in address book" not working, instead applying to such messages different rules.

Git commit 77349b1a20cb4e3d2689bcb3148a3c3e36e43492 by Andre Heinecke.
Committed on 25/05/2011 at 19:53.
Pushed by aheinecke into branch '4.6'.

Fix message filtering on startup.

    When the FilterManager was constructed it tried to filter the
    unread messages in the inbox before it had loaded the filtersettings.
    This caused it only to filter new messages while it was running.

    With this patch the tryToFilterInboxOnStartup function will always
    be called when new filters are added. (e.g. After the config is read)
    The delay in which the filter checks if the inbox is valid has also
    been reduced to avoid situations where the user can look at a mail
    in the inbox which is about to be filtered.

    This patch was provided by steveire

    BUG: 255388
    CCBUG: 257984

M  +2    -2    mailcommon/filtermanager.cpp     

http://commits.kde.org/kdepim/77349b1a20cb4e3d2689bcb3148a3c3e36e43492

Updated KDEPIM yesterday (25 May 2011 22:14:38 GMT+1) from Git and I still have to manually run filters on incoming IMAP messages.

I double-checked that the revision that was pulled and compiled was newer then the one with the bug solution.

The filters stil do not apply automatically in today's revision 3ad3f45d94297c19e6093a4704f8b91ee72f17ce

Applying them by hand (still) works just as well.

I propose reopening it.

Created attachment 60418 [details]
kmail2rc

I'm attaching my (somewhat anonymised) kmail2rc config file, if it happens to be something wrong with my settings.

Git commit 1dbe3bf4fee4ed41338f196765f4cdd5d2424d30 by Andre Heinecke.
Committed on 31/05/2011 at 20:38.
Pushed by aheinecke into branch '4.6'.

Fix inbox filtering for mailservers that use .inbox

    The remoteID .inbox was not recognized by the kmail kernel
    before. With this commit inbox mail filters will also be
    applied for inboxes with the remoteID .inbox

    CCBUG: 255388

M  +3    -1    mailcommon/mailkernel.cpp     

http://commits.kde.org/kdepim/1dbe3bf4fee4ed41338f196765f4cdd5d2424d30

Andre's patch solved it for me as well.

Thanks :)

Hello everyone,

Sorry for the bad news, but this bug has hit me with kmail 4.7.1, after the update from kmail 4.7.0, as part of the kubuntu oneiric beta versions.

Same symptom as everyone, i.e. manually applying the filter works, but it does not apply automatically.

One of the differences between 4.7.0 and 4.7.1 is that "inbox" is now called "boîte de réception", which is French for "inbox". Perhaps it's related?

In akonadi, the "Dossiers locaux" data source uses ~/.local/share/local-mail/.

I suggest reopening the bug report, as this looks like a regression.

Created attachment 63669 [details]
Filter section in kmail2rc

reopening then

Merci Christophe / thank you

I had the same problem and after starting a clean kde session (removing ~/.kde) the problem has disappeared.

The same problem here after updating to 4.7.4 (and all previous 4.7 I guess): in configuration bogofilter applies to everything apart of IMAP, mailbox folder is called differently and tied in account's configuration - manual filtering works but spam messages aren't automatically moved to trash.

I made filter (yes)|(spam\b) available in RMB menu and executing it manually for a message works.

I upgraded to 4.7.3 (Gnetoo stable) from 4.4.11 and also noticed the Mail filter problem. Filter on POP3 accounts works, but on IMAP it doesn't. Appllying filters manually works, but only on already fetched mails. So Ctrl+A and Ctrl+J does only work on mais previosly fetched (same for click and shift+click to get a list).
But i noticed something interesting: Filter _do_ work on IMAP accounts, if I filter on To, rather than List-ID for mailing lists. This obviously only works for initial mails and not replies.
So I launched the Filter Log Viewer and noticed the following filter output on a mail i wrote to myself in To and CC:
------------------------------
[14:24:26] Evaluating filter rules: (match any of the following) "To" <contains> "<linux-arm-kernel.lists.infradead.org>" "CC" <contains> "<linux-arm-kernel.lists.infradead.org>"
[14:24:26] 0 = "To" <contains> "<linux-arm-kernel.lists.infradead.org>" (alexander.stein@informatik.tu-chemnitz.de)
[14:24:26] 0 = "CC" <contains> "<linux-arm-kernel.lists.infradead.org>" ()
[14:24:26] Evaluating filter rules: (match all of the following) "List-Id" <contains> "<linux-arm-kernel.lists.infradead.org>"
[14:24:26] 0 = "List-Id" <contains> "<linux-arm-kernel.lists.infradead.org>" ()
[14:24:26] Evaluating filter rules: (match all of the following) "CC" <contains> "linux-kernel@vger.kernel.org"
[14:24:26] 0 = "CC" <contains> "linux-kernel@vger.kernel.org" ()
[14:24:26] Evaluating filter rules: (match any of the following) "To" <contains> "linux-kernel@vger.kernel.org" "CC" <contains> "linux-kernel@vger.kernel.org"
[14:24:26] 0 = "To" <contains> "linux-kernel@vger.kernel.org" (alexander.stein@informatik.tu-chemnitz.de)
[14:24:26] 0 = "CC" <contains> "linux-kernel@vger.kernel.org" ()
[14:24:26] Evaluating filter rules: (match all of the following) "List-Id" <contains> "<u-boot.lists.denx.de>"
[14:24:26] 0 = "List-Id" <contains> "<u-boot.lists.denx.de>" ()
---
If I manually apply the filter, I get the following output:
------------------------------
[14:25:42] Evaluating filter rules: (match all of the following) "List-Id" <contains> "<wireshark-dev.wireshark.org>"
[14:25:42] 0 = "List-Id" <contains> "<wireshark-dev.wireshark.org>" ()
[14:25:42] Evaluating filter rules: (match any of the following) "To" <contains> "<linux-arm-kernel.lists.infradead.org>" "CC" <contains> "<linux-arm-kernel.lists.infradead.org>"
[14:25:42] 0 = "To" <contains> "<linux-arm-kernel.lists.infradead.org>" (alexander.stein@informatik.tu-chemnitz.de)
[14:25:42] 0 = "CC" <contains> "<linux-arm-kernel.lists.infradead.org>" (alexander.stein@informatik.tu-chemnitz.de)
[14:25:42] Evaluating filter rules: (match all of the following) "List-Id" <contains> "<linux-arm-kernel.lists.infradead.org>"
[14:25:42] 0 = "List-Id" <contains> "<linux-arm-kernel.lists.infradead.org>" ()
[14:25:42] Evaluating filter rules: (match all of the following) "List-Id" <contains> "<linux-can.vger.kernel.org>"
[14:25:42] 0 = "List-Id" <contains> "<linux-can.vger.kernel.org>" ()
[14:25:42] Evaluating filter rules: (match all of the following) "CC" <contains> "linux-kernel@vger.kernel.org"
[14:25:42] 0 = "CC" <contains> "linux-kernel@vger.kernel.org" (alexander.stein@informatik.tu-chemnitz.de)
[14:25:42] Evaluating filter rules: (match any of the following) "To" <contains> "linux-kernel@vger.kernel.org" "CC" <contains> "linux-kernel@vger.kernel.org"
[14:25:42] 0 = "To" <contains> "linux-kernel@vger.kernel.org" (alexander.stein@informatik.tu-chemnitz.de)
[14:25:42] 0 = "CC" <contains> "linux-kernel@vger.kernel.org" (alexander.stein@informatik.tu-chemnitz.de)
[14:25:42] Evaluating filter rules: (match all of the following) "List-Id" <contains> "<u-boot.lists.denx.de>"
[14:25:42] 0 = "List-Id" <contains> "<u-boot.lists.denx.de>" ()
---
It seems not all E-Mail headers are passed to the filter upon initial mail reception.

Having a setup with several filters that apply to my IMAP account followed by 
several other filters that only apply to one of the five POP3 accounts, IMAP 
filters work fine and the POP3 filters did not work (i.e. email landed into 
Inbox) until I selected "from all accounts".

Running:

   $ kmail --version
   Qt: 4.7.4
   KDE Development Platform: 4.7.3 (4.7.3)
   KMail: 4.7.3


Much thanks.

It stopped to work for me in 4.8.0...

(In reply to comment #45)
> It stopped to work for me in 4.8.0...

It was fixed in 4.8.0 for some filter criteria but unfortunately there is a regression for other criteria. See bug 292497.

once upon a time, removing all filters and then creating new ones solved this for me.

Not anymore.

Seriously guys - bad enough that the migration to akonadi nuked my mails, that you need nepomuk for searching and everything is fucking slow compared to kmail1 - but no filtering?

Confirming it doesn't work in master.

Doing Ctrl-A Ctrl-J all the time when I enter my inbox.

Sorry, commented on a wrong bug

This is a dup of bug 292497 . For a workaround see https://bugs.kde.org/show_bug.cgi?id=292497#c18

This problem lives still in 4.8.5 but not so bad as before. When I receive a bunch of cron job mails - sometimes up to 1000 - only ~15% of the mails are not moved to the right folder. Applying the filter to all mails in the inbox does not work. When I select only same mails the filter works.

Same thing happens with spam mails.

This nasty behavior last since month. It seems a too hard task to get the filter working again. Must we accept the 85% solution?

Never change a running system!

(In reply to comment #51)
> This problem lives still in 4.8.5 but not so bad as before. 

4.9.0 is current and includes a lot of fixes regarding filtering. So please try again with that version.

> When I receive a
> bunch of cron job mails - sometimes up to 1000 - only ~15% of the mails are
> not moved to the right folder. Applying the filter to all mails in the inbox
> does not work. When I select only same mails the filter works.

Any debug output from the filter logs and akoandi logs? Please only from 4.9.

> This nasty behavior last since month. It seems a too hard task to get the
> filter working again. Must we accept the 85% solution?

It's a matter of time. Do you have some spare time left to help?
 
> Never change a running system!

You were given the possibility to change it, i.e. upgrade, you changed it, nobody forced you to do so.

But if you are into smart phrases:

Stagnation equals death.

or

If you don't go forward, you go backwards.

Same problem here since the start of kmail2. I'll try to get some logs from kmail and akonadi as soon as possible.

Do you need anything else ? (trying patches is quite easy for me as I'm using Gentoo)

If you are running openSUSE you can try the packages from the KR49 repo which includes a patch that should fix this issue.

Otherwise you can wait for 4.9.1 which will includes it as well.

For me, the bug reappeared with KDE 4.9.1. According to the filter protocols, no filters are applied to incoming mails. Manually applying filters work.

The same setting worked without problems with KDE 4.8.x (I have skipped 4.9.0).

Still same here with KDE 4.9.1 :(

Works for me with 4.9.1 openSUSE packages. Hence it's probably a config issue or a different bug which causes your trouble.

Can confirm this bug. 

With update to 4.9.1 it is now worse than ever. Also selected mails - which definately fit the rule - are not filtered.

Please do not add any "me too".

The only useful thing to add are ways to reproduce, including logs from akoandiconsole, after akonadictl restart in a konsole, info about the account type etc.

Are comment 8 and comment 20 still valid?

Hello,

the update from 4.9 to 4.9.1 sets the target folder to "" for the operation move to folder in all filters. The destination folders are lost. Here is the part you get when you export your filters.
  
  action-args-0=
  action-name-0=transfer
  actions=1

This is not the case for other actions. 

Nearly all filters have a destination folder and 99.99% of my incoming messages are handled by these filters. This is why it looked as if no message was filtered. (The rule move to folder is there but the folder is NULL. Maybe this is the default for inbox).
   
This is good and bad news. Bad news is that there is some routine which destroys filters (this happens only to destination folders as far as I can see). Good news: Easy to fix manually when you know about it.

Second bad news: Filter don't work reliable! We will see this later. The filtering of incoming messages don't work with a new account.

I deactivated all but one filter and set the lost target folder. Manually filtering on selected messages worked. (You have to be a little bit patient, the update of the message list may need some seconds even on a very fast machine.)

Next time I activated a second filter. Filtering 970 messages (real world) in the inbox was surprisingly fast. (Never before saw such fast filtering, another good news, but there are enough bad left). All messages (42 fit the rule) were correctly filtered.

In the next step I received the outstanding messages from the server. 12 of 125 messages (cron job reports) which fit the rule cron-job rule (Criteria is "X-Cron-Env: <SHELL") were not filtered / moved to the destination folder. (Criteria is "X-Cron-Env: <SHELL") Even after some reasonable time the messages stayed in the inbox (Posteingang in my case).

Filtering all the left messages in the inbox manually moved the remaining messages cron-job-reports into the folder for the cron jobs. I have no reasonable explanation why the filter works only on 90% of the messages in the first run.

To test this I created an new account, created a filter (see export at the end) and send 1000 messages to the new account (kmail2test@example.com) with following command on the mail server.

for ((i=0 ; i<1000 ; i++))
do 
   echo test `printf %4.4d $i` |  mail -s "Filter Test `printf %4.4d $i`" kmail2test@example.com
done

When I received the messages no message was filtered and the filter log is empty.

I filtered some single messages. All worked fine. Filter log as you expect. Fine.

I filtered the rest of the ~ 995 messages. All but one (0948) were moved. #948 stayed in the "Posteingang".

This is the filter log entry:
[13:59:27] Applying filter action: Move Into Folder "Lokale Ordner/Posteingang/Filter Test Mails"
[13:59:27] Evaluating filter rules: (match all of the following) "Subject" <contains> "Filter Test"
[13:59:27] 1 = "Subject" <contains> "Filter Test" (Filter Test 0948)
[13:59:27] Filter rules have matched.

The log entry gives no hint why the message was not transfered. 1 of 1000 is better than 12 from 125 but not good enough. There must be a reason. Applying the filter manually move the message to the folder.

Below are the filter and message example.

Have a nice weekend
Thomas

--- Exported filter ---
[Filter #0]
Applicability=0
AutomaticName=true
ConfigureShortcut=true
ConfigureToolbar=false
Enabled=true
StopProcessingHere=true
ToolbarName=<Subject>: Filter Test
action-args-0=Lokale Ordner/Posteingang/Filter Test Mails
action-name-0=transfer
actions=1
apply-on=check-mail,manual-filtering
contentsA=Filter Test
fieldA=Subject
funcA=contains
identifier=T3iGHGAHWt9PDyjT
name=<Subject>: Filter Test
operator=and
rules=1

[General]
filters=1
---- end of filter ---- 

This is the first messages. (Please notice the violation of RfC 4155 by kmail2 in the From_ line; https://bugs.kde.org/show_bug.cgi?id=297198; this bug does not apply to kmail1)

--- begin received message ---
From thomas@arend-rhb.de Sat, 15 Sep 2012 13:19:54 +0200
Return-Path: <thomas@arend-rhb.de>
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on k2.arend.tksd
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=AWL,NO_RELAYS autolearn=ham
        version=3.3.1
X-Original-To: kmail2test@example.com
Delivered-To: kmail2test@example.com
Received: by k2.arend.tksd (Postfix, from userid 1000)
        id 0A84028BC4; Sat, 15 Sep 2012 13:19:54 +0200 (CEST)
Date: Sat, 15 Sep 2012 13:19:54 +0200
To: kmail2test@example.com
Subject: Filter Test 0000
User-Agent: Heirloom mailx 12.2 01/07/07
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-Id: <20120915111955.0A84028BC4@k2.arend.tksd>
From: thomas@arend-rhb.de (Thomas Arend)
X-UIDL: O>_"!;[F"!+G%#!TlA!!

test 0000


-- end of message ---

--- message no 948 ----
From thomas@arend-rhb.de Sat, 15 Sep 2012 13:20:11 +0200
Return-Path: <thomas@arend-rhb.de>
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on k2.arend.tksd
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=AWL,NO_RELAYS autolearn=ham
        version=3.3.1
X-Original-To: kmail2test@example.com
Delivered-To: kmail2test@example.com
Received: by k2.arend.tksd (Postfix, from userid 1000)
        id 06CE14F7F8; Sat, 15 Sep 2012 13:20:11 +0200 (CEST)
Date: Sat, 15 Sep 2012 13:20:11 +0200
To: kmail2test@example.com
Subject: Filter Test 0948
User-Agent: Heirloom mailx 12.2 01/07/07
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-Id: <20120915112042.06CE14F7F8@k2.arend.tksd>
From: thomas@arend-rhb.de (Thomas Arend)
X-UIDL: NKe"!1Q8"!@5?"!fJ(!!

test 0948
--- message 948 ---

I've just upgraded from 4.9 to 4.9.1 and have not lost the destination folders of my move filters, unlike what happened to Thomas (comment #60).
However, my filters were more or less working ok in 4.9, as long I kept them simple, with a single rule in each filter, and did not activate the anti-spam filters.
However, after upgrading to 4.9.1, all filters stopped working automatically, even the ones that only test a single condition and move messages based on that condition.
Now I have to select all messages in my inbox and use CTRL+J to start the filtering manually.

If you add a comment please specify the account type (pop3/imap(online/offline)) and add the filter and akonadi logs.

Created attachment 73964 [details]
Akonadi Log

This is the log file for the message receive 8pop3 account) There where 200 messages send and but I got 230 messages. Three where not filtered correct. The additional 30 result from messages which were twice or more times in the target folder.

Created attachment 73965 [details]
Akonadi Log Information view

See the reports:
AgentBase(akonadi_maildir_resource_0): Die Nachricht „1347816734.R427.x1“ kann nicht verschoben werden.
AgentBase(akonadi_maildir_resource_0): Die Nachricht „1347816816.R722.x1“ kann nicht verschoben werden.

(In reply to comment #62)
> If you add a comment please specify the account type
> (pop3/imap(online/offline)) and add the filter and akonadi logs.

I only use pop3 accounts.

Could you search your email folder for the filenames that failed? Do they actually exist?

3 out of 200 hundred does not seem like a bug affecting most emails but something "special" which happens to a few.

What about those that report that no filters work anymore? Maybe they have some logs which show what causes 3 out of 200 for some and all emails for others not getting filtered.

Comment #61 is about POP3 accounts.
My filters are like this one:

[Filter #0]
Applicability=0
AutomaticName=false
ConfigureShortcut=false
ConfigureToolbar=false
Enabled=true
Icon=system-run
StopProcessingHere=true
ToolbarName=Lista: BovespaBrasil
action-args-0=779
action-name-0=transfer
actions=1
apply-on=check-mail,manual-filtering
contentsA=<BovespaBrasil.yahoogrupos.com.br>
contentsB={BovespaBrasil}
fieldA=List-Id
fieldB=Subject
funcA=contains
funcB=contains
identifier=RktTwRRigV3lRbS7
name=Lista: BovespaBrasil
operator=or
rules=2

If you want, I can post them all here, but they are all very similar to this one and were created right-clicking a message and selecting Create Filter -> Filter on discussion list NAME_OF_THE_LIST.  They do not work automatically when new messages arrive anymore, but they do work if I select all messages in my inbox and use CTRL+J.
I don't know how to see and save akonadi logs. Where do I find them?

Thanks

(In reply to comment #66)
> Could you search your email folder for the filenames that failed? Do they
> actually exist?
> 
> 3 out of 200 hundred does not seem like a bug affecting most emails but
> something "special" which happens to a few.
> 
> What about those that report that no filters work anymore? Maybe they have
> some logs which show what causes 3 out of 200 for some and all emails for
> others not getting filtered.

This test installation is much better than real life. Today I had 10% or more of the real messages not filtered.
What makes it worst: Some deleted messages from Saturday and Sunday reappear. But this is another bug.

I'm not very happy with the idea to provide a full log of the real life messages.

In the log I found an entry saying, that the message could not be moved. That is a true and obvious information. It only gives the hint that the program tried to move but not why it failed to do so. 

What about more detailed information in the filter log to make the task a bit easier..

How can I relate the file names to the mails in the e-mail folder?

Can some body write a script to extract the needed information? This would be helpful to users not so familiar with kmail2 / akonadi.

Messages are being filtered automatically again after a reboot.

This bug appears to be valid in kmail2 4.9.4. I've got multiple mail accounts (POP3), all with their own inbox folder. I've setup a filter which moves certain messages from one of the accounts' inbox into a subfolder of the inbox folder. This filter should trigger on incoming messages, but it fails to do so. It only works manually on 'apply filter'.

Automatic filtering doesn't work on local mbox or maildir accounts.

I set up a filter to move incoming messages to a different folder. When I use mailx to send myself a test message, the message appears in the incoming folder and isn't moved, and nothing shows in the filter log. The filter works fine when applied manually.

Reproducible always. Kmail 4.9.4.

(In reply to comment #71)
> Automatic filtering doesn't work on local mbox or maildir accounts.
> 
> I set up a filter to move incoming messages to a different folder. When I
> use mailx to send myself a test message, the message appears in the incoming
> folder and isn't moved, and nothing shows in the filter log. The filter
> works fine when applied manually.
> 
> Reproducible always. Kmail 4.9.4.

For me this partly right. I received some hundred mail a day (outputs of cron jobs) and spam. Some mails are filtered some are not. At the moment ~10% of the mails are not filtered.

I use Kmail 4.9.4. I would like to admit:
1. Go to "Inbox" (or some other) folder
1b. Make sure you have some messages that match filters
2. Select all mails in the foler (e.g. press Ctrl+a)
3. Run message filters (e.g. press Ctrl+j)

Expected results:
Filters are applied to all selected messages

Actual results:
Filters are applied only to some messages (sometimes some == 0). I mean that not all (selected) messages that  match filters are filtered.

I think I have the same, a similar problem. I have set up several filters to move messages to separate folders. Sometimes an incoming mail is filtered, sometimes not. Applying the filter manually usually helps. This could be annoying. 

Furthermore I don't understand why filtered/moved messages show first up in the inbox folder after all. This was working much better with the old kmail.

(In reply to comment #74)
> I think I have the same, a similar problem. I have set up several filters to
> move messages to separate folders. Sometimes an incoming mail is filtered,
> sometimes not. Applying the filter manually usually helps. This could be
> annoying. 
> 
> Furthermore I don't understand why filtered/moved messages show first up in
> the inbox folder after all. This was working much better with the old kmail.

I confirm that all messages show up first in the inbox. It seem that they are sorted first to the inbox and the filters are applied later. I would expect that an incoming message is parsed by the filters first. I get some hundred messages a day. Parsing first would save a lot processing power.

Showing the messages first in inbox is a limitation of the current architecture. The filters act when messages appear in a certain folder, namely folders marked as inbox. There is no way at the moment to apply filters *before* mails are put in a folder.

bug 311199 explains a method to generate lots of emails for the imap inbox which apparently triggers the "some mails are filtered, some mails are not" bug.

(In reply to comment #77)
> bug 311199 explains a method to generate lots of emails for the imap inbox
> which apparently triggers the "some mails are filtered, some mails are not"
> bug.

I thought I had explained it long ago in another bug. But here is the easy way:

for ((i=0;i<1000;i++;)); doe echo "Test" | mail -s "test $i" you@yourdomain.tdl ; done

Upps it was in this bug some comment above.

Fixed in KDE 4.10.1 and later by Wolfgang Rohdewald.
http://commits.kde.org/kdepim/185c572e079ef40bee2225e582c11dbdbb35840f

(In reply to comment #80)
> Fixed in KDE 4.10.1 and later by Wolfgang Rohdewald.
> http://commits.kde.org/kdepim/185c572e079ef40bee2225e582c11dbdbb35840f

Does not work for me. Using Kubuntu 13.04, KDE 4.10.2.

Setting:
- POP3 Mail account used for mailing lists, mails go to local folder -> inbox on default.
- Filters definded to move mailing list mails into local folders for the dedicated list

Result:
- All mails remain in local folder -> inbox.
- when manual filtering is used (ctrl + j), mails are moved to the defined foldres

Works for me reliably, same configuration (Kubuntu 13.04).

This morning it is working for me as well. Yet, I have not changed anything...

I will have a more detailed look at the "Mail Filter Agent" in the Akonadi Console if filtering stops to work again.

KMail 4.10.3 and still not working here.

Today, filtering was missing again.

Could the issue be based on "undefined" startup bahavour? What if mails are alread fetched and the "mail filter agent" is not already up and running but still starting up?

My computer is on 24/7, so I don’t think this is because of the mail filter agent not being ready yet. Also, this does not happen sometimes to me. No List-ID-based filter has ever worked here on incoming emais (they work when applied manually), to the point where I have modified half of them to work based on the To field instead (which does work), and those I did not change I kept so that I can provide feedback here.

Also, yesterday I edited one of the List-Id-based filters to apply to only selected accounts (and selected my IMAP account), since somewhere in this report it was mentioned that such option (which is by default “apply to all accounts”) might have something to do with this issue. However, today I got mail that should have been filtered by the modified filter, and it wasn’t.

Same here. KMail 4.10.3, I'm trying to apply marks with filters, but also do not see working.

Works for me (also for List-Id) since I originally fixed the bug - all mails are filtered - , so those who have problems should give me more information, like:
- the exact filters (basically the filter rules file with sensitive info replaces by some string, like XXXX)
- detailed description about the account types used

Created attachment 80168 [details]
Filters configuration file.

Attached the content of my “.kde4/share/config/akonadi_mailfilter_agentrc” file, which I guess is what you are asking for. I’m not sure of what other details my be of help.

PS: I didn’t bother to hide information, I sell my privacy cheap ;)

For those still having the problem, can you check in the advanced tab that the filter is enable for ALL accounts? If not, and the filter in question does not work for POP3 you might see bug 308008.

Adrian: none of the List-Id based filters work for you? As I see you are using an IMAP account, is it disconnected imap or online imap?

I guess it is online IMAP, as the “Enable offline mode” option under Advanced is not checked. It does happen with any List-Id-based filter.

I’m actually in a different computer right now, with a different system (my PC is Chakra, I’m currently on a Kubuntu with the KDE SC 4.10.4), and I’ve been able to reproduce teh issue easily. I just added a List-ID-based filter for KDE PIM messages, filtered all mail manually, and it works, but new KDE PIM emails do not get filtered as they are retrieved.

Filtering has been much better for me in the last few releases, but I've hit a one-off problem.

I filter messages from a particular mailing list to its own folder. A little while ago I did a manual Check Mail (Ctrl+L), and two new messages from the list were left in my Inbox. These were correctly filtered with a manual Apply Filter (Ctrl+J).

A few minutes later I again did a manual Check Mail, and two new messages from the list were correctly filtered and moved out of my Inbox.

This is a POP3 account, and the filter looks at List-Id. The filter is applied just to the particular POP3 account (since bug 308008 was fixed).

I guess it would be tricky to reproduce this, as the timing would have to be just right (or wrong).

Following on from my comment 93, now a message has been left in my inbox after an automatic mail check. Again, the message was moved to the correct folder after applying filtering manually.

So this is more than a one-off, but an intermittent bug, and can occur with automatic or manual mail checks. I'm on 4.11.1, using the Kubuntu PPA.

Created attachment 82421 [details]
Filter log showing filtering applied twice to same message

I've hit another problem, which seems to be a timing problem, maybe tickled by the fact that I've now turned on filter logging.

After reading a message in the mailing list I went back to my inbox and saw what turned out to be the same message (I saved the two copies and compared them to make sure). From both the inbox and the other folder, I was able to save the message, but there was only one copy on disk, which I found using grep. This was in ~/.local/share/.local-mail.directory/inbox/new/ (before I opened it).

I've uploaded the relevant part of the filter log (https://bugs.kde.org/attachment.cgi?id=82421). The entries at 08:07 and 08:37 are just for context, to show that I've included all entries at 08:27. The log entry for the first message at 08:27:12 says "Move Into Folder "akonadi-local/london.pm", and indeed that message was visible in that folder through kmail, but as I say, grep showed it was really in inbox. The timestamp of the file (using ls --full-time) is 08:27:12.79. Now see the filter log entry at 08:27:13. This refers to the same message that was filtered one second earlier (I know it's the same because there was only one message sent by that person at 09:07 their local time).

After manually applying filtering to the message, the two (visible) copies were both in akonadi-local/london.pm. Again, there was only one version actually on disk.

About an hour later, the same thing happened (same mailing list; different author).

Same problem with kmail 4.11.4 on mageia 4.

Same here, 4.12.3 on Ubuntu ( Ubuntu + KDE ppa backports) 12.04.3

Still happening in 4.13. I think this is the same as bug 293918.

kmail 4.8.5, using KDE development Platform 4.8.5, installed by apt-get

A simple mail filter, (anywhere in the headers contains 'some string'), move into a folder refuses to work.  I cleaned up the filter logging yesterday pm.  A mail (which should have been filtered) was sent around 00:07 (gmt+2).  The computer was online at that time..

went to sleep, awakened the computer around 11:00.  Around 15:30, still seeing the mail in the inbox) i looked at the filter log (everything enabled, no limits).  Nothing done since yesterday !

I selected all messages in the inbox, requesting to apply the filters, and the filter was applied (as usual when i request manual filtering).

Any idea ?

I downloaded linux mint kde 17.1 64b and kmail is not applying rules to the incoming mail.

Will this ever be fixed? Its important for a corporate enviroment.

Thank you.

Kmail filters dont work in incoming mail

kmail-4.14.8

Hello,
Kmail2; 4.14.8; Linux (x86_64) release 3.16.7-21-desktop; cc

I have about 7 imap account, and a good 30 filters.

Ctrl+J works when I select 1 email already in my inbox.

But it doesn't work on incoming emails ....

I confirm this bug with KMail 4.14.10 on Fedora 23.

maybe related: https://bugs.kde.org/show_bug.cgi?id=358041

Maybe the importance should be increased as this can lead to data loss.

Modifying the filter rule seems to fix it for a couple of days.

When kmail (5.1.2) is started after login (Leap 42.1), no filter is applied and all new emails remain in the inbox folder (very annoying). Even manual filtering and Ctrl+J don't work. My standard procedure then is to just open the filter settings dialog (Einstellungen|Filter einrichten...) and close it immediately with Ok, i.e. without changing anything. From then on filtering works as expected. It seems like the whole filtering mechanism is only started upon firing up the configuration dialog. Couldn't find this in the previous comments, so maybe it's a clue for the developers.

May be related: https://bugs.kde.org/show_bug.cgi?id=362599

Metko exactly describes  the behavior here.

Hello, I also can confirm the behavior described by Metko. Any update on this issue? Anything I can provide from my side? 
Thanks!

I just came across this.

Normally it was enough to fire up akonadiconsole, restart the mail filter agent, and then quickly start kontact.

Not today.

When, for the umpteenth time, the mail filter was stuck at 0%, I opened the akonadiconsole, and selected "Abort Activity" on the mail filter agent.
And it immediately began to filter the 1,046 mails I had selected.

So it seems, that the filter agent got ordered to do something else before answering my request and got stuck on that, and not the requested filtering itself.

It is slow right now, because I had activated the filter log, but at least it works again.

Update on my previous report:

After the mail filter finished, it got stuck at 100%. The akonadi_mailfilter_agent process used 100% CPU, and was neither interruptible nor stoppable using akonadiconsole.

I quit kontact, shut down akonadi and even logged out and back into plasma, but the process remained until I killed it.

After that it worked just fine yesterday.

Today I wanted to filter some mails and it just started at 100% and was stuck again until the process got killed.

I tried several times, with kontact and akonadi restarting in between, but always ended up killing the akonadi_mailfilter_agent.

Then, all of a sudden, a myriad of status windows popped up, all with the filter being at 100%, but eventually it started its work regularly.

If I had to guess, I'd think the filter has at least one race condition and a problematic endless loop somewhere. Didn't look up the sources, though...

I had the same problem with OpenSuSE 13.2 KDE 4.14, and also remembering it came after installing some updates.
Now after removing some update packages and downgraded to the following original packages, filtering is working again:

libakonadiprotocolinternals1|1.13.0-2.2.3
python-kde4|4.14.2-1.1
libakonadi4|4.14.2-1.1
akonadi-runtime|1.13.0-2.2.3
python-kde4-plasma|4.14.2-1.1
python-kde4-phonon|4.14.2-1.1
python-kde4-knewstuff|4.14.2-1.1
python-kde4-khtml|4.14.2-1.1
libkdepimlibs4|4.14.2-1.1
python-kde4-akonadi|4.14.2-1.1
libkdepim4|4.14.2-1.1
kdepim4-runtime|4.14.2-1.1
kalarm|4.14.2-1.1
akregator|4.14.2-1.1
plasma-addons-akonadi|4.14.2-1.1
akonadi|4.14.2-1.1
kdepim4|4.14.2-1.1
korganizer|4.14.2-1.1
kontact|4.14.2-1.1
knotes|4.14.2-1.1
kmail|4.14.2-1.1
kaddressbook|4.14.2-1.1

Very similar here, after rebooting the computer, filtering consistently does nothing.  Restarting the Akonadi server always makes filtering work again.

Debian testing here, Kontact, KMail and Akonadi are all at version 16.04.3-2.

(In reply to quazgar from comment #116)
> Debian testing here, Kontact, KMail and Akonadi are all at version 16.04.3-2.

Sorry, that was the Debian version, of course.  Kontact says it's 5.2.3, Akonadi is at 5.2.2.