Bug 283682 - KMail duplicates filtered messages
Summary: KMail duplicates filtered messages
Status: REOPENED
Alias: None
Product: Akonadi
Classification: Frameworks and Libraries
Component: Mail Filter Agent (show other bugs)
Version: 4.12
Platform: OpenSUSE Linux
: NOR normal with 507 votes (vote)
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
: 282296 286053 293918 316419 322126 322676 323980 333035 346695 362398 377344 (view as bug list)
Depends on:
Blocks:
 
Reported: 2011-10-10 05:57 UTC by Pavel Baranchikov
Modified: 2022-08-24 07:26 UTC (History)
74 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.9.3


Attachments
imap log, a duplicate message is fetched (86.47 KB, text/plain)
2012-10-14 10:13 UTC, Anders Lund
Details
filter log, a duplicated message is fetched (12.33 KB, text/html)
2012-10-14 10:15 UTC, Anders Lund
Details
Akonadi Console Notification Monitor (178.37 KB, image/png)
2014-03-29 14:31 UTC, Marcus
Details
akonadi log all (3.10 MB, text/plain)
2014-08-26 01:04 UTC, Sylvain
Details
akonadi log simple (6.11 KB, text/plain)
2014-08-26 01:04 UTC, Sylvain
Details
akonadi maildir resource (4.97 KB, text/plain)
2014-08-26 01:05 UTC, Sylvain
Details
akonadi mailfilter agent (44.44 KB, text/plain)
2014-08-26 01:05 UTC, Sylvain
Details
new mailnotifier agent (18.72 KB, text/plain)
2014-08-26 01:06 UTC, Sylvain
Details
mail filter kernel ETM (21.28 KB, text/plain)
2014-08-26 01:06 UTC, Sylvain
Details
kmail filter log applying the bogofilter pipe (556 bytes, text/plain)
2014-08-26 01:07 UTC, Sylvain
Details
Filter <unknown> after removing all filters manually - in this state it works (55.90 KB, image/png)
2015-09-24 12:43 UTC, René Krell
Details
signature.asc (818 bytes, application/pgp-signature)
2015-09-25 17:24 UTC, Eike Sauer
Details
attachment-14215-0.dat (45 bytes, multipart/alternative)
2015-09-30 05:22 UTC, JDiel
Details
duplicateMailExample.png (45.63 KB, image/png)
2015-09-30 05:22 UTC, JDiel
Details
Duplicated mails in Kmail 5.2.3 (247.94 KB, image/png)
2017-09-01 23:46 UTC, Alexandre Bonneau
Details
signature.asc (833 bytes, application/pgp-signature)
2022-04-27 09:08 UTC, Christian Hilberg
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Pavel Baranchikov 2011-10-10 05:57:53 UTC
Version:           4.7 (using Devel) 
OS:                Linux

I have a Jira account. I have set up 2 filters: one is used to move messages for JIRA folder and mark it as read. Filtered by sender (my own activieties).
The second one is used to move all other JIRA maessages to the same folder, but do not mark them as read.

As a result, I have a lot of duplicated messages in my JIRA folder. Some messages (during 2-3 weeks) became duplicated to 10 times.

Reproducible: Sometimes

Steps to Reproduce:
1. create some filters, as I mentioned.
2. try to get the mail.

Actual Results:  
New messages arrive in the INBOX folder (that are already placed to the JIRA folder). After this, KMail starts to filter them and send the duplicates to JIRA folder again.

Expected Results:  
Messages should be placed to the folder. No duplicating.

As I remember, one of e-mail clients, used by mine, has a function 'remove duplicates'. This should be very useful in this case.
Comment 1 Vladimir Kolesnikov 2011-10-30 20:58:38 UTC
Could be the same bug: https://bugs.launchpad.net/ubuntu/+source/kdepim/+bug/883459
Comment 2 Laurent Montel 2011-11-08 07:35:21 UTC
*** Bug 286053 has been marked as a duplicate of this bug. ***
Comment 3 Geoffrey Clements 2011-12-01 10:53:49 UTC
I have the same problem, nearly all messages in my inbox are duplicated. I use only IMAP so don't know if this affects pop3 accounts. I do have filtering enabled.
Comment 4 rick 2011-12-09 02:49:45 UTC
Running Gentoo stable, just pulled in kmail2.  my pop account is showing multiple copies of messages that have been moved to another folder by a filter rule.  Also, when I click on the Inbox messages that had been moved by filter suddenly appear in the inbox, then are moved back into the the folder specified by the filter rule (and thus the duplicate).
Also checking email will cause the filtered messages to duplicate.
This is not happening with my IMAP accounts (though none of my filters affect messages that typically come in through the accounts I access via IMAP).

Happy hunting.
Comment 5 rick 2011-12-09 03:55:45 UTC
I exported all my filters, and deleted them from KMail2.  KMail2 is now running without filters.  If I move an email from the inbox to another folder (ie, the "destination" folder), the email will appear again in the inbox (and remain in the destination folder as well).  This occurs both when the destination folder is a sub-folder of the inbox, and when the destination folder is directly under the Local Folders.
Comment 6 Szymon Stefanek 2011-12-27 15:34:23 UTC
I experience this problem in 4.7.90 (4.8 beta2).

If a mail just downloaded is moved by a filter to a different folder then there is a good chance that it will re-appear again in the original folder at the next update

It does not happen with all the accounts or all the filtered messages. However if a message is affected then it's 100% reproduceable: it will be duplicated at every folder update (forever?).
Comment 7 ggrau 2012-01-26 07:47:06 UTC
Same here... with 4.7.4 on Fedora16
Most mails I try to move from my Inbox will appear again as duplicate.
This happens also, when moved by filter rule.
Pressing <Del> to move them into trash surprisingly seems to work.
Comment 8 Matthias Nagel 2012-01-31 10:35:48 UTC
Same problem for me. I am using kmail 4.7.4-r1 from the Gentoo Stable amd64 repository. I have to POP3 accounts and some filters that examine the mail header (mostly the "from" attribute) and move the mails into folders and marks them as read. Sometimes new messages are duplicated. This also affects mails that are passed through the filter but not modified, because the filter condition does not apply.

The mails are not duplicated immediately but reemerge after a folder update, especially after starting kmail. Suddenly mails that are already some days old and have already been processed by the filter show up again.

I can observe the same behaviour as in comment #6. Not every mail that is processed by some filter is affected, but if some mail is affected once, the mail reappears over and over again. I have the feeling that a mail is more likely to be affected if the filter has to process a lot of new mails at once.
Comment 9 Tobias Koenig 2012-02-11 23:38:19 UTC
Hej all,

can you please check whether akonadi reports some errors regarding the target folders you try to move the mails to, about a missing remote id?
You can also use akonadiconsole and check the folder properties on the 'internal' tab to find out the remote id.
Comment 10 Matthias Mailänder 2012-02-12 13:39:17 UTC
Found this while trying to reproduce and understand akonadiconsole logs:

AgentBase(akonadi_imap_resource_0): Select failed, server replied: A000068 NO Mailbox doesn't exist: &APw-lleimer 

The local Trash folder is called "Lokale Ordner/Mülleimer". This is where the spam check wants to move spam. Putting it on the IMAP Junk folder as Thunderbird does is not possible/allowed.
Comment 11 Geoffrey Clements 2012-02-12 15:59:22 UTC
(In reply to comment #9)
> Hej all,
> 
> can you please check whether akonadi reports some errors regarding the target
> folders you try to move the mails to, about a missing remote id?
> You can also use akonadiconsole and check the folder properties on the
> 'internal' tab to find out the remote id.

Tobias,

Sorry this will sound lame but I'm not sure where to look for the akonadi error log. I did try this:

$ grep -R remote ~/.local/share/akonadi/
grep: /home/geoff/.local/share/akonadi/socket-dhcppc2: No such file or directory
grep: /home/geoff/.local/share/akonadi/socket-dhcppc1: No such file or directory
Binary file /home/geoff/.local/share/akonadi/db_data/ibdata1 matches
Binary file /home/geoff/.local/share/akonadi/db_data/akonadi/pimitemtable.frm matches
Binary file /home/geoff/.local/share/akonadi/db_data/akonadi/collectiontable.frm matches
Binary file /home/geoff/.local/share/akonadi/db_data/ib_logfile0 matches
grep: /home/geoff/.local/share/akonadi/akonadiserver.socket: No such device or address
grep: /home/geoff/.local/share/akonadi/socket-scotty/akonadiserver.socket: No such device or address
grep: /home/geoff/.local/share/akonadi/socket-scotty/mysql.socket: No such device or address

As the only matches are in binary files I didn't take this any further.

Using akonadiconsole there is nothing at all displayed in the browser->internals tab. I do notice that non of the remote IMAP folders are displayed in the panel on the left hand side. I see only:

Personal Contacts
Local Folders
    |
     - Outbox
Search
Comment 12 ggrau 2012-02-12 18:54:42 UTC
All I can see when moving a Mail is:

==================================
NotificationManager::notify ( Item (129274, 1329071216.R951.neptun:2,S) from collection 467 movedto collection 535 )
==================================

When rescanning the derectory, I just moved it out with F5:
========================================
NotificationManager::notify ( Item (129275, 1329071216.R951.neptun:2,S) in collection 467 added )
NotificationManager::notify ( Item (129275, 1329071216.R951.neptun:2,S) in collection 467 modified parts (PLD:RFC822) )
=======================================

There is a lot of debug traffic in the upper window, but (as I can see) nothin g suspecious.
Comment 13 Milian Wolff 2012-02-23 13:36:09 UTC
Tobias, please give us more information on what exactly we should look at and post here.
Comment 14 Milian Wolff 2012-02-24 16:02:23 UTC
rightclick on (some) of my target folders in akonadiconsole shows that they all have a "remote identifier" set.

I note though, that the situation has improved, I don't get as many duplicated emails anymore as I used to before. I'll see whehter I can reproduce it at all, maybe this is actually fixed in recent git?
Comment 15 Christian Trippe 2012-02-24 18:46:27 UTC
*** Bug 282296 has been marked as a duplicate of this bug. ***
Comment 16 Geoffrey Clements 2012-02-25 09:43:42 UTC
(In reply to comment #9)
> Hej all,
> 
> can you please check whether akonadi reports some errors regarding the target
> folders you try to move the mails to, about a missing remote id?
> You can also use akonadiconsole and check the folder properties on the
> 'internal' tab to find out the remote id.

Tobias,

I have checked a number of times now in akonadiconsole, the debugger does not show any error messages when the messages are duplicated. Please note however that the duplicated messages are the ones that are /not/ moved because they do not meet any filter criteria. The messages that are moved by the filters are not duplicated.
Comment 17 Matthias Nagel 2012-02-25 10:02:23 UTC
Opposed to comment #16 the mails that are moved are duplicated with me.

Even mails that are not moved by a filter but that are move manually (drag and drop) are duplicated. Mails that are "moved" to the bin folder by hitting "DEL" are duplicated, too. In all three cases the mail re-appears in the original folder after some period of time. The problem can be avoid by copying the mail to the destination folder and then purge the original mail from the source folder by hitting "Shift+DEL". This work-around also applies to the filter. If I create two filter actions - copy mail first, then purge the original - everything is fine.

And in all three cases there is no error report in the akonadiconsole.
Comment 18 Milian Wolff 2012-02-25 16:01:29 UTC
reopening, as I also just noticed this again. The target folder has a proper remote identifier set, yet I saw that some mails where duplicated by the filter manager...

Tobias, any other ideas what I could look into?
Comment 19 Alexandre Bonneau 2012-03-30 19:59:39 UTC
This nasty bugs affect 4.8.1 too.
Comment 20 Simon Reed 2012-04-10 15:56:55 UTC
I believe the problem I have in 4.8.4 is the same problem: emails passing through filters are sometimes being duplicated.

I have a cron job on my servers which send me an "hourly test email" message which I receive on my desktop PC; these have been running for about two years without trouble.  I have upgraded my desktop from Ubuntu 10.04 and kmail to Xubuntu 64-bit 11.10 and kmail v4.7.3 on my desktop and I am now seeing some of these hourly emails as identical duplicates (same ids in the headers, etc.) in their folders in kmail2.

For example, my PC was unattended for 11 hours last night and during that time 3 of the hourly emails appear as duplicates in their folder.  

The emails are fetched by kmail2 using POP3 and filed into mbox folders by filters.  When using kmail I had it set up to remove emails from servers; I set kmail2 to leave it on the servers (just in case I reverted back).  On the server I can see there are no duplicates so they were not sent as duplicates.

The rule checks the subject of the email contains "Hourly Test Email" and the "Envelope-to" equals the email address I specified for this sending server.  it then moves the email to the correct folder and processing rules is then stopped.

These email checking filters are applied before "Bogofilter check", "Spam Handling" "Classify as Spam" and "Classify as NOT spam" filters, so they are all probably innocent.

The email was downloaded and processed by "interval email checking".

To reproduce it:
1. Set up a filter to receiving incoming email and automatically file it in a folder.
2. Send lots of emails to it.
3. Check the folder.

I just did this by manually sending 18 emails; one of them in the middle was duplicated.

I set up a script to send an email every 5 seconds and sent 77 emails.  79 arrived; two had been duplicated.

The package I have installed is: kmail 4:4.7.4+git111222-0ubuntu0.1
"About KMail" says "KMail Version 4.7.3, Using KDE Development Platform 4.7.4 (4.7.4)"
Comment 21 Alexandre Bonneau 2012-04-10 18:35:19 UTC
Just for information, the bug is still seen in 4.8.2.
Comment 22 PhobosK 2012-06-25 21:56:46 UTC
Affects heavily KDE/KMail 4.8.4 too...
The problem is definitely in moving messages by an applied filter. Manual filtering doesn't have it.
The affected messages are marked for deletion on the remote server (IMAP in my case), deleted from the KMail folder and then reappearing back in the same Kmail folder and in the remote folder too. So at some point in time, the remote folder has the marked for deletion messages + one fresh new and duplicate of all marked for deletion. The strange thing is that some of the duplicates have only headers and not the full messages.
So the problem seems to be in the Move action function of the filtering rules.
Comment 23 András Manţia 2012-07-04 15:36:10 UTC
Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'master'.

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364

M  +7    -11   kmail/kmcommands.cpp
M  +10   -7    kmail/kmcommands.h
M  +1    -1    kmail/kmmainwidget.cpp
M  +1    -5    mailcommon/filter/filteraction.cpp
M  +6    -5    mailcommon/filter/filteraction.h
M  +6    -0    mailcommon/filter/filteractionaddheader.cpp
M  +2    -0    mailcommon/filter/filteractionaddheader.h
M  +2    -2    mailcommon/filter/filteractionaddtag.cpp
M  +1    -1    mailcommon/filter/filteractionaddtag.h
M  +9    -3    mailcommon/filter/filteractionaddtoaddressbook.cpp
M  +2    -0    mailcommon/filter/filteractionaddtoaddressbook.h
M  +7    -0    mailcommon/filter/filteractionbeep.cpp
M  +1    -0    mailcommon/filter/filteractionbeep.h
M  +18   -1    mailcommon/filter/filteractioncopy.cpp
M  +7    -0    mailcommon/filter/filteractioncopy.h
M  +6    -0    mailcommon/filter/filteractiondelete.cpp
M  +1    -0    mailcommon/filter/filteractiondelete.h
M  +5    -0    mailcommon/filter/filteractionexec.cpp
M  +1    -0    mailcommon/filter/filteractionexec.h
M  +5    -0    mailcommon/filter/filteractionforward.cpp
M  +1    -0    mailcommon/filter/filteractionforward.h
M  +6    -0    mailcommon/filter/filteractionmove.cpp
M  +2    -0    mailcommon/filter/filteractionmove.h
M  +6    -0    mailcommon/filter/filteractionpipethrough.cpp
M  +1    -0    mailcommon/filter/filteractionpipethrough.h
M  +6    -0    mailcommon/filter/filteractionplaysound.cpp
M  +2    -0    mailcommon/filter/filteractionplaysound.h
M  +5    -0    mailcommon/filter/filteractionredirect.cpp
M  +1    -0    mailcommon/filter/filteractionredirect.h
M  +6    -1    mailcommon/filter/filteractionremoveheader.cpp
M  +1    -0    mailcommon/filter/filteractionremoveheader.h
M  +6    -0    mailcommon/filter/filteractionreplyto.cpp
M  +1    -0    mailcommon/filter/filteractionreplyto.h
M  +6    -0    mailcommon/filter/filteractionrewriteheader.cpp
M  +1    -0    mailcommon/filter/filteractionrewriteheader.h
M  +6    -0    mailcommon/filter/filteractionsendfakedisposition.cpp
M  +1    -0    mailcommon/filter/filteractionsendfakedisposition.h
M  +8    -2    mailcommon/filter/filteractionsendreceipt.cpp
M  +1    -0    mailcommon/filter/filteractionsendreceipt.h
M  +6    -0    mailcommon/filter/filteractionsetidentity.cpp
M  +1    -0    mailcommon/filter/filteractionsetidentity.h
M  +6    -0    mailcommon/filter/filteractionsetstatus.cpp
M  +1    -0    mailcommon/filter/filteractionsetstatus.h
M  +6    -0    mailcommon/filter/filteractionsettransport.cpp
M  +1    -0    mailcommon/filter/filteractionsettransport.h
M  +5    -0    mailcommon/filter/filteractionstatus.cpp
M  +2    -0    mailcommon/filter/filteractionstatus.h
M  +5    -2    mailcommon/filter/filteractionunsetstatus.cpp
M  +2    -1    mailcommon/filter/filteractionunsetstatus.h
M  +4    -4    mailcommon/filter/filterimporter/filterimporterevolution.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimporterprocmail.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimportersylpheed.cpp
M  +2    -2    mailcommon/filter/filtermanager.cpp
M  +2    -8    mailcommon/filter/filtermanager.h
M  +8    -2    mailcommon/filter/itemcontext.cpp
M  +8    -1    mailcommon/filter/itemcontext.h
M  +10   -17   mailcommon/filter/kmfilterdialog.cpp
M  +2    -1    mailcommon/filter/kmfilterdialog.h
M  +25   -8    mailcommon/filter/mailfilter.cpp
M  +4    -3    mailcommon/filter/mailfilter.h
M  +48   -20   mailcommon/searchpattern.cpp
M  +34   -11   mailcommon/searchpattern.h
M  +1    -1    mailcommon/tests/searchpatterntest.cpp
M  +78   -47   mailfilteragent/filtermanager.cpp
M  +10   -8    mailfilteragent/filtermanager.h
M  +8    -6    mailfilteragent/mailfilteragent.cpp
M  +3    -0    mailfilteragent/mailfilteragent.h

http://commits.kde.org/kdepim/d64d61b470c56b92294f6adee6d74305a217628d
Comment 24 Aaron Williams 2012-07-05 05:46:40 UTC
I really hope this does get backported. The duplicate emails is a deal-killer for me and I'd LOVE to dump Thunderbird.
Comment 25 András Manţia 2012-07-05 11:40:44 UTC
Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'KDE/4.9'.

(Backport filter fix from master)

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364
(cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d)

FIXED-IN: 4.9

M  +7    -11   kmail/kmcommands.cpp
M  +10   -7    kmail/kmcommands.h
M  +1    -1    kmail/kmmainwidget.cpp
M  +1    -5    mailcommon/filter/filteraction.cpp
M  +6    -5    mailcommon/filter/filteraction.h
M  +6    -0    mailcommon/filter/filteractionaddheader.cpp
M  +2    -0    mailcommon/filter/filteractionaddheader.h
M  +2    -2    mailcommon/filter/filteractionaddtag.cpp
M  +1    -1    mailcommon/filter/filteractionaddtag.h
M  +9    -3    mailcommon/filter/filteractionaddtoaddressbook.cpp
M  +2    -0    mailcommon/filter/filteractionaddtoaddressbook.h
M  +7    -0    mailcommon/filter/filteractionbeep.cpp
M  +1    -0    mailcommon/filter/filteractionbeep.h
M  +18   -1    mailcommon/filter/filteractioncopy.cpp
M  +7    -0    mailcommon/filter/filteractioncopy.h
M  +6    -0    mailcommon/filter/filteractiondelete.cpp
M  +1    -0    mailcommon/filter/filteractiondelete.h
M  +5    -0    mailcommon/filter/filteractionexec.cpp
M  +1    -0    mailcommon/filter/filteractionexec.h
M  +5    -0    mailcommon/filter/filteractionforward.cpp
M  +1    -0    mailcommon/filter/filteractionforward.h
M  +6    -0    mailcommon/filter/filteractionmove.cpp
M  +2    -0    mailcommon/filter/filteractionmove.h
M  +6    -0    mailcommon/filter/filteractionpipethrough.cpp
M  +1    -0    mailcommon/filter/filteractionpipethrough.h
M  +6    -0    mailcommon/filter/filteractionplaysound.cpp
M  +2    -0    mailcommon/filter/filteractionplaysound.h
M  +5    -0    mailcommon/filter/filteractionredirect.cpp
M  +1    -0    mailcommon/filter/filteractionredirect.h
M  +6    -1    mailcommon/filter/filteractionremoveheader.cpp
M  +1    -0    mailcommon/filter/filteractionremoveheader.h
M  +6    -0    mailcommon/filter/filteractionreplyto.cpp
M  +1    -0    mailcommon/filter/filteractionreplyto.h
M  +6    -0    mailcommon/filter/filteractionrewriteheader.cpp
M  +1    -0    mailcommon/filter/filteractionrewriteheader.h
M  +6    -0    mailcommon/filter/filteractionsendfakedisposition.cpp
M  +1    -0    mailcommon/filter/filteractionsendfakedisposition.h
M  +8    -2    mailcommon/filter/filteractionsendreceipt.cpp
M  +1    -0    mailcommon/filter/filteractionsendreceipt.h
M  +6    -0    mailcommon/filter/filteractionsetidentity.cpp
M  +1    -0    mailcommon/filter/filteractionsetidentity.h
M  +6    -0    mailcommon/filter/filteractionsetstatus.cpp
M  +1    -0    mailcommon/filter/filteractionsetstatus.h
M  +6    -0    mailcommon/filter/filteractionsettransport.cpp
M  +1    -0    mailcommon/filter/filteractionsettransport.h
M  +5    -0    mailcommon/filter/filteractionstatus.cpp
M  +2    -0    mailcommon/filter/filteractionstatus.h
M  +5    -2    mailcommon/filter/filteractionunsetstatus.cpp
M  +2    -1    mailcommon/filter/filteractionunsetstatus.h
M  +4    -4    mailcommon/filter/filterimporter/filterimporterevolution.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimporterprocmail.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimportersylpheed.cpp
M  +2    -2    mailcommon/filter/filtermanager.cpp
M  +2    -8    mailcommon/filter/filtermanager.h
M  +8    -2    mailcommon/filter/itemcontext.cpp
M  +8    -1    mailcommon/filter/itemcontext.h
M  +10   -17   mailcommon/filter/kmfilterdialog.cpp
M  +2    -1    mailcommon/filter/kmfilterdialog.h
M  +25   -8    mailcommon/filter/mailfilter.cpp
M  +4    -3    mailcommon/filter/mailfilter.h
M  +48   -20   mailcommon/searchpattern.cpp
M  +34   -11   mailcommon/searchpattern.h
M  +1    -1    mailcommon/tests/searchpatterntest.cpp
M  +78   -47   mailfilteragent/filtermanager.cpp
M  +10   -8    mailfilteragent/filtermanager.h
M  +8    -6    mailfilteragent/mailfilteragent.cpp
M  +3    -0    mailfilteragent/mailfilteragent.h

http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69
Comment 26 Alexandre Bonneau 2012-07-05 15:04:00 UTC
Nice job !
/me is eager to try 4.9
Comment 27 András Manţia 2012-07-06 09:13:44 UTC
Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia.
Committed on 06/07/2012 at 11:12.
Pushed by amantia into branch 'KDE/4.8'.

Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport.
An unwanted side-effect might be that more data is downloaded than before,
based on the type of filters used.

FIXED-IN: 4.8.5
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364

M  +0    -1    mailfilteragent/mailfilteragent.cpp

http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec
Comment 28 Christian Trippe 2012-07-18 17:51:25 UTC
First, thanks for working on the filter problems!

Using KDE 4.8 with the minimal 4.8 patch I sometimes still see duplicated messages, but much less then before. And it looks like the duplicates now always have content.

I'll report back how 4.9 works for me.
Comment 29 Marcus Gama 2012-08-22 23:18:22 UTC
I'm using KDE 4.9 and, at least for me, the issue was not fixed. I have duplicated messages in my Inbox.
Comment 30 Aaron Williams 2012-08-22 23:22:49 UTC
It is even worse for me. Somehow it's mangling my email and nuking the body, leaving only the header in place. I had to nuke Akonadi.
Comment 31 Alexandre Bonneau 2012-09-05 18:33:48 UTC
This issue is indeed still affecting KDE 4.9.
It's worse as it happens much more than before, and instead of duplicating approximatively 12 times each mail, it seems it's more around 16 times now.

This is a killer for Kmail..
Comment 32 S. Burmeister 2012-09-11 09:58:39 UTC
This only happens for me to messages which get marked as spam by bogofilter, i.e. piped through an external app and marked as spam.

The spam filtering, i.e. piping filter, is the last one in the list. All messages that do not reach that filter but are just filtered on List-ID etc. do not re-appear.
Comment 33 Alexandre Bonneau 2012-09-11 22:18:45 UTC
Duplicate mails no only happens when you go into a folder containing unread mails.
For instance if you have 2 unread mail in folder 'mymails', when you select that folder the unread mails are duplicated 'live' as ~32 unread mails.

I do use bogofilter, but those duplicated mails aren't marked as spam.
Comment 34 S. Burmeister 2012-09-12 00:08:03 UTC
But they are piped through bogofilter before they get moved?

Without logs and debug output no dev will be able to understand what the issue is. Not mentioning the need to reproduce it in order to know what to fix.

So those that experience this should get that info into the report.

Please use 4.9.1 for testing since filtering did receive a lot of fixes and findings with 4.8 are a lot less likely to help debugging and hence fixing.

You can get debugging by enabling it for akonadi etc. in kdebugdialog and restarting akonadi in a konsole with akonadictl restart. akonadiconsole does also offer possibilities to get debug info.

In KMail > Tools/Extra you can find a possibility to log filtering.

Starting akonaditray might also be handy.

For imap see: http://techbase.kde.org/Projects/PIM/Akonadi/Debug_IMAP
Comment 35 Alexandre Bonneau 2012-09-17 16:14:04 UTC
I used the filter logging tools as pointed, and when opening a folder containing unread mails, they get duplicated, but nothing shows in the log.

Which components should we activate in kdebugdialog, kmail and all akonadi* entries ?
Comment 36 Alexandre Bonneau 2012-09-17 16:27:39 UTC
This bug is still affecting 4.9.1 by the way (4:4.9.1-0ubuntu1~ubuntu12.04.1~ppa2).
Comment 37 S. Burmeister 2012-09-18 07:00:18 UTC
(In reply to comment #35)
> I used the filter logging tools as pointed, and when opening a folder
> containing unread mails, they get duplicated, but nothing shows in the log.

AFAIK filtering is not applied to unread emails but just to new emails, i.e. "just arrived" in the inbox. However, duplication taking place when switching folders would hint towards filtering not being the cause of the duplication.

> Which components should we activate in kdebugdialog, kmail and all akonadi*
> entries ?

Yes. To see that output issue akonadictl restart from a konsole and/or watch the general log file with tail -f ~/.xsession-errors
Comment 38 Alexandre Bonneau 2012-09-26 16:56:09 UTC
I activated all akonadi* and kmail occurence in kdebugdialog, then restarted akonadi in a console.

Browsing like always in folders containing unread mails, I sometimes see :
"QStringList Akonadi::NepomukSearch::search(const QString&) Calling blockingQuery() failed! 
QStringList Akonadi::NepomukSearch::search(const QString&) Calling blockingQuery() failed!", but it doesn't necessarily duplicates unread mails.
That doesn't help I guess, but I can't really see any special debug lines tied specifically to the problem when it appears in kmail..

I do get the usual " AkonadiAgentServer(14273) MaildirResource::itemRemoved: Item removed 379717  in collection : 625" repeated over and over when I use the function to delete duplicated mails.

~/.xsession-errors on the other hand is much more verbose. When entering a folder and seeing the duplicate bug happening, I get this :
kmail2(15566) MessageList::StorageModel::StorageModel: Using model: Akonadi::EntityTreeModel
kmail2(15566) Nepomuk::ResourceWatcher::start: Failed to connect to watch service "The name org.kde.nepomuk.DataManagement was not provided by any .service files"
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) MessageList::Core::View::slotSelectionChanged: View message selected [ " Debian Project News - September 17th, 2012" ]
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done
kmail2(15566) KMReaderWin::setMessage: void KMReaderWin::setMessage(const Akonadi::Item&, MessageViewer::Viewer::UpdateMode) QSplitter(0x1c79610, name = "splitter2")
kmail2(15566) makeShowAuditLogLink: not showing link (not implemented)
kmail2(15566) KEmoticonsTheme::tokenize: Broken HTML entity, trying to recover.
...repeating 50 times...
kmail2(15566) KEmoticonsTheme::tokenize: Broken HTML entity, trying to recover.
kmail2(15566) MailCommon::SendMdnHandler::Private::handleMessages: AKONADI PORT: Disabled code in   void MailCommon::SendMdnHandler::Private::handleMessages()
kmail2(15566) makeShowAuditLogLink: not showing link (not implemented)
kmail2(15566) KEmoticonsTheme::tokenize: Broken HTML entity, trying to recover.
kmail2(15566) KEmoticonsTheme::tokenize: Broken HTML entity, trying to recover.
...repeating 50 times...
kmail2(15566) KEmoticonsTheme::tokenize: Broken HTML entity, trying to recover.
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done"

sometimes I see that :
"kmail2(15566) MessageList::StorageModel::StorageModel: Using model: Akonadi::EntityTreeModel
kmail2(15566) Nepomuk::ResourceWatcher::start: Failed to connect to watch service "The name org.kde.nepomuk.DataManagement was not provided by any .service files"
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done
kmail2(15566) MessageList::Core::View::slotSelectionChanged: View message selected [ "Upgrade to the New SourceForge; PeaZip Is POTM; Developers Wanted" ]
kmail2(15566) KMReaderWin::setMessage: void KMReaderWin::setMessage(const Akonadi::Item&, MessageViewer::Viewer::UpdateMode) QSplitter(0x1c79610, name = "splitter2")
kmail2(15566) MailCommon::SendMdnHandler::Private::handleMessages: AKONADI PORT: Disabled code in   void MailCommon::SendMdnHandler::Private::handleMessages()
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done"

and sometimes that :
"kmail2(15566) KMReaderWin::setMessage: void KMReaderWin::setMessage(const Akonadi::Item&, MessageViewer::Viewer::UpdateMode) QSplitter(0x1c79610, name = "splitter2")
kmail2(15566) MailCommon::SendMdnHandler::Private::handleMessages: AKONADI PORT: Disabled code in   void MailCommon::SendMdnHandler::Private::handleMessages()
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done
kmail2(15566) MessageList::StorageModel::StorageModel: Using model: Akonadi::EntityTreeModel
kmail2(15566) Nepomuk::ResourceWatcher::start: Failed to connect to watch service "The name org.kde.nepomuk.DataManagement was not provided by any .service files"
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) MessageList::Core::View::slotSelectionChanged: View message selected [ "[Bug 283682] KMail duplicates filtered messages" ]
"/usr/bin/kmail(15566)" Soprano: "Could not connect to server at /tmp/ksocket-myuser/nepomuk-socket (Aucun fichier ou dossier de ce type)"
"/usr/bin/kmail(15566)" Soprano: "Could not connect to server at /tmp/ksocket-myuser/nepomuk-socket (Aucun fichier ou dossier de ce type)"
"/usr/bin/kmail(15566)" Soprano: "Unsupported operation (2)": "Invalid model"
"/usr/bin/kmail(15566)" Soprano: "Unsupported operation (2)": "Invalid model"
"/usr/bin/kmail(15566)" Soprano: "Invalid iterator."
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 2 (X_ChangeWindowAttributes)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 2 (X_ChangeWindowAttributes)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 2 (X_ChangeWindowAttributes)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x1000658
"/usr/bin/kmail(15566)" Soprano: "Could not connect to server at /tmp/ksocket-myuser/nepomuk-socket (Aucun fichier ou dossier de ce type)"
"/usr/bin/kmail(15566)" Soprano: "Could not connect to server at /tmp/ksocket-myuser/nepomuk-socket (Aucun fichier ou dossier de ce type)"
"/usr/bin/kmail(15566)" Soprano: "Unsupported operation (2)": "Invalid model"
"/usr/bin/kmail(15566)" Soprano: "Unsupported operation (2)": "Invalid model"
"/usr/bin/kmail(15566)" Soprano: "Invalid iterator."
"/usr/bin/kmail(15566)" Soprano: "Could not connect to server at /tmp/ksocket-myuser/nepomuk-socket (Aucun fichier ou dossier de ce type)"
"/usr/bin/kmail(15566)" Soprano: "Could not connect to server at /tmp/ksocket-myuser/nepomuk-socket (Aucun fichier ou dossier de ce type)"
"/usr/bin/kmail(15566)" Soprano: "Unsupported operation (2)": "Invalid model"
"/usr/bin/kmail(15566)" Soprano: "Unsupported operation (2)": "Invalid model"
"/usr/bin/kmail(15566)" Soprano: "Invalid iterator."
kmail2(15566) KMReaderWin::setMessage: void KMReaderWin::setMessage(const Akonadi::Item&, MessageViewer::Viewer::UpdateMode) QSplitter(0x1c79610, name = "splitter2")
kmail2(15566) MailCommon::SendMdnHandler::Private::handleMessages: AKONADI PORT: Disabled code in   void MailCommon::SendMdnHandler::Private::handleMessages()
kmail2(15566) KIO::DataProtocol::DataProtocol:
kmail2(15566) KIO::DataProtocol::get: kio_data@ KIO::DataSlave(0xda8f870) ::get(const KUrl& url)
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done
"/usr/bin/kmail(15566)" Soprano: "Could not connect to server at /tmp/ksocket-myuser/nepomuk-socket (Aucun fichier ou dossier de ce type)"
"/usr/bin/kmail(15566)" Soprano: "Could not connect to server at /tmp/ksocket-myuser/nepomuk-socket (Aucun fichier ou dossier de ce type)"
"/usr/bin/kmail(15566)" Soprano: "Unsupported operation (2)": "Invalid model"
"/usr/bin/kmail(15566)" Soprano: "Unsupported operation (2)": "Invalid model"
"/usr/bin/kmail(15566)" Soprano: "Invalid iterator."
"/usr/bin/kmail(15566)" Soprano: "Could not connect to server at /tmp/ksocket-myuser/nepomuk-socket (Aucun fichier ou dossier de ce type)"
"/usr/bin/kmail(15566)" Soprano: "Could not connect to server at /tmp/ksocket-myuser/nepomuk-socket (Aucun fichier ou dossier de ce type)"
"/usr/bin/kmail(15566)" Soprano: "Unsupported operation (2)": "Invalid model"
"/usr/bin/kmail(15566)" Soprano: "Unsupported operation (2)": "Invalid model"
"/usr/bin/kmail(15566)" Soprano: "Invalid iterator.""

Is that helping ?
Comment 39 JDiel 2012-09-28 05:34:09 UTC
OS: Linux 
KMail Version: 4.8.5

Dear,
I'd like to rephrase the problem:
"Inbox" is displayed empty.
When I select the inbox folder, I see old mails re-appearing for a fraction of a second, but my filtering rules move the mails to their destination folders, where they appear again as "unread" and create duplicates. The "Inbox" looks empty again.
<F5> (I think it is "refresh" in English) while in the inbox triggers the same behavior.
My theory: This wrong behavior has nothing to do with the filtering rules, they work just fine.
The problem is, that old emails, which have been moved away from "inbox" already and seem to be away in the frontend, are still lurking in the dark and magically reappear when the inbox is refreshed.
Basic problem acc. to this theory: When a mail is moved, it looks moved in the front-end, but it is not really moved.
For me, this problem is really severe, because it makes it impossible to identify really new mails and creates an enormous # of duplicates (and "delete duplicates" sometimes works, sometimes not or it takes forever, and can't be done recursively).
Comment 40 JDiel 2012-09-28 07:38:38 UTC
I could verify my theory:
 - switched off all filters
 - <F5> in inbox
 - about 50 old mails reappeared w/o being moved by filters
 - really deleted these mails (<shift> + <del>)
 - <F5> in inbox
 - no old mails showed up again (q.e.d)
Comment 41 András Manţia 2012-10-13 18:48:10 UTC
Can anybody reproduce the problem with KDE 4.9.2 (and preferably Akonadi 1.8.1)?
If yes, what kind of account does the filter run on (maildir, mixedmaildir, mbox, imap - if imap, online or disconnected)?
Does the mails come up if you delete simply with "Del" and refresh after?
Comment 42 JDiel 2012-10-14 06:24:00 UTC
Since my previous entry, I changed to these versions of KMail and Akonadi = 4.9.2-356.1-x86_64
Account = maildir
del and refresh now do NOT make the mail reappear anymore, but:
1. Move a mail manually from the inbox to to a subdir
2. Mail is not visible anymore in inbox, but visible in the subdir
3. <F5> Refresh: mail is visible in Inbox AND subdir, i.e., duplicate
Comment 43 András Manţia 2012-10-14 08:39:34 UTC
Is the subdir a subirectory of inbox or a different subdirectory? Just to double check, is it maildir or mixedmaildir account?
Can you check that after the move (but before refresh), is the mail still physically on the disk in the inbox folder? You can check the location in the account settings, and locate the mail file e.g with a search in content on the file system.
Comment 44 JDiel 2012-10-14 09:45:20 UTC
It is 3 levels below inbox, like:
/inbox/L1/L2/targetOfMove
in the properties of the target folder, it says "maildir"
before move: /home/juergen/.local/share/local-mail/Posteingang/cur/1350028982.R502.DIY-TUX:2,S
("Posteingang" = German for "Inbox")
after move: Still in the same directory.
Comment 45 Anders Lund 2012-10-14 10:13:51 UTC
Created attachment 74537 [details]
imap log, a duplicate message is fetched

Since that other bug does not fit the problem I experience, I attach my logs here. More to follow.
Comment 46 Anders Lund 2012-10-14 10:15:58 UTC
Created attachment 74538 [details]
filter log, a duplicated message is fetched

This should be the counterpart to the imap log attached above.
Comment 47 András Manţia 2012-10-14 10:45:45 UTC
 Which one gets duplicated, the spam (Donación de la Hermana Alice Jahab)
or the mail sent to @kde.org ([Bug 283548] Only first filter in chain works for incoming messages)? 
I see the second (the kde.org mail) is deleted from inbox, so i guess that's not the problematic one.

The spam is more interesting. Can you paste the two filters that go through it from the akonadi_mailfiter_agentrc?
These ones:
[11:48:19] Evaluerer filterregler: (match all of the following) "<size>" <greater> "0"

[11:48:19] 1 = "<size>" <greater> "0" ( 14414 )

[11:48:19] Filterregler er blevet matchet.

[11:48:19] Applying filter action: Pipe Through "bogofilter -p -e"

[11:48:28] Evaluerer filterregler: (match any of the following) "X-Bogosity" <regexp> "(yes)|(spam\b)"

[11:48:28] 1 = "X-Bogosity" <regexp> "(yes)|(spam\b)" (Spam, tests=bogofilter, spamicity=1.000000, version=1.2.2)

What should happen here is:
- mail is moved from IMAP inbox to local folder
- mail is marked as spam and read (in the local folder)

From the IMAP log though what happens is:
- the mail is appended again to the inbox immediately after it is fetched and marked as spam and read (\Seen $JUNK)
- the original mail is deleted (two times!)
Seems like the filter actions are executed in wrong order, resulting in code duplication.
Comment 48 Marcelo Sales 2012-10-14 14:01:37 UTC
I'm seeing this in 4.9.2 (Kubuntu 12.04 x86_64)
Several messages get duplicated when filtered, and very often the duplicate one has only the header, no body. When this error happens, only one duplicate is created for each message affected.
The duplicated message (the one without a body) is sometimes left in inbox and sometimes moved automatically to the filter destination folder along with the original message. Sometimes the duplicate stays in the inbox, but as soon as I click the inbox the message is moved automatically to the filter destination folder. Sometimes, it is moved only if I trigger the filters manually with CTRL+J.
Almost always I can delete both the original and the duplicate by pressing DEL. Sometimes, however, one of them can't be deleted. When I press DEL the selection goes to the next message, but the message I tried to delete remains where it was. I can then repeat this several times with that message and it will not go away. If I close KMail and reopen it, the problematic message is gone from the folder where it was.
Comment 49 Marcelo Sales 2012-10-14 14:04:52 UTC
I forgot to mention, I get e-mails only from POP3 accounts (no IMAP) and they are locally stored in a maildir storage.
Comment 50 Alexandre Bonneau 2012-10-17 10:01:44 UTC
For info, I checked if the messages were really physically duplicated in the folders, or only in akonadi cache ; they are duplicated each 17 times physically.
Using maildir, each mail has a distinct name like '1350381035.R668.linux', '1350352842.R783.linux:2,S' or '1349859050.R458.linux:2,FS'.

This problem still means NOGO for use in a production environment.
Comment 51 András Manţia 2012-10-17 10:09:28 UTC
FYI, I have a tentative patch for this, but need to clean up and backport to 4.9 branch. I will post here once it is done, so you can test it.
Comment 52 András Manţia 2012-10-18 20:54:00 UTC
Git commit 37515fa641381b1357f26b0372f69bf9300dc10f by Andras Mantia.
Committed on 18/10/2012 at 22:53.
Pushed by amantia into branch 'KDE/4.9'.

Backport of the filter pipeline refactoring fix, that should fix the duplication of mails when it goes through several filter that move and modify the mail.
Related: bug 305073
FIXED-IN: 4.9.3

M  +29   -41   mailfilteragent/filtermanager.cpp

http://commits.kde.org/kdepim/37515fa641381b1357f26b0372f69bf9300dc10f
Comment 53 Alexandre Bonneau 2012-11-12 16:54:21 UTC
Using Kmail 4:4.9.3-0ubuntu0.1~ppa1, I indeed see close to no more duplication !
Great job !!

I did saw 2 or 3 duplications while browsing previously unread mails (mails that were fetched before using 4.9.3), but so far, so good.

I'll report back here in a week for my last statement on this annoying bug.
Comment 54 Christian Trippe 2012-11-15 20:59:59 UTC
I still get duplicated messages with KDE 4.9.3 when downloading mails from a pop3 account which then get directly moved to a subfolder via a filter. Both the original and the duplicate message are displayed in the destiantion folder of the filter and you can "remove" the duplicate by right click on the folder and choosing "refresh folder".
Comment 55 András Manţia 2012-11-16 14:33:12 UTC
Christian: what kind of accounts are involved (source/destination). Can you post the filter in question from the filter config file (akonadi_mailfilter_agentrc) and the log? Which duplicate disappears on the refresh (original or dest folder)?
Comment 56 Christian Trippe 2012-11-16 19:01:32 UTC
The account is a pop3 account 


christian@asterix:~> cat .kde4/share/config/akonadi_pop3_resource_2rc
[General]
host=pop.gmx.net
intervalCheckEnabled=true
leaveOnServerCount=0
leaveOnServerDays=0
leaveOnServerSize=0
login=XXX@gmx.de
storePassword=true
targetCollection=35
useTLS=true

My local collection is a maildir-resource


The filter is
[Filter #10]
Applicability=1
AutomaticName=false
ConfigureShortcut=false
ConfigureToolbar=false
Enabled=true
Icon=system-run
StopProcessingHere=true
ToolbarName=<List-Post>:<mailto:opensuse-factory@opensuse.org>
action-args-0=44
action-name-0=transfer
actions=1
apply-on=check-mail,manual-filtering
contentsA=<mailto:opensuse-factory@opensuse.org>
fieldA=List-Post
funcA=contains
identifier=A6fHojqTOAJOHYm7
name=<List-Post>:<mailto:opensuse-factory@opensuse.org>
operator=and
rules=1

Both files are in the destination folder. But there is only one mail under .local/share/.local-mail.directory/some/sub/directory but two ar under ./local/share/akonadi/file_db_data/

As soon as the last folder gets cleaned up. The second mail which kmail displays vanishes.

Regarding the logs: I am not sure how to obtain them.
Comment 57 Alexandre Bonneau 2012-11-20 09:34:16 UTC
After one week of intense testing, I see no more "17-duplicates" happening, which make Kmail usable one again, nice work there !

I did see once Kmail duplicate 2 mails with 1 clone each. The cloned mails shows no content, only headers, and unfortunately I realized that after deleting the good mails.
I use IMAP + Maildir.

As I said, I saw the behavior only one in one week, So...so far so good.
Comment 58 Travis Evans 2012-11-20 16:06:54 UTC
I still have the problem described in Comment 54 in KDE 4.9.3. Maildir, POP3. I have three messages that keep appearing in inbox every time I try to move, mark as read, or delete them. I also get a ton of “Maildir '' for collection '' is invalid” when I change folders. Anything I can do to try to figure out what's going on?
Comment 59 András Manţia 2012-11-20 16:22:49 UTC
Travis: your problem might be the one you describe in bug 288020, unrelated to the filtering.
Comment 60 Alexandre Bonneau 2012-11-21 17:41:48 UTC
To be more precise, even after this fix, you can still see the "17-duplicates" bug happens, not on mails that came since you started to use 4.9.3, but on previously received mails ; I had to click on every single folder of my mail folder tree to see some duplications, and remove duplicates manually.

I guess now I checked all my folders, and regarding how the fix from András fixed things for me, I won't see anymore duplication, hopefully.
Comment 61 Tim Holy 2013-01-22 09:53:40 UTC
Are there any chances this will be backported to 4.8 and hence appear in the default packages for Kubuntu 12.04? Or should I upgrade to 4.9?
Comment 62 Marc Collin 2013-04-16 18:51:46 UTC
problem exist in 4.10.2
Comment 63 Alexandre Bonneau 2013-04-16 22:59:31 UTC
Using 4.10.2 since its release, I haven't encountered this bug so far.
Comment 64 Hussam Al-Tayeb 2013-05-12 11:02:03 UTC
problem still in 4.10.3
akonadictl stop 
rm -rf ~/.local/share/akonadi
akonadictl start

remvoes the duplicates.
Comment 65 Hussam Al-Tayeb 2013-07-08 16:31:31 UTC
same email being filtered twice

---------------------
[18:50:24] Begin filtering on message "[Bug 627699] Port GTK2 to GTK3" from "bugzilla-daemon@mozilla.org" at "Monday 08 July 2013 15:47" :
[18:50:24] Evaluating filter rules: (match any of the following) "From" <contains> 
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "bugs@archlinux.org"
[18:50:24] 0 = "From" <contains> "bugs@archlinux.org" (bugzilla-daemon@mozilla.org)
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "clementine-player@googlecode.com"
[18:50:24] 0 = "From" <contains> "clementine-player@googlecode.com" (bugzilla-daemon@mozilla.org)
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "bugzilla-daemon@freedesktop.org"
[18:50:24] 0 = "From" <contains> "bugzilla-daemon@freedesktop.org" (bugzilla-daemon@mozilla.org)
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "bugzilla@gnome.org"
[18:50:24] 0 = "From" <contains> "bugzilla@gnome.org" (bugzilla-daemon@mozilla.org)
[18:50:24] Evaluating filter rules: (match any of the following) "Sender" <contains> "bugzilla_noreply@kde.org" "Reply-To" <contains> "bug-control@bugs.kde.org"
[18:50:24] 0 = "Sender" <contains> "bugzilla_noreply@kde.org" ()
[18:50:24] 0 = "Reply-To" <contains> "bug-control@bugs.kde.org" ()
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "bugzilla-daemon@llvm.org"
[18:50:24] 0 = "From" <contains> "bugzilla-daemon@llvm.org" (bugzilla-daemon@mozilla.org)
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "bugzilla-daemon@mozilla.org"
[18:50:24] 1 = "From" <contains> "bugzilla-daemon@mozilla.org" (bugzilla-daemon@mozilla.org)
[18:50:24] Filter rules have matched.
[18:50:24] Applying filter action: Move Into Folder "Local Folders/inbox/Bugs/Mozilla"
------------------------------
[18:50:25] Begin filtering on message "[Bug 627699] Port GTK2 to GTK3" from "bugzilla-daemon@mozilla.org" at "Monday 08 July 2013 15:47" :
[18:50:25] Evaluating filter rules: (match all of the following) "From" <contains> "bugs@archlinux.org"
[18:50:25] 0 = "From" <contains> "bugs@archlinux.org" (bugzilla-daemon@mozilla.org)
[18:50:25] Evaluating filter rules: (match all of the following) "From" <contains> "clementine-player@googlecode.com"
[18:50:25] 0 = "From" <contains> "clementine-player@googlecode.com" (bugzilla-daemon@mozilla.org)
[18:50:25] Evaluating filter rules: (match all of the following) "From" <contains> "bugzilla-daemon@freedesktop.org"
[18:50:25] 0 = "From" <contains> "bugzilla-daemon@freedesktop.org" (bugzilla-daemon@mozilla.org)
[18:50:25] Evaluating filter rules: (match all of the following) "From" <contains> "bugzilla@gnome.org"
[18:50:25] 0 = "From" <contains> "bugzilla@gnome.org" (bugzilla-daemon@mozilla.org)
[18:50:25] Evaluating filter rules: (match any of the following) "Sender" <contains> "bugzilla_noreply@kde.org" "Reply-To" <contains> "bug-control@bugs.kde.org"
[18:50:25] 0 = "Sender" <contains> "bugzilla_noreply@kde.org" ()
[18:50:25] 0 = "Reply-To" <contains> "bug-control@bugs.kde.org" ()
[18:50:25] Evaluating filter rules: (match all of the following) "From" <contains> "bugzilla-daemon@llvm.org"
[18:50:25] 0 = "From" <contains> "bugzilla-daemon@llvm.org" (bugzilla-daemon@mozilla.org)
[18:50:25] Evaluating filter rules: (match all of the following) "From" <contains> "bugzilla-daemon@mozilla.org"
[18:50:25] 1 = "From" <contains> "bugzilla-daemon@mozilla.org" (bugzilla-daemon@mozilla.org)
[18:50:25] Filter rules have matched.
[18:50:25] Applying filter action: Move Into Folder "Local Folders/inbox/Bugs/Mozilla"
Comment 66 Hussam Al-Tayeb 2013-07-08 16:48:10 UTC
I will open a new bug. seems this is a different issue. sorry for the spam.
Comment 67 Dmitri Koulikoff 2013-11-12 07:13:16 UTC
4.10.5 I experience the problem.
Comment 68 jaanus 2014-01-22 18:06:57 UTC
Still not fixed. Account which has filters will duplicate messages which are not matched by any filters ad infinitum. Which is extremely annoying.

When will it be fixed?

$ rpm -q kdepim
kdepim-4.11.5-1.fc20.x86_64
Comment 69 Frédéric COIFFIER 2014-01-23 08:35:37 UTC
Still occurs in KDE 4.12.1 (Gentoo)...

I've just deleted my ~/.local/share/akonadi as mentionned in a comment above.
I'll see in the next days if it changes something...

I have also delayed the Kmail startup as the problem occurs often when the whole KDE session is restored (in this case, the net connection can be overloaded and emails download can be slow...).
Comment 70 jaanus 2014-02-22 21:06:49 UTC
Still broken in kdepim-4.12.2-1.fc20.x86_64.

Is this issue being worked on?
Comment 71 Christian Boltz 2014-02-23 14:02:22 UTC
I also still see this bug with KDE 4.12.1 - reopening...
Comment 72 Frédéric COIFFIER 2014-02-24 14:41:01 UTC
I confirm too (kdepim 4.12.2 and akonadiserver 1.11.0). I can't find a working solution : deleting  ~/.local/share/akonadi, removing bogofilter filters, delaying KMail start. Nothing can solve the problem. And the problem is mainly due to a slow harddisk (due to a highload) or a slow network connection.
Comment 73 Daniel Vrátil 2014-02-25 08:31:04 UTC
If you can reproduce this manually on one or two emails, please start Akonadi Console, enable the "Debugger", "Job Tracker" and "Notification Monitor" tabs and try to reproduce the issue.

Then provide output from the "All" tab in "Debuger" tab and screenshots from the other two tabs (please expand all subtrees). Sorry, we don't have text export yet :-)
Comment 74 Marcus 2014-03-29 14:31:34 UTC
Created attachment 85825 [details]
Akonadi Console Notification Monitor
Comment 75 Marcus 2014-03-29 19:48:37 UTC
the debugging log was actually quite fast 130MB big and had some emails content. I dont feel like uploading it. But, the email content in the debuglog was exact the one, which was appearing 7 times (now) as new email.
Comment 76 Marcus 2014-03-31 16:55:06 UTC
I first deleted all filters, only left Spam and virus filters. Before I had a filter sorting some emails by sender into a subfolder of the SENT folder. Lets call the subfolder 123.

After manually moving emails from the SENT folder to the subfolder 123 they reappear soon in the original SENT folder, now doubled, one in the subfolder 123, one in the SENT folder. The subfolder 123 was created a longer time ago.
Comment 77 Marcus 2014-03-31 17:34:37 UTC
and as other people described, it's an ongoing duplication - sometimes. And it not every email I manually put into that subfolder 123.
Comment 78 Hussam Al-Tayeb 2014-04-06 20:32:16 UTC
It seems like I hit this with kmail 4.13rc1 when I manually check for mail when there are emails being currently downloaded and filtered.
Comment 79 Nikos Alexandris 2014-04-11 14:18:38 UTC
I think iI experience the same here under KDE Development Platform: 4.12.0, KMail: 4.12. Very annoying.
Comment 80 Joachim Wagner 2014-06-25 10:58:11 UTC
The bug seems to be triggered by something that doesn't happen very often. I've been using kmail in kontact 4.11.5 for 8 days without problems and now just got hit by this bug.

While this report is not for the most recent version, the following observations may help the developers to reproduce the bug. Like other bug reporters above, I cannot attach akonida debug log files as they contain confidential information.

Setup:
 * fresh OpenSUSE 13.1 with default updates (kmail/kontact 4.11.5)
 * fresh home directory
 * pop3 account connected to inbox
 * approximately 80 destination folders under "Local Folders"
 * no automatic filters defined
 * manually moving e-mails to destination folders
 * often using the "Search" input field over the message list and selecting multiple messages (shift + left click or ctrl + left click) to move multiple messages in one go

Observation:
 * in my case, the problem seems to be related to a specific local destination folder (it was newly created just before the bug was observed, let's call this folder "A") as the moved message doesn't re-appear as a duplicate in the inbox if I move it to a different destination folder (let's call it "B") and press F5
 * if I move the message from folder B to folder A, it is still duplicated in folder B (after pressing F5) but not in the inbox
 * if I create a new local folder "C", move the message from B to C and press F5, everything is normal
 * however, if I create "C2" under the A or the parent of A (which had been created just seconds before A) and move a message to it, duplication occurs

Workaround 1:
 * the "copy first, then delete with ctrl-del" trick worked for me

Workaround 2:
 * create a new folder "A-Parent-2" and subfolder "A2"
 * move message from A to A2
 * press F5 (no duplicates should re-appear in A)
 * select A2 and run "Folder - Remove Duplicate Messages"
 * delete empty folders A and A-Parent
 * rename A2 to A  and A-Parent-2 to A-Parent

JJ
Comment 81 Hussam Al-Tayeb 2014-06-28 10:18:14 UTC
on kmail 4.13.2, this happens with every single filtered email.
pop3 account.
I end up stopping akonadi and clearing the database, starting akonadi, importing the filters again over a 100 times a day which is too time consuming.
Any way i can monitor some logs to tell what is happening?
This is a really bad bug. I hate to complain and rant (because complaining doesn't solve bugs) but it sounds that this isn't getting the attention it needs.
At least tell us please what we can do to help.
Thank you.
Comment 82 Christophe Giboudeaux 2014-06-28 16:11:19 UTC
(In reply to comment #81)
> Any way i can monitor some logs to tell what is happening?

tools / Filter Log Viewer 
Check "Log filter activities" and the relevant boxes below then run your filter.

Don't forget to uncheck "Log filter activities" once done
Comment 83 Dennis Andress 2014-07-03 19:10:54 UTC
On version 4.13.2 this problem began two days ago. There may have been Fedora updates precipitating the start.

Duplicate emails are disabled in the message list and can not be selected or deleted. If I stop Kontact/Kmail and start another email client, SeaMonkey in my case, the duplicate emails will not be present. If, and only if, I select the folder which contained duplicates in KMail, and then restart KMail, the duplicates will be gone.
Comment 84 Hussam Al-Tayeb 2014-07-03 21:14:45 UTC
(In reply to comment #83)
> On version 4.13.2 this problem began two days ago. There may have been
> Fedora updates precipitating the start.
> 
> Duplicate emails are disabled in the message list and can not be selected or
> deleted. If I stop Kontact/Kmail and start another email client, SeaMonkey
> in my case, the duplicate emails will not be present. If, and only if, I
> select the folder which contained duplicates in KMail, and then restart
> KMail, the duplicates will be gone.

They won't show up in another mail client because the duplicates are in akonadi database.
Comment 85 Dennis Andress 2014-07-03 21:18:34 UTC
(In reply to comment #84)
> (In reply to comment #83)
> 
> They won't show up in another mail client because the duplicates are in
> akonadi database.

I hadn't considered that, thank you. Yet, restarting KMail does not fix the problem, running akonadictl vacuum and fsck does not fix it either. But, viewing mail from another client does, or at least seems to affect the problem.
Comment 86 Kevin 2014-07-27 05:13:27 UTC
New to KDE.  Running Fedora 20, KDE 4.13.2.  Created about 20 email identities, with about the same number of filters to simply move messages into appropriate folders.  Filter processing is duplicating email messages.  If I disable filters, problem goes away.  Example: With filters enabled my inbox contains three copies of the KDE Bug Tracking Account Creation email (I just signed up to provide this input).  Each time I click Check Mail, additional email messages are duplicated for new email messages.

Am surprised and disappointed such a fundamental, common feature does not work.  Must seek another email client solution as a result.
Comment 87 schuschq 2014-08-11 09:28:46 UTC
had the same problem, 

after migrating a really big mailbox (lots of old mails, mostly small mails, using 1,9 GB now on drive) from pine some two years ago, i used another program on opensuse 12.x (the standard one, forgot the name) and transferred all the mailbox folders to it. After setting up a new linux box with opensuse 13.1 I finally transferred all the mail to K-Mail - which all worked without problems. But since introducing new filters in K-Mail everytime i changed to the incoming mail folder (Posteingang) now, I had a set of spam mails which always reappeared ... they are then immediately filtered in a spam folder ... in this folder I had dozens of duplicates after some weeks. The duplicates all "belonged" to one of the filters.

here: Opensuse 13.1, K-Mail/KDE 4.11.5

what I did to get rid of this duplicates

- i changed the folder for incoming mail to anotherone (e.g. _incoming)
- i moved all the mails from the incoming folder to _incoming
- quitted K-Mail

- found still mail in 
  /home/schorsch/.local/share/local-mail/inbox/new/
  (all from this spamming shopping-de.cc mailer)

- deleted these mails manually

- opened K-Mail again, switched incoming mail folder back to original folder
- quitted K-Mail, opened it again, moved mail back to original folder
- done

no more duplicates :-) ... perhaps this helps?
Comment 88 schuschq 2014-08-12 06:54:26 UTC
(In reply to schuschq from comment #87)
> what I did to get rid of this duplicates

> no more duplicates :-) ... perhaps this helps?

but the next new message hitting the filter leads to new duplicates ... so still not solved (at least the old duplicates are no longer replicated)
Comment 89 mau 2014-08-18 12:08:03 UTC
(In reply to Joachim Wagner from comment #80)
> The bug seems to be triggered by something that doesn't happen very often.
> I've been using kmail in kontact 4.11.5 for 8 days without problems and now
> just got hit by this bug.
> 
> While this report is not for the most recent version, the following
> observations may help the developers to reproduce the bug. Like other bug
> reporters above, I cannot attach akonida debug log files as they contain
> confidential information.
> 
> Setup:
>  * fresh OpenSUSE 13.1 with default updates (kmail/kontact 4.11.5)
>  * fresh home directory
>  * pop3 account connected to inbox
>  * approximately 80 destination folders under "Local Folders"
>  * no automatic filters defined
>  * manually moving e-mails to destination folders
>  * often using the "Search" input field over the message list and selecting
> multiple messages (shift + left click or ctrl + left click) to move multiple
> messages in one go
> 
> Observation:
>  * in my case, the problem seems to be related to a specific local
> destination folder (it was newly created just before the bug was observed,
> let's call this folder "A") as the moved message doesn't re-appear as a
> duplicate in the inbox if I move it to a different destination folder (let's
> call it "B") and press F5
>  * if I move the message from folder B to folder A, it is still duplicated
> in folder B (after pressing F5) but not in the inbox
>  * if I create a new local folder "C", move the message from B to C and
> press F5, everything is normal
>  * however, if I create "C2" under the A or the parent of A (which had been
> created just seconds before A) and move a message to it, duplication occurs
> 
> Workaround 1:
>  * the "copy first, then delete with ctrl-del" trick worked for me
> 
> Workaround 2:
>  * create a new folder "A-Parent-2" and subfolder "A2"
>  * move message from A to A2
>  * press F5 (no duplicates should re-appear in A)
>  * select A2 and run "Folder - Remove Duplicate Messages"
>  * delete empty folders A and A-Parent
>  * rename A2 to A  and A-Parent-2 to A-Parent
> 
> JJ

This might have the same reason than something I observed just this weekend:
I added a new folder to my local folders and it took several seconds until it was shown. Afterwards I manually moved messages to this new folder. The showed up correctly in KMail, but no in the corresponding folder on my disk (.local/share/local-mail/...) so they seem to stay in the akonadi cache forever. Just a second after trying to move them the reappear in the original folder. The problem didn't show up with another folder I created. So it looks like the main problem being a folder not being created correctly.
Comment 90 Hussam Al-Tayeb 2014-08-18 12:18:53 UTC
On Monday 18 August 2014 12:08:03 you wrote:
> https://bugs.kde.org/show_bug.cgi?id=283682
> 
> mau <b-misc@gmx.ch> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
> CC|                            |b-misc@gmx.ch
> 
> --- Comment #89 from mau <b-misc@gmx.ch> ---
> (In reply to Joachim Wagner from comment #80)
> 
> > The bug seems to be triggered by something that doesn't happen very often.
> > I've been using kmail in kontact 4.11.5 for 8 days without problems and
> > now
> > just got hit by this bug.
> > 
> > While this report is not for the most recent version, the following
> > observations may help the developers to reproduce the bug. Like other bug
> > reporters above, I cannot attach akonida debug log files as they contain
> > confidential information.
> > 
> > Setup:
> >  * fresh OpenSUSE 13.1 with default updates (kmail/kontact 4.11.5)
> >  * fresh home directory
> >  * pop3 account connected to inbox
> >  * approximately 80 destination folders under "Local Folders"
> >  * no automatic filters defined
> >  * manually moving e-mails to destination folders
> >  * often using the "Search" input field over the message list and
> >  selecting
> > 
> > multiple messages (shift + left click or ctrl + left click) to move
> > multiple messages in one go
> > 
> > Observation:
> >  * in my case, the problem seems to be related to a specific local
> > 
> > destination folder (it was newly created just before the bug was observed,
> > let's call this folder "A") as the moved message doesn't re-appear as a
> > duplicate in the inbox if I move it to a different destination folder
> > (let's call it "B") and press F5
> > 
> >  * if I move the message from folder B to folder A, it is still duplicated
> > 
> > in folder B (after pressing F5) but not in the inbox
> > 
> >  * if I create a new local folder "C", move the message from B to C and
> > 
> > press F5, everything is normal
> > 
> >  * however, if I create "C2" under the A or the parent of A (which had
> >  been
> > 
> > created just seconds before A) and move a message to it, duplication
> > occurs
> > 
> > Workaround 1:
> >  * the "copy first, then delete with ctrl-del" trick worked for me
> > 
> > Workaround 2:
> >  * create a new folder "A-Parent-2" and subfolder "A2"
> >  * move message from A to A2
> >  * press F5 (no duplicates should re-appear in A)
> >  * select A2 and run "Folder - Remove Duplicate Messages"
> >  * delete empty folders A and A-Parent
> >  * rename A2 to A  and A-Parent-2 to A-Parent
> > 
> > JJ
> 
> This might have the same reason than something I observed just this weekend:
> I added a new folder to my local folders and it took several seconds until
> it was shown. Afterwards I manually moved messages to this new folder. The
> showed up correctly in KMail, but no in the corresponding folder on my disk
> (.local/share/local-mail/...) so they seem to stay in the akonadi cache
> forever. Just a second after trying to move them the reappear in the
> original folder. The problem didn't show up with another folder I created.
> So it looks like the main problem being a folder not being created
> correctly.

That's one of the symptoms but not the only problem. I have cleared the 
akonadi cache more than once. So non existing emails/folders don't show up in 
clean akonadi folders. 
Putting emails in non existing folders will probably cause data loss by the 
way so be careful.

There are many issues here:
1) emails get filtered twice for some reason and thus appearing twice in 
akonadi database.
2) some race condition in updating the akonadi database because on occasions, 
the extra copy only appears for a split second then immediately disappears.
3) possibly many other reasons.
Comment 91 Sylvain 2014-08-26 01:02:53 UTC
Using Kmail version 4.14 on archlinux. I enabled bogofilter and I have the problem of multiple copies (thousands!) of the same mail when a new mail arrives. seems to be the same as this bug. If I disable bogofilter, the problem disappear.

I sent me an email from $MAIL@gmail.com to $PROMAIL@univ_mail and logged everything. I waited to have multiple copies (5) before putting kmail offline. when I put it online again it will download one more copies (read state directly) and stop there (sorry no log).

Hope this can help
Comment 92 Sylvain 2014-08-26 01:04:16 UTC
Created attachment 88424 [details]
akonadi log all
Comment 93 Sylvain 2014-08-26 01:04:50 UTC
Created attachment 88425 [details]
akonadi log simple
Comment 94 Sylvain 2014-08-26 01:05:23 UTC
Created attachment 88426 [details]
akonadi maildir resource
Comment 95 Sylvain 2014-08-26 01:05:48 UTC
Created attachment 88427 [details]
akonadi mailfilter agent
Comment 96 Sylvain 2014-08-26 01:06:21 UTC
Created attachment 88428 [details]
new mailnotifier agent
Comment 97 Sylvain 2014-08-26 01:06:52 UTC
Created attachment 88429 [details]
mail filter kernel ETM
Comment 98 Sylvain 2014-08-26 01:07:30 UTC
Created attachment 88430 [details]
kmail filter log applying the bogofilter pipe
Comment 99 Dennis Andress 2014-08-28 13:11:55 UTC
KMail 4.13.3
Fedora 20  (testing and unstable repos in use)

I had the problem where a spam filter would endlessly duplicate emails. I quickly removed the spam filter and things settled down. The problem returns when I delete or move an email. The email will move to the detination folder, only to shortly reappear, disabled and not selectable, in the original folder.

I have two email accounts configured, GMail and the my employer's (ms exchange server). The problem occurs almost exclussively with my employer's account. Emails duplicate in the GMail spam folder, curiously taking several minutes to show up instead of almost immediately.

I don't see akonadi being at fault. Running akonadictl fsck and vacuum show no errors, and don't affect the problem.  Deleteing the Adkonadi database and starting over didn't help either. Yet, I can clear the duplicates by using any other mail client and synching each folder individually. To make a SWAG (stupid wild ass guess), I'd look for something in LDAP usage.

Sorry to report a problem and not offer help. Thanks.
Comment 100 Dennis Andress 2014-09-25 17:13:57 UTC
This bug is annoying...

I restarted Akonadi this morning and left the terminal window open. I then deleted several emails from one KMail folder and noticed this, repeated once for each deleted email, in the terminal window

akonadi_imap_resource_7(3524)/kdepimlibs (kimap) KIMAP::StoreJob::handleResponse: We asked for UID but the server didn't give it back, resultingFlags not stored. 

After awhile I hacked around with Akonadi Console and eventually found that for the duplicated emails the table PIMITEMFLAGRELATIONS will have two records, one with the flag /SEEN and one with the flag /DELETED.  I didn't look very hard, but it seems that pimitemflagrelations has only one record for emails that are not duplicated.

This SQL run from the DB Query tab of Akonadi Console seems to confirm that.

select * from pimitemflagrelation pif
INNER JOIN pimitemflagrelation pir ON pir.pimitem_id = pif.pimitem_id
where pif.flag_id = 6 and pir.flag_id = 2

--   where flag_id 6 == /DELETED and flag_id 2 == /SEEN

I admit that is poor SQL. I'm not a fan of MySQL, having decided long ago that its implementation of SQL is not compatible with my career goals.  

It seems that message I saw in Akonadi's stdout is saying that the /SEEN record was not removed from pimitemflagrelations when the /DELETED record was added.
Comment 101 Hussam Al-Tayeb 2014-10-10 10:06:26 UTC
I found a way to clear the duplicates without clearing the whole akonadi 
database.

Close kmail and open akonadiconsole.
In the browser tab, right click on the collection/folder where you have 
duplicates.
Select the "Clear akonadi cache" context menu option.
Now open kmail and it will rescan that folder and hopefully the duplicates 
will be gone.
Comment 102 Tobias Leupold 2014-10-10 10:26:24 UTC
Does not work for me. Even if so, this does not solve the problem, but only handles it … because we still have lots and lots of duplicate mails generated by filtering :-(
Comment 103 Tobias Leupold 2014-10-10 13:58:45 UTC
Well, I think the fact that clearing the akonadi cache is not removing the duplicate mails for me is mostly due to the fact that mail filtering actually duplicates mails physically.

When I get duplicated mails, I actually have two files with different names but the very identical content in my target maildir, created at the same time.
Comment 104 Hussam Al-Tayeb 2014-11-04 16:22:15 UTC
I don't think they are being duplicated physically. you might need to try once and make sure everything using akonadi is closed first before purging tables/database.
You will also need to clear the baloo database too to avoid double indexing.
But this is all just messy. I wonder why no one is investigating this thoroughly...
Comment 105 Tobias Leupold 2014-11-04 17:32:11 UTC
Honestly, I really do get the duplicate emails as separate files in the actual physical maildir.
Comment 106 Hussam Al-Tayeb 2014-11-04 17:34:39 UTC
Then there are two issues here ;(
Comment 107 Tobias Leupold 2014-11-04 17:37:30 UTC
(In reply to Hussam Al-Tayeb from comment #104)
> But this is all just messy. I wonder why no one is investigating this
> thoroughly...
I'm not completely sure if the KMail guys really use KMail themselves :-P

Even if we have two causes for the problem, it's still the right place here as the bug's name is "KMail duplicates filtered messages" – and that's what happens, no matter if it's caused by some Akonadi/Baloo/Whatever black hole or something different.
Comment 108 András Manţia 2014-11-04 18:01:58 UTC
If you look at the history, you see we care about it and use it. Unfortunately I have no real time for KMail these days, and for me it works. Both for POP3 and IMAP accounts, I have no mail duplication with filters, so it is really hard to do anything without spending a lot of time on it,
Comment 109 Tobias Leupold 2014-11-04 19:02:11 UTC
No harm meant :-) But I think there's a general filtering problem since KMail uses Akonadi. I have e. g. been using bogofilter since ever to filter my mails, and since the last few versions, some of my mails are piped through it, some are not.

I really don't want to start a discussion, as that has been done 1000 times elsewhere and this is not the right place to do so. But KMail before Akonadi has simply been rock solid and by far more reliable. These days, it's really hard (at least for me) to still stick to KMail with all the problems, esp. with this filtering issue which is really a PITA.

The problem is that there are so many components involved in KMail that it's almost impossible to describe a problem with a reproducible scenario. Sometimes it all works, most of the time it doesn't. And I have no idea how to debug it. And all you get is "for me it works" ...
Comment 110 Hussam Al-Tayeb 2014-11-04 19:07:20 UTC
multiprocess systems are vulnerable to race issues anyway.
Comment 111 Hussam Al-Tayeb 2014-11-17 10:59:16 UTC
To add even more complexity :(  , I got an email now that was filtered into the KDE bugs folder and I saw duplicates for a second and it autocleared the duplicate.
Exactly how does akonadi/kmail filtering work? A copy is made, filtered and original copy deleted?
Comment 112 Hussam Al-Tayeb 2014-11-17 11:02:02 UTC
http://i.imgur.com/2svl8Hj.png
The extra copy cannot be clicked on and disappeared when I restarted kmail.
Comment 113 András Manţia 2014-11-17 14:32:56 UTC
Unless someone changed meantime, the way it works is:
- mail is delivered to Inbox
- the filtering code gets a notification of new mails
- it runs the filters on them and does what you have set up in the filter, like moves the mail
Comment 114 Tobias Leupold 2014-11-22 08:45:45 UTC
Perhaps, this helps?

I have two maildir resources. The "normal" ~/.local/share/local-mail and an additional ~/.local/share/tmp-mail, of which I don't make backups. All e. g. mailing list mails, automatically generated status updates of my homepage etc. are filtered to that place.

Practically each and every mail filtered to the tmp-mail resource appears twice in the destination folder, alway with a physically existing duplicate file with the very same content and creation date, but a completely different file name.

Could it be that filtering mails to another maildir resource is part of the problem? Perhaps, not everybody creates multiple maildirs and thus does not always experience the filtering problem?
Comment 115 Hussam Al-Tayeb 2014-11-29 00:00:45 UTC
(In reply to Tobias Leupold from comment #114)
> Perhaps, this helps?
> 
> I have two maildir resources. The "normal" ~/.local/share/local-mail and an
> additional ~/.local/share/tmp-mail, of which I don't make backups. All e. g.
> mailing list mails, automatically generated status updates of my homepage
> etc. are filtered to that place.
> 
> Practically each and every mail filtered to the tmp-mail resource appears
> twice in the destination folder, alway with a physically existing duplicate
> file with the very same content and creation date, but a completely
> different file name.
> 
> Could it be that filtering mails to another maildir resource is part of the
> problem? Perhaps, not everybody creates multiple maildirs and thus does not
> always experience the filtering problem?

Only one maildir here. ~/.local/share/.local-mail.directory/


(In reply to András Manţia from comment #113)
> Unless someone changed meantime, the way it works is:
> - mail is delivered to Inbox
> - the filtering code gets a notification of new mails
> - it runs the filters on them and does what you have set up in the filter,
> like moves the mail
Sure but the duplication in my case is in the database ~/.local/share/akonadi and not in ~/.local/share/.local-mail.directory/
Maybe akonadi is thinking the cached copy in ~/.local/share/akonadi/file_db_data/ is a second copy and then making an extra entry in the database?
Comment 116 Tobias Leupold 2014-12-08 20:56:24 UTC
I just tried to track it down (once again). I deleted all config files and directories that belong to kmail and akonadi and started with a "fresh" install. I downloaded all mails from my account via pop3 (~17,000) and created new filters.

Filtering all the mails manually was no problem. Not a single mail was duplicated. It sometimes took almost a minute until the mails were actually moved and written to the respective place on the disc, but it worked.

As soon as I got new mails from my account, and they were filtered as they arrived, most of them were duplicated again.

I'm very sure this is a timing problem. First, the mail only exists inside the (fast) akonadi cache and is processed by the filter, which puts it in the target directory. It's also written on the respective place on the disc there. Then, some (milli)seconds later, the original mail is written to the inbox (which probably happens in a thread started at the very same time the filter agent starts which does not know what the filter agent does). And then, we have the un-filtered mail again inside the inbox. The inbox notifies the filter agent that there's some new mail inside the inbox and it's moved from there a second time by the newly started filter agent.

I'll eat my hat if this is not the very cause of the duplicated filtered mails. All the existing already-downloaded mails were already written to the disc and did not live only inside the Akonadi cache, and thus, they were not duplicated on filtering. For the few not-duplicated filtered mails, it actually worked to write them to the disc before the filter agent did his work. That's it.

Does really not one single KMail developer see this problem each and every day?! I'd really like to help to fix this if anybody told me how …
Comment 117 Tobias Leupold 2014-12-08 20:58:55 UTC
*** Bug 316419 has been marked as a duplicate of this bug. ***
Comment 118 Tobias Leupold 2014-12-12 11:31:52 UTC
To investigate this further I tried the following: I added a filter executing a "/usr/bin/sleep 0.3" for each mail downloaded. I did not get any duplicated filtered mails anymore after doing so.
Comment 119 Claudio Monteleone 2014-12-16 09:03:53 UTC
In the IMAP setting, selecting "Automatically compact folders" removes grayed messages
Comment 120 Claudio Monteleone 2014-12-16 09:04:47 UTC
In the IMAP setting, selecting "Automatically compact folders" removes grayed messages for me (kubuntu 14.10)
Comment 121 András Manţia 2014-12-16 10:16:40 UTC
Tobias Leopold, you are probably right of being a timing issue. If you can reproduce it, and could work on the code, the chances of fixing it would be high(er). As I pointed out already, I use POP3 and IMAP with local filters on two machines and didn't see this problem for a long time, so best would be to fix on a machine where it actually manifests it.
 It was a long time ago when I last worked on Akonadi/KMail code, but here are some hints:
- add debug output (possibly with timestamps) in the filtering and the akonadi code. You might have to do that both in the akonadi server and the akonadi resources (filter agent and the base classes)
- the filter agent is in kdepim/mailfilteragent, the starting point could be mailfilteragent.cpp and there the itemAdded() method. This is called when a new mail appeares in a monitored folder (ie. the inbox)
- itemsReceiviedForFiltering() there is called when the filter agent retrieves the above item (so it can analyze that it needs to be filtered or not)
- interesting code could be also in Akonadi::AgentBase class (kdepimlibs repository) and the changeRecorder() part, but only if you can't figure out anything useful from the above
- mailfilteragent/filtermanager.cpp is the code that does the filtering. processContextItem() is the method that does the move/copy of the messages. Adding debug output here also makes sense to see when does this happen
- for the server itself you can enable debugging in the akonadiconsole app, although this will give you lots of data and will make things slow, so better make sure you don't get lot of mails while debugging is enabled. But it can show all the communication between the akonadi server and the clients and all the jobs running and their order.
Comment 122 Tobias Leupold 2014-12-16 12:33:28 UTC
András, I'll have a look at the sources, but I fear the KMail/Akonadi codebase is a bit too hard for me (I have not been into the C++/Qt business very long yet ;-). But I'll try.

Another thing to substantiate the suspicion that the duplication does not happen when the mails are filtered but when they are delivered:

Yesterday, I saw that the default setting for filtering when selecting "Filter incoming messages" is that all "Accounts" are used for filtering. This includes not only the POP3 accounts (which are actual ones), but also the local maildir "Accounts".

I selected only my POP3 account and deselected everything else. I also deactivated my "/usr/bin/sleep 0.3" hack filter. After that, I ended up with filtered mails being written to their respective target directory and a (physical!) copy of those very mails being written to my inbox.

So it's actually the case that the emails are simply delivered twice, one by the filter agent and one by the, (?) pop3 agent, and the duplicating really happens because the inbox notifies new mails and the filter agent is invoked again.

Perhaps, this is "only" a sequence problem and the  filter agent is invoked too early?! Can anybody who's into the code base right now check that? Otherwise, I fear we have a design problem here :-(
Comment 123 Tobias Leupold 2014-12-16 12:51:25 UTC
Ps: Interestingly, if I only choose my local maildir as the account to filter incoming mails (and deactivate the sleep hack filter), nothing happens and the mails are simply delivered to the inbox. According to the mail filter protocol, the mail filtering starts when the new mail is added, but no filter is invoked.
Comment 124 Laurent Montel 2014-12-16 13:04:34 UTC
Indeed I confirm the duplicate mail.
Comment 125 georg 2015-01-08 12:56:27 UTC
Same issue here with KMail version 4.14.2 running on Kubuntu 14.10.
Also using bogofilter on my incoming mails on my IMAP-Account. Please let me know, if you need further information.

When receiving a mail KMail duplicates it endlessly (>500 times). I hav to turn off Akonadi server, log in onto my webmail-account, delete the mails, to keep one I move one copy to a different IMAP-Folder and restart Akonadi. The kept mail appears in the folder I moved it to, but it stays greyed out in Kmail. Happens now every-time, when I get a mail into KMail.
Comment 126 Eric Depagne 2015-01-09 07:42:13 UTC
Issue present also in kmail 4.13.3 and kubuntu 14.04. Emails get duplicated when manually moved from the inbox by drag and drop to any directory or when using the "Move Message To" Menu entry.
Comment 127 christof_kde 2015-02-21 18:09:56 UTC
Same bug here. Opensuse 13.2, Kmail 4.14.4 :-(
Comment 128 Christian Hilberg 2015-03-09 08:33:03 UTC
Seeing the same issue on Debian for 4.14.1 (jessie) as well as 4.14.2 (sid).
For me, duplicates occur only if mail filtering does NOT move the mail to a folder different from INBOX (using IMAP), where I let the filtering happen.

Moreover, "Folder"->"Remove Duplicate Messages" does not work. Of multiple duplicates, two will remain (if the operation is started at all after selecting this action, which does not seem to reliably be the case for me). I've seen an open bug report for this somewhere but I cannot seem to find it now.

Maybe there is some issue here in that Kmail/Akonadi do not reliably detect what ought to be considered "the same mail object", and hence the duplication occurs after the filters did not move the mail to a different folder?
Comment 129 Christian Hilberg 2015-03-09 09:32:06 UTC
Duplicate of this bug: https://bugs.kde.org/show_bug.cgi?id=95064
Possibly related: https://bugs.kde.org/show_bug.cgi?id=288040
Possibly related: https://bugs.kde.org/show_bug.cgi?id=333035
Comment 130 Tony O'Bryan 2015-05-31 11:11:33 UTC
The bug is present in KMail 4.14.6 using Kubuntu 15.04.  I also see duplicated Inbox messages appear in Inbox for a second, before disappearing.
Comment 131 Eike 2015-08-15 15:02:05 UTC
Got duplicated mails as well, Debian Jessie, KMail 4.14.2.
The mails are pop3'ed to the inbox and moved by a (single) filter to their destination folder.
Comment 132 Hussam Al-Tayeb 2015-09-14 18:10:48 UTC
After months away, I found out today this is still heavily happening in KF5 Kmail port.
Half of my emails are getting duplicates in the akonadi database.
Truely a classic bug :)
Comment 133 Christian Hilberg 2015-09-18 16:45:59 UTC
Confirming this still happens with KF5 Kontact/Kmail 4.14.10 on today's debian/stretch (where it seems almost all KF5 packages have materialized).
Comment 134 René Krell 2015-09-24 11:52:15 UTC
The problem does still appear in Akonadi + KMail 15.08.1.

Many duplicate mails from IMAP. Have just automatically created filter by the Anti Spam Wizard (Bogofilter) activated.
Comment 135 René Krell 2015-09-24 12:43:41 UTC
Created attachment 94708 [details]
Filter <unknown> after removing all filters manually - in this state it works

Once more for KMail + Akonadi 15.08.1.

After removing all four filter auto-generated by the Anti Spam Wizard for Bogofilter the messages from IMAP are retrieved in a regular manner again, without any duplicates. Before that I saw a race in the progress bars at the bottom right-hand side between the mail filter agent and the IMAP agent, which doen't happen now, but there is a hanging progressbar at 0% for the Mail Filter Agent.

Opening the filter dialog again results in an entry I've never made: <unknown>. In this state the messages are retrieved regularly.
Comment 136 Hussam Al-Tayeb 2015-09-24 12:48:51 UTC
(In reply to René Krell from comment #135)
> Created attachment 94708 [details]
> Filter <unknown> after removing all filters manually - in this state it works
> 
> Once more for KMail + Akonadi 15.08.1.
> 
> After removing all four filter auto-generated by the Anti Spam Wizard for
> Bogofilter the messages from IMAP are retrieved in a regular manner again,
> without any duplicates. Before that I saw a race in the progress bars at the
> bottom right-hand side between the mail filter agent and the IMAP agent,
> which doen't happen now, but there is a hanging progressbar at 0% for the
> Mail Filter Agent.
> 
> Opening the filter dialog again results in an entry I've never made:
> <unknown>. In this state the messages are retrieved regularly.

<unknown> is the default empty filter which is there before you add yours. It doesn't do anything. Since this is likely some sort of race issue with filters, it's not going to appear when there are no filters (only the <unknown> one).

Maybe junk filtering and virus scans should not be exposed as filters but instead some internal plugins (with configuration options) that hook in as filters.
Comment 137 René Krell 2015-09-24 13:09:47 UTC
(In reply to Hussam Al-Tayeb from comment #136)
> 
> <unknown> is the default empty filter which is there before you add yours.
> It doesn't do anything. Since this is likely some sort of race issue with
> filters, it's not going to appear when there are no filters (only the
> <unknown> one).
> 
> Maybe junk filtering and virus scans should not be exposed as filters but
> instead some internal plugins (with configuration options) that hook in as
> filters.

To have an empty <unknown> filter without any setting makes no sense to me. I would put an empty list there. But this is just a view.

Worse is that filtering and mail fetcher resources actually have synchronization problems about four years (!) according to the creation time of this report.

I can provide logs or any information I'm able to gather from my system for analyzing. I'm not able to solve this.
Comment 138 René Krell 2015-09-24 13:14:36 UTC
Once more for KMail + Akonadi 15.08.1 regarding the hanging Mail Filter Agent progressbar at 0%: The progressbar hanging forever (along with just having <unknown> as filter) disappeared after:
akonadictl stop
akonadictl start
Comment 139 Christian Boltz 2015-09-24 16:50:13 UTC
Renè, the workaround from comment #118 still works for me: Add a filter like "From contains @" and let it execute "/usr/bin/sleep 0.5" (the time you need may vary, 0.3 wasn't enough for me).
Comment 140 Hussam Al-Tayeb 2015-09-24 16:58:28 UTC
(In reply to Christian Boltz from comment #139)
> Renè, the workaround from comment #118 still works for me: Add a filter like
> "From contains @" and let it execute "/usr/bin/sleep 0.5" (the time you need
> may vary, 0.3 wasn't enough for me).

Should I add that at the top so it gets executed before other filters?
Thank you.
Comment 141 Tobias Leupold 2015-09-24 17:02:53 UTC
I have it as my first filter, executed on each incoming mail (but I'm also using POP3, not IMAP – just in case this has an effect).
Comment 142 Hussam Al-Tayeb 2015-09-24 17:06:58 UTC
That breaks execution of other filters. I want it to still get filtered to 
Inbox/Bugs/KDE
Comment 143 René Krell 2015-09-25 07:05:26 UTC
(In reply to Hussam Al-Tayeb from comment #142)
> That breaks execution of other filters. I want it to still get filtered to 
> Inbox/Bugs/KDE

It should not break any other filters if you uncheck Advanced -> "If this filter matches stop processing here".
I created such a filter with the simple condition "All messages" and give it a try.
Thanks for the hint. Should be seriously solved anyway.
Comment 144 Eike Sauer 2015-09-25 17:23:49 UTC
Created attachment 94721 [details]
signature.asc

Am Donnerstag, 24. September 2015, 17:06:58 schrieben Sie:
> https://bugs.kde.org/show_bug.cgi?id=283682
> 
> --- Comment #142 from Hussam Al-Tayeb <hussam@visp.net.lb> ---
> That breaks execution of other filters. I want it to still get filtered to
> Inbox/Bugs/KDE

Obviously you need to deactivate "stop here if filter applies" on the extended 
tab.
Comment 145 JDiel 2015-09-30 05:21:47 UTC
Created attachment 94779 [details]
attachment-14215-0.dat

Dear,

I've a case of a mail, which reproducibly and consistently creates duplicates, 
and a lot of them.
When I try to remove duplicates (<CTRL> *>), I get an error message.
I've attached the screenshot of the error message and the mail file itself 
(after anonymising my mail address), maybe that helps for further analysis.
This mail file resides here: 
~/.local/share/akonadi_maildir_resource_0/inbox/new/
and the creation of duplicates ends only, when I manually delete this file.

Am Freitag, 25. September 2015, 17:23:49 schrieben Sie:
> https://bugs.kde.org/show_bug.cgi?id=283682
> 
> --- Comment #144 from Eike Sauer <eikesauer@t-online.de> ---
> 
> Am Donnerstag, 24. September 2015, 17:06:58 schrieben Sie:
> > https://bugs.kde.org/show_bug.cgi?id=283682
> > 
> > --- Comment #142 from Hussam Al-Tayeb <hussam@visp.net.lb> ---
> > That breaks execution of other filters. I want it to still get filtered to
> > Inbox/Bugs/KDE
> 
> Obviously you need to deactivate "stop here if filter applies" on the
> extended tab.
Comment 146 JDiel 2015-09-30 05:22:15 UTC
Created attachment 94780 [details]
duplicateMailExample.png
Comment 147 Graeme Hewson 2015-09-30 06:12:46 UTC
It's best to put error messages in text form for easy reference.

The title of the popup message is "Fehler beim Löschen von doppelten Nachrichten – Kontact", which Google translates as "Failed to delete duplicate messages".

The content of the message is "Unable to fetch item from backend (collection 186) : Unable to retrieve item from resource: Ungļltigen Eintrag erhalten".
Comment 148 megalithic 2015-11-05 22:00:56 UTC
Hello, I'm getting this problem on KMail Version 4.14.9
The 'sleep 0.5' command workaround didn't work for me.
However I have found that it only seems to happen when moving messages into mail folders that I imported (via Evolution 3 import from my previous maildir files.)

If I create a new folder and filter into that then the new messages move OK and stop reappearing back in the Inbox.

The problem also occurs on a manual move into an imported mail folder - the mail moves OK but pops up again in the Inbox

But again this doesn't happen with a newly created mail folder

Let me know if you want any debug information from these 'funny' folders - let me know what I need to do.
Thanks
Comment 149 Eric Depagne 2015-11-06 07:09:05 UTC
Filtering issues disappeared with kontact version 5.0.2 shipped with Kubuntu 15.10
Comment 150 Wolfgang Mader 2015-12-25 19:57:39 UTC
I experienced the described issue for all akonadi/kmail versions which are ported to frameworks. The system I am using is Arch with the following (relevant) packages installed

- plasma-desktop: 5.5.1
- plasma-framework: 5.17
- framework components 5.17
- kde applications: 15.12, including kmail and akonadi

Moreover, as a dependency of digikam, the following kde 4 stuff is there
- kdebase-runtime: 15.12
- kdebase-lib: 15.12
- kdepimlibs4: 4.14.10 
- libkipi4: 15.08
- libakonadi-qt4: 1.13.0
- libbaloo4: 4.14.3

I configure two generic imap resource in kmail. The first points to the mail server of my university, the second to the one of my web provider. I do not know what imap server they are running, but likely they are running different software. Thus, I guess, the issue is not with the server but with the client side. I do not configure anything special like server side subscription or sieve script.

On initial population of the mailboxes, everything seem fine. All messages are there and readable. The trouble starts as soon as I start to move around mails manually or via kmail filters or delete messages. Say, I move a message from the Inbox to the Private folder. The message disappears in the Inbox and seems to be moved to Privat. However, the message is not really moved. I is still in the Inbox and reappears the next time kmail refreshes the Inbox. The mail is grayed in the Inbox (everything is good with the messages in Private). In the Inboxr, I can not click on it, move or delete it. It just sits there, grayed out, not touchable. Now, using the web interface of my universities mail system, the mail is marked as deleted in the Inbox, but still there. At least, using the web interface, I can select the massage and delete it for real. Then, also in kmail, the message disappears. Unfortunately, I can not properly name the state of the deleted message in "imap"-terms.

On top of this issue, moved messages start to appear in the target folder multiple times. There is always one instance of the message which is clickable and good, but with time, multiple grayed out versions start to accumulate. They seem to be in the same state as the moved one in the source folder.

If there is any more information I can provide, do not hesitate to drop me a mail.

Reproducible: Always Steps to Reproduce:
1. Create a Imap resource in framework based kmail
2. Move or delete messages
3. Refresh mailbox and observe grayed out, uncklickable messages

Actual Results: Messages are not moved or deleted from mailbox.

Expected Results: Messages should be moved or deleted from mailbox.
Comment 151 Wolfgang Mader 2016-01-08 12:01:23 UTC
Dear devs,

would if help if you had access to a mailbox showing this issue. If so, let me know, and I see what I can do.
Comment 152 Wolfgang Mader 2016-01-11 08:28:17 UTC
I see the same bug in trojita. Do trojita and kmail2 share code?
Comment 153 Viorel-Cătălin Răpițeanu 2016-02-10 11:21:19 UTC
I'm having the same problem on Kontact 5.1.1. For every filtered mail I receive, I get 19 more duplicates. This is getting absurd.
Comment 154 Viorel-Cătălin Răpițeanu 2016-02-10 11:21:55 UTC
I'm having the same problem on Kontact 5.1.1. For every filtered mail I receive, I get 19 more duplicates. This is getting absurd.
Comment 155 Mike Goodwin 2016-04-17 15:47:47 UTC
I just got hit by this.... Duplicated 17,000 messages, all did was hit the "apply all filters on folder" button, and then noticed that my phone was freaking out due to all the dupes. 

kmail-libs-4.14.10-10.fc23.x86_64
baloo-akonadi-4.14.3-5.fc23.x86_64
akonadi-mysql-1.13.0-20.fc23.x86_64
akonadi-1.13.0-20.fc23.x86_64
kdepimlibs-akonadi-4.14.10-8.fc23.x86_64
kmail-4.14.10-10.fc23.x86_64

On Fedora 23. 

Not happy, rest of the day is going to be fixing this mess
Comment 156 Christian Hilberg 2016-07-11 11:13:38 UTC
Any updates on this regarding KF5-based KDEPIM 16.04? Has the issue been resolved?
Comment 157 Dennis Andress 2016-07-12 15:49:58 UTC
I had this bug a couple of years ago. It is more like a feature that can be turned on and off.

Tools>Configure KMail>Accounts>Select Account>Modify
On the Advanced tab check Automatically compact folders (expunges deleted messages)
Comment 158 Christian Hilberg 2016-07-15 08:44:46 UTC
@Dennis: Hum, that setting does not change anything for me regarding dupes. Even when activated, the dupes appear.

Confirming that the bug persists for me (newly created IMAP kmail account, newly created filters) as of kmail 5.2.2, KDE SC 16.04.02, KF 5.23.0, on Debian/Sid.
Comment 159 Eric Depagne 2016-07-15 18:40:09 UTC
The activities, once you get used to use them do exactly what is missing. Unvotting for this bug and removing my email from the list.
Comment 160 Christian Hilberg 2016-07-21 07:02:46 UTC
@Eric: While it might be true that activities do the trick for mail filtering, I guess we're talking about a bug in the existing Kmail2 mail filter functionality.

Testing with Kmail 5.2.3 (KDEPIM 16.04.3) on Debian/Sid shows a slightly different behaviour. Simple "move to" filters do not duplicate any filtered emails for me any longer.

However, messages filtered with the "pipe through" filter, as used by the anti-spam filters do get duplicated inside the destination folder if that folder resides on my IMAP server. If the destination folder is a local maildir, then the duplication does not occur.
Comment 161 Christian Hilberg 2016-07-21 08:12:41 UTC
Okay, there is still some duplication in KMail 5.2.3 with the pipe-through ... if the filtered message is *not* spam, then it gets dupe'd inside the folder it resides, typically INBOX.
Comment 162 Joachim Wagner 2016-07-21 08:48:32 UTC
Note that testing for a few hours might not be enough. Last time I tried, the bug only showed after 2-3 weeks of daily use. That's also why I'm hesitant about trying it again.

It would be nice if somebody with good knowledge of the inner working of KMail reviewed the workarounds suggested in this thread and posted a commented summary and recommendations for users (and possibly recommendations for developers).
Comment 163 Eric Depagne 2016-07-21 11:48:58 UTC
(In reply to Christian Hilberg from comment #160)
Hi Christian, 
I am a bit embarrassed, because I answered the wrong bug report. Thus, my message has really nothing to do with this one!
Sorry about that.
Éric.

> @Eric: While it might be true that activities do the trick for mail
> filtering, I guess we're talking about a bug in the existing Kmail2 mail
> filter functionality.
> 
> Testing with Kmail 5.2.3 (KDEPIM 16.04.3) on Debian/Sid shows a slightly
> different behaviour. Simple "move to" filters do not duplicate any filtered
> emails for me any longer.
> 
> However, messages filtered with the "pipe through" filter, as used by the
> anti-spam filters do get duplicated inside the destination folder if that
> folder resides on my IMAP server. If the destination folder is a local
> maildir, then the duplication does not occur.
Comment 164 Christian Hilberg 2016-09-09 06:59:30 UTC
(In reply to Eric Depagne from comment #163)
> (In reply to Christian Hilberg from comment #160)
> Hi Christian, 
> I am a bit embarrassed, because I answered the wrong bug report. Thus, my
> message has really nothing to do with this one!

Seems there has been some confusion about local mail filtering inside Kmail and the general idea of "activities" (which I haven't fully grokked myself, still need to find my own use case for that). So...

> Sorry about that.

...nothing to worry about! :-)

> Éric.

Christian

> > @Eric: While it might be true that activities do the trick for mail
> > filtering, I guess we're talking about a bug in the existing Kmail2 mail
> > filter functionality.
> > 
> > Testing with Kmail 5.2.3 (KDEPIM 16.04.3) on Debian/Sid shows a slightly
> > different behaviour. Simple "move to" filters do not duplicate any filtered
> > emails for me any longer.
> > 
> > However, messages filtered with the "pipe through" filter, as used by the
> > anti-spam filters do get duplicated inside the destination folder if that
> > folder resides on my IMAP server. If the destination folder is a local
> > maildir, then the duplication does not occur.
Comment 165 Christian Hilberg 2016-09-09 07:06:43 UTC
Update to behaviour of Kmail 5.2.3 (KDEPIM 16.04.3) on Debian/Testing:

Currently, I do not get any more dupes while pipe-through-filtering.

Changes in setup (now: working, before: dupes):

* IMAP now Dovecot (instead of ye olde Binc IMAP used before)
* Filtering now spamc+daemon (instead of spamassassin-perl before)

The spambayes filter is used in addition, which is the same as before.
Seems the bug could be triggered depending on the IMAP server implementation used. I tend to rule out the spamassassin-perl vs. smapc+daemon thing, as I've played with manually injecting sleeps at various steps in the filter processing, to no avail.
Comment 166 megalithic 2016-09-09 10:55:47 UTC
Hello, it happens for me on mail coming in via POP so not IMAP related in my case - more details of my experience above on  Comment 148 megalithic 2015-11-05 22:00:56 UTC
Comment 167 ianseeks 2017-03-21 16:13:00 UTC
(In reply to megalithic from comment #166)
> Hello, it happens for me on mail coming in via POP so not IMAP related in my
> case - more details of my experience above on  Comment 148 megalithic
> 2015-11-05 22:00:56 UTC

Try uncheck the "Apply filter to incoming messages" on each filter.  Now just click on the inbox and click 'Folders/Apply all filters on Folder'.  It has so far not produced a duplicate email for so its looking like filters on incoming emails is the culprit.
Comment 168 Daniel Vrátil 2017-03-22 00:02:57 UTC
*** Bug 322126 has been marked as a duplicate of this bug. ***
Comment 169 Daniel Vrátil 2017-03-22 00:03:16 UTC
*** Bug 377344 has been marked as a duplicate of this bug. ***
Comment 170 Daniel Vrátil 2017-03-22 00:04:12 UTC
*** Bug 322676 has been marked as a duplicate of this bug. ***
Comment 171 Daniel Vrátil 2017-03-22 00:04:44 UTC
*** Bug 293918 has been marked as a duplicate of this bug. ***
Comment 172 Daniel Vrátil 2017-03-22 00:06:11 UTC
*** Bug 323980 has been marked as a duplicate of this bug. ***
Comment 173 Daniel Vrátil 2017-03-22 00:06:38 UTC
*** Bug 362398 has been marked as a duplicate of this bug. ***
Comment 174 Daniel Vrátil 2017-03-22 00:10:48 UTC
*** Bug 333035 has been marked as a duplicate of this bug. ***
Comment 175 Daniel Vrátil 2017-03-22 00:11:31 UTC
*** Bug 346695 has been marked as a duplicate of this bug. ***
Comment 176 megalithic 2017-03-22 15:11:28 UTC
(In reply to ianseeks from comment #167)
> (In reply to megalithic from comment #166)
> > Hello, it happens for me on mail coming in via POP so not IMAP related in my
> > case - more details of my experience above on  Comment 148 megalithic
> > 2015-11-05 22:00:56 UTC
> 
> Try uncheck the "Apply filter to incoming messages" on each filter.  Now
> just click on the inbox and click 'Folders/Apply all filters on Folder'.  It
> has so far not produced a duplicate email for so its looking like filters on
> incoming emails is the culprit.

Hello - if I do that my filters won't work (will they)? Do you want me to try a test?

In my case I only get duplicates when moving messages into some subfolders and not others. I also only get duplicates on some messages. I can send a screen shot of this. Interestingly the two duplicate/filtered messages come through with different file sizes. eg the first is 5.1KB and the second is 8.7KB. But they are the same message duplicated with the same time stamp. (not sent twice accidentally)
Comment 177 ianseeks 2017-03-22 17:24:40 UTC
(In reply to megalithic from comment #176)
> (In reply to ianseeks from comment #167)
> > (In reply to megalithic from comment #166)
> > > Hello, it happens for me on mail coming in via POP so not IMAP related in my
> > > case - more details of my experience above on  Comment 148 megalithic
> > > 2015-11-05 22:00:56 UTC
> > 
> > Try uncheck the "Apply filter to incoming messages" on each filter.  Now
> > just click on the inbox and click 'Folders/Apply all filters on Folder'.  It
> > has so far not produced a duplicate email for so its looking like filters on
> > incoming emails is the culprit.
> 
> Hello - if I do that my filters won't work (will they)? Do you want me to
> try a test?

The filters won't work as the emails come into the Inbox.  Wait until the download of emails has finished then you click ont he Inbos foler to highlight it and then select "Apply All Filters on Folder" from the Folder menu (or add this option to the tool bar)
 
> In my case I only get duplicates when moving messages into some subfolders
> and not others. I also only get duplicates on some messages. I can send a
> screen shot of this. Interestingly the two duplicate/filtered messages come
> through with different file sizes. eg the first is 5.1KB and the second is
> 8.7KB. But they are the same message duplicated with the same time stamp.
> (not sent twice accidentally)

I didn't get duplicates on all emails either but i don't get any duplicates at all now because of the way i do it as described.  
Its just temporary until the devs fix filtering, there are a few more problems with filtering other than duplication of emails.
Comment 178 ianseeks 2017-03-22 17:27:35 UTC
Correction of last message typing screwup

"ont he Inbos foler" = on the Inbox folder
Comment 179 Andreas Mahel 2017-03-28 13:51:58 UTC
I'm experiencing email duplication after filter activity as well (openSUSE Tumbleweed, most recent versions).

The filters move mainly mailing list traffic from a POP3 account into local sub folders.

Duplication happens both for automatic filtering as well as executing the filters manually.

In most cases, the duplicated entry shows a different size than the original one (but not always).

On top, selecting any of the two entries will result in Kmail endlessly waiting for Akonadi.

A akonadictl fsck and vacuum does not remove the duplicates; also a "Remove duplicate emails" from the "Folder" menu does'nt, and ends with an error message.

In the meantime I have deleted the Akonadi database several times, as well as re-created the filters manually, to no avail.

As a workaround for myself I've written a short SQL script to show / delete the offending rows from the database (from within akonadiconsole) - this is tedious, but at least allows me to read all mails.

(How) can I gather more information about what exactly is going on? Both Kmail and Akonadi are kind of cryptic in their output - is there a way to increase verbosity?
Comment 180 ianseeks 2017-04-02 18:15:20 UTC
(In reply to Andreas Mahel from comment #179)
> I'm experiencing email duplication after filter activity as well (openSUSE
> Tumbleweed, most recent versions).
> 
> The filters move mainly mailing list traffic from a POP3 account into local
> sub folders.
> 
> Duplication happens both for automatic filtering as well as executing the
> filters manually.
> 
> In most cases, the duplicated entry shows a different size than the original
> one (but not always).
> 
> On top, selecting any of the two entries will result in Kmail endlessly
> waiting for Akonadi.
> 
> A akonadictl fsck and vacuum does not remove the duplicates; also a "Remove
> duplicate emails" from the "Folder" menu does'nt, and ends with an error
> message.
> 
> In the meantime I have deleted the Akonadi database several times, as well
> as re-created the filters manually, to no avail.
> 
> As a workaround for myself I've written a short SQL script to show / delete
> the offending rows from the database (from within akonadiconsole) - this is
> tedious, but at least allows me to read all mails.
> 
> (How) can I gather more information about what exactly is going on? Both
> Kmail and Akonadi are kind of cryptic in their output - is there a way to
> increase verbosity?

Since I posted my change to manual filtering, i've only had 3 emails duplicated so there is still a problem but not as bad as it was on filtering on incoming messages.
Comment 181 Frits Spieker 2017-04-05 07:34:54 UTC
(In reply to Andreas Mahel from comment #179)
> I'm experiencing email duplication after filter activity as well (openSUSE
> Tumbleweed, most recent versions).
> 
> The filters move mainly mailing list traffic from a POP3 account into local
> sub folders.
> 
> Duplication happens both for automatic filtering as well as executing the
> filters manually.
> 
> In most cases, the duplicated entry shows a different size than the original
> one (but not always).
> 
> On top, selecting any of the two entries will result in Kmail endlessly
> waiting for Akonadi.
> 
> A akonadictl fsck and vacuum does not remove the duplicates; also a "Remove
> duplicate emails" from the "Folder" menu does'nt, and ends with an error
> message.
> 
> In the meantime I have deleted the Akonadi database several times, as well
> as re-created the filters manually, to no avail.
> 
> As a workaround for myself I've written a short SQL script to show / delete
> the offending rows from the database (from within akonadiconsole) - this is
> tedious, but at least allows me to read all mails.
> 
> (How) can I gather more information about what exactly is going on? Both
> Kmail and Akonadi are kind of cryptic in their output - is there a way to
> increase verbosity?

Exactly as you describe. Sometimes I can get things going again for a short time by doing a restart of akonadi, but lately more often than not this causes akonadi to crash.

Weird thing though; in some sub-folders the "delete duplicates" functionality works fine, in other sub-folders (underneath the same account) it will indeed hang everything solid with an error message at the end, just like you describe.
Comment 182 Martin Koller 2017-04-05 07:53:00 UTC
Just for the records, I once tried to dig into the Akonadi code to find the bug.
I put my analysis to the other bug report, which is now set to a duplicate of this.
https://bugs.kde.org/show_bug.cgi?id=362398#c5
Just that it does not get lost, and I still think my findings are correct, here is it again:

I'm hunting the "duplicate mail bug" and I think I've found a bug in the Akonadi Server.
What happens here is the following:
- a resource adds a new mail item (id=3, collection=17)
- Akonadi sends "Item added" to the filter agent
- filter agent sends a "move" command to move item id=3 from collection=17 to collection=19
- Akonadi sends "item added" to the maildir resource (this is the first "added" message from the creation, so the
  item still has id=3 collection=17)
- I'd say in the Akonadi DB the collectionId of item=3 is now 19 (after the move)
- Now the maildir resource gets triggered by its internal file watcher and triggers a sync
- maildir gets called with retrieveItems and lists what it has on disc, which is the item (id=3) in the collection=17
- Akonadi receives this and inside Merge::parseStream() there the bad thing happens:
  It hits this "if" branch (server/src/handler/merge.cpp Line 301)
      // No item with such GID/RID exists, so call AkAppend::insert() and behave
      // like if this was a new item
 and INSERTS the item AGAIN into collection=17 with a new id=4
 Now I have the same mail item twice in the DB
 Since the maildir "itemsRetrieved" answer contains for the first time the RID, the newly inserted item gets this RID,
 and the previously intserted one id=3 does not and is left alone with a NULL RID


Someone more knowledgeable about the Akonadi internals should have a look, please.
Comment 183 BingMyBong 2017-04-05 10:00:26 UTC
On Wednesday, 5 April 2017 08:53:00 BST you wrote:
> https://bugs.kde.org/show_bug.cgi?id=283682
> 
> --- Comment #182 from Martin Koller <kollix@aon.at> ---
> Just for the records, I once tried to dig into the Akonadi code to find the
> bug.
> I put my analysis to the other bug report, which is now set to a duplicate
> of this.
> https://bugs.kde.org/show_bug.cgi?id=362398#c5
> Just that it does not get lost, and I still think my findings are correct,
> here is it again:
> 
> I'm hunting the "duplicate mail bug" and I think I've found a bug in the
> Akonadi Server.
> What happens here is the following:
> - a resource adds a new mail item (id=3, collection=17)
> - Akonadi sends "Item added" to the filter agent
> - filter agent sends a "move" command to move item id=3 from collection=17
> to collection=19
> - Akonadi sends "item added" to the maildir resource (this is the first
> "added" message from the creation, so the
>   item still has id=3 collection=17)
> - I'd say in the Akonadi DB the collectionId of item=3 is now 19 (after the
> move)
> - Now the maildir resource gets triggered by its internal file watcher and
> triggers a sync
> - maildir gets called with retrieveItems and lists what it has on disc,
> which is the item (id=3) in the collection=17
> - Akonadi receives this and inside Merge::parseStream() there the bad thing
> happens:
>   It hits this "if" branch (server/src/handler/merge.cpp Line 301)
>       // No item with such GID/RID exists, so call AkAppend::insert() and
> behave
>       // like if this was a new item
>  and INSERTS the item AGAIN into collection=17 with a new id=4
>  Now I have the same mail item twice in the DB
>  Since the maildir "itemsRetrieved" answer contains for the first time the
> RID, the newly inserted item gets this RID,
>  and the previously intserted one id=3 does not and is left alone with a
> NULL RID
> 
> 
> Someone more knowledgeable about the Akonadi internals should have a look,
> please.

Good work.
You could post this to kdepim-users list as Daniel Vrátil, who i think is the 
prime dev on akonadi, appears to frequent this list.  Or if you are not a 
member i can forward for you.
Comment 184 Martin Koller 2017-04-05 10:11:21 UTC
In fact I sent this information back then directly to Daniel but he obviously did not have time to look into it or does not see it as important.
Comment 185 graham 2017-04-05 19:07:12 UTC
(In reply to Martin Koller from comment #182)

Martin... if that's the case, then this "duplicates" issue would _only_ happen for messages being auto-filtered, correct?  Instead, if messages were first picked up and dropped into a folder and then _manually_ filtered, the events would reorder such that the "trigger the filter, and move the msg" doesn't happen until after the maildir resource had finished accepting the message.

(just wrapping my brain around it, and wanted to confirm that the above sounds reasonable)
Comment 186 Joachim Wagner 2017-04-06 09:13:36 UTC
As to "manually filtered" (comment #185), please distinguish between "manually applied filters (menu: Message --> Apply Filter)" and "manually filtered by moving messages between folders (menu: Message --> Move Message To)".

Some comments report duplication even in the later case.
Comment 187 Andreas Mahel 2017-04-06 11:28:14 UTC
As per my comment above (comment #179), it would be "running the filters manually (via Ctrl-J)". I haven't yet reverted to not using filters at all and dispatching several mailing lists to their respective sub folders by using the "Move to folder" functionality.

I can also confirm that automatic filtering leads to considerably more duplicate entries than manual filtering.

I have set up a VM with a simple mail setup (one POP3 resource receiving messages from 3 mailing lists), and with automatic filtering, the issue was clearly visible with about 10 to 20% of mails affected. I have reset that one now, and switched to manual filtering, to see if I catch any duplicates here as well.
Comment 188 ianseeks 2017-04-06 15:37:40 UTC
For me, manual filtering means using the "Apply All Filters on Folder" menu item or as i have done, put it as a button on the toolbar.
Comment 189 megalithic 2017-04-07 09:42:24 UTC
I've switched to filtering after the messages come in by clicking 'Folder - Apply All filters on Folder' on the Inbox. 

Just to confirm this has totally stopped the duplication in my case. 
Great to see we might be getting somewhere on this - thanks folks
Comment 190 Frits Spieker 2017-05-03 17:29:12 UTC
But is one of the developers actually looking at this? The duplication of the messages, combined with the akonadi hanging itself out to dry when deleting messages (resulting in the famous "retrieving folder content" screen) makes KMail basically unusable. I'm really baffled how such a crucial piece of software (unless developers use other program to read email) can be so broken for so long.
Comment 191 Martin Koller 2017-05-03 17:34:59 UTC
I am currently looking into this and have already implemented some patches which hopefully improve the situation (although I have no final fix yet).
E.g. https://phabricator.kde.org/D5667
It's very complicated and has to do with timing of messages between 3 processes ...
Comment 192 Frits Spieker 2017-05-03 18:48:02 UTC
Excellent! Good to know there is some progress.
Comment 193 graham 2017-05-03 20:12:51 UTC
Agreed!  Exciting to see progress, as KMail is close to unusable at this point.

MUCH appreciated!
Comment 194 BingMyBong 2017-05-04 10:29:42 UTC
(In reply to Martin Koller from comment #191)
> I am currently looking into this and have already implemented some patches
> which hopefully improve the situation (although I have no final fix yet).
> E.g. https://phabricator.kde.org/D5667
> It's very complicated and has to do with timing of messages between 3
> processes ...

Thanks Martin, your work on this and other bugs will be most appreciated by lots of users
Comment 195 Alexandre Bonneau 2017-09-01 23:43:18 UTC
Kmail version 5.2.3 (in Debian Stable) is definitely showing this bug (see the attached screenshot).
Comment 196 Alexandre Bonneau 2017-09-01 23:46:29 UTC
Created attachment 107642 [details]
Duplicated mails in Kmail 5.2.3
Comment 197 Allan Sandfeld 2017-09-26 09:56:28 UTC
Just hit the same. I noted that the duplicated messages, were messages never fully moved in the first place. They ghosted in the original folder, and the ghosts were copied.

The folders this happens to also does not have working folder properties. Nothing you change in folder properties have ANY effect and are immediately lost
Comment 198 Allan Sandfeld 2017-09-26 09:56:41 UTC
Just hit the same. I noted that the duplicated messages, were messages never fully moved in the first place. They ghosted in the original folder, and the ghosts were copied.

The folders this happens to also does not have working folder properties. Nothing you change in folder properties have ANY effect and are immediately lost
Comment 199 Árpi 2018-01-12 18:40:05 UTC
With upgrading to version 5.5.3 on Kmail, the duplication problem got worse in my case.
I have a rather large (~ 10000) collection of messages. It happens after filtering, it can be "fixed" by clearing akonadi cache with akonadiconsole, but it appears again.
Adding a filter on top of the filter list, that executes "/usr/bin/sleep 15" did not help. I could not find out if it happens with the folders that contain many messages, or in all of them (i think the latter).
What did change with the new version (compared to the one there was in kubuntu 14.04) is that now I get an error message when trying to access the folder with the duplicate messages, that kmail cannot fetch the message because of multiple candidates. Previously, i could erase one.
Comment 200 DeMus 2018-03-13 17:37:39 UTC
I can confirm this bug with both an IMAP and a POP account. Mails are sometimes (not always, filters don't always work well) transferred from the inbox to the correct folder and a little bit later the mails are also back in the inbox. This can go on forever.
When I delete them from the inbox it is over and they don't appear anymore.
Comment 201 Roger Larsson 2018-04-07 09:19:21 UTC
I have also seen this (and my wife on her setup - similar to mine)
and I wonder if some of this is due to the huge number of options and the more complex environment.

I want to be able to read mails both on my computer and on my mobile.
I want the computer to be the primary storage of my mails.
POP/IMAP account - using IMAP for both

Computer
- copied new messages locally using filters into correct folder
- filters that mark a message, resulted in two copies (one shadow - not readable) the updated state could be seen from mobile too...
- had "download for offline processing" enabled (trying with it off now)
   noticed that it 
- push on
- recheck 5 minutes
...

I wonder if my settings caused numerous rereads, store on disk with different generations of each mail...

-rw-r--r-- 1 roger users     4589 11 mar 00.20 /home/roger/.local/share/akonadi/file_db_data/99/822899_r0
-rw-r--r-- 1 roger users     4100  6 apr 18.29 /home/roger/.local/share/akonadi/file_db_data/99/827899_r61
-rw-r--r-- 1 roger users     6372  6 apr 18.29 /home/roger/.local/share/akonadi/file_db_data/99/828199_r1360
-rw-r--r-- 1 roger users     4348  6 apr 18.30 /home/roger/.local/share/akonadi/file_db_data/99/831799_r1359
-rw-r--r-- 1 roger users     4126  6 apr 18.30 /home/roger/.local/share/akonadi/file_db_data/99/833599_r61
Comment 202 Tristan Miller 2019-04-23 06:44:58 UTC
Is this problem still reproducible in the latest version of KMail?  There haven't been any comments on this bug in over a year; I'm wondering if that's because the problem has been fixed or because it has driven most of the user base away.  I'd love to return to using KMail but can't as long as this message duplication problem persists.  (I was getting dozens of duplicated messages every day, which was too much to put up with.)
Comment 203 Eric Depagne 2019-04-23 06:55:02 UTC
(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail?  There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away.  I'd love to return to using KMail but can't as long as
> this message duplication problem persists.  (I was getting dozens of
> duplicated messages every day, which was too much to put up with.)

It has been fixed I believe, since it disappeared from my kmail in 2015.
Comment 204 Tristan Miller 2019-04-23 07:20:35 UTC
(In reply to Eric Depagne from comment #203)

> It has been fixed I believe, since it disappeared from my kmail in 2015.

And yet there are plenty of comments here from 2015 to 2018 claiming the problem still exists.  I would prefer to hear from someone who was experiencing the problem more recently (say, 2018) who can confirm that it is no longer reproducible.
Comment 205 Peter Humphrey 2019-04-23 07:30:45 UTC
(In reply to Eric Depagne from comment #203)
> (In reply to Tristan Miller from comment #202)
> > Is this problem still reproducible in the latest version of KMail?  There
> > haven't been any comments on this bug in over a year; I'm wondering if
> > that's because the problem has been fixed or because it has driven most of
> > the user base away.  I'd love to return to using KMail but can't as long as
> > this message duplication problem persists.  (I was getting dozens of
> > duplicated messages every day, which was too much to put up with.)
> 
> It has been fixed I believe, since it disappeared from my kmail in 2015.

It certainly hasn't. I still get numerous duplicates daily on this Gentoo box.

I think the comments dried up because (a) there was nothing new to add; (b) users grew tired of banging on to no avail.
Comment 206 Árpi 2019-04-23 07:50:17 UTC
I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3), that came with Ubuntu 18.04. Then I started using the workaround from comment #118.
Comment 207 Tristan Miller 2019-04-23 08:05:26 UTC
(In reply to Árpi from comment #206)
> I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3),
> that came with Ubuntu 18.04. Then I started using the workaround from
> comment #118.

Lucky you; that workaround did nothing for me.

Is there some sort of bug bounty program for KDE?  If so I wouldn't mind chipping in €20 or so towards a fix for this issue.
Comment 208 Graeme Hewson 2019-04-23 08:43:03 UTC
I haven't seen the problem for a while, maybe two years. I use POP3 only, not IMAP. I run Kubuntu; currently I'm on 19.04.

Sometimes kmail crashes, and I wonder if the way I clean up is related to my not seeing duplicate filtered messages. When kmail crashes, I stop akonadi ("akonadictl stop") and look in the maildir structure like this:

cd .local/share/.local-mail.directory
ls */new/*S

The S indicates messages which have been Seen, and according to the Maildir specification the files should not be in a"new" subdirectory; they should have been moved to a "cur" subdirectory. I therefore move them while akonadi is stopped. E.g.,

mv -i wastebin/new/*S wastebin/cur

I then run "akonadictl fsck" and restart akonadi ("akonadictl start").

As I say, it's pure speculation that this is why I don't see duplicate messages now.
Comment 209 Graeme Hewson 2019-04-23 08:46:08 UTC
Apologies.

akonadictl start
akonadictl fsck

is the correct order.
Comment 210 Alexandre Bonneau 2019-04-23 08:55:50 UTC
On my end, I'm currently using Kmail 5.9.3 (Debian buster), and am seeing no duplicates using imap only.
Some very rare times I can see a few duplicates, but they disappear by themselves pretty quicly.
Comment 211 Joachim Wagner 2019-04-23 09:31:57 UTC
(In reply to Tristan Miller from comment #202)
> [...] because the problem has been fixed or because it has driven most of
> the user base away.  I'd love to return to using KMail [...]

I didn't stop using KMail. I stopped using filters.

If I find time to explore alternatives and find something suitable I may switch.  Also possible I will use two mail clients in parallel.

Coincidentally, I am well positioned to test alternatives as I moved away from using folders under "Local Folders" to using a local imap server a while ago (because, at that time, KMail stored messages in a database instead of a maildir). This should make it easy to use multiple clients in parallel.
Comment 212 Frits Spieker 2019-04-23 13:23:33 UTC
(In reply to Eric Depagne from comment #203)
> (In reply to Tristan Miller from comment #202)
> > Is this problem still reproducible in the latest version of KMail?  There
> > haven't been any comments on this bug in over a year; I'm wondering if
> > that's because the problem has been fixed or because it has driven most of
> > the user base away.  I'd love to return to using KMail but can't as long as
> > this message duplication problem persists.  (I was getting dozens of
> > duplicated messages every day, which was too much to put up with.)
> 
> It has been fixed I believe, since it disappeared from my kmail in 2015.

Most definitely not fixed. Still run into it multiple times per day with some messages showing up 5 or more times. Sometimes I just delete the entire folder and filter and just set everything up again. Sometimes seems to do the trick. I have been unable to find any logic behind why some messages get duplicated where others don't (in the same folder).
Comment 213 Árpi 2019-04-23 13:29:45 UTC
(In reply to Tristan Miller from comment #207)
> (In reply to Árpi from comment #206)
> > I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3),
> > that came with Ubuntu 18.04. Then I started using the workaround from
> > comment #118.
> 
> Lucky you; that workaround did nothing for me.
> 
> Is there some sort of bug bounty program for KDE?  If so I wouldn't mind
> chipping in €20 or so towards a fix for this issue.

I checked my filters; what I am doing is not exactly as described in comment #118, but instead, I added an extra action, to execute
/bin/sleep 1
to each of my filters. Since then, no duplicates.
Comment 214 Tobias Leupold 2019-04-23 14:12:51 UTC
I've been using IMAP for some years now, and I don't get duplicates anymore when filtering. In the case my filters ever do filter, which is roughly the case in 80 % of the mails they should work on (otherwise, nothing happens).

After almost eight years and more than 200 comments, I think one should rather wonder if this even can be fixed inside this Akonadi monster I personally wouldn't even dare to look at the code, or if one should reconsider the whole thing.

Only thing I can't understand is that the KMail guys are not annoyed themselves by this ... I mean, I don't think they use Trojitá, do they?! But no harm meant, 
remarkably enough I still stick to KMail ;-)
Comment 215 BingMyBong 2019-04-23 14:31:48 UTC
(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail?  There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away.  I'd love to return to using KMail but can't as long as
> this message duplication problem persists.  (I was getting dozens of
> duplicated messages every day, which was too much to put up with.)

The way i reduced the duplicates to a point where it hardly happens (maybe once a week) is to stop filtering on incoming messages and only apply the filters manually once all emails are in. I'm using IMAP but I think the POP3 still has a lot of issues.
Comment 216 graham 2019-04-23 15:27:37 UTC
(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail?  There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away.

Driven away.

Was a KMail for what seems like forever (~2000?) (I've been a KDE user since it first came out in '96), but after dealing with workarounds and hacks for two+ years I eventually gave in and abandoned KMail back in mid-2017.

Took me multiple weeks to get all of my mail (~12GB) imported into Thunderbird, get my 200+ mail filters set up again, and to remove all of the duplicates and junk that various KMail bugs had done to my mail.  Was the most painful thing I'd done in years, and I knew the switch was going to suck bad so I held out as long as I could.

KMail eventually just got unusable, period.  I'd have to do workarounds and restart akonadi every single time I picked up my POP mail, which just made KMail effectively useless.  After realizing I was using Alpine to read all of my new mail and delete junk _before_ fetching it into KMail, I finally gave in and switched.

LOVED KMail, but THIS bug led me to finally throw in the towel and give it up.
Comment 217 ederag 2019-04-23 16:38:29 UTC
(In reply to graham from comment #216)
> Was a KMail for what seems like forever (~2000?) (I've been a KDE user since
> it first came out in '96), but after dealing with workarounds and hacks for
> two+ years I eventually gave in and abandoned KMail back in mid-2017.

There was a commit in 2017-05-04
https://phabricator.kde.org/R206:43f2cde61f98317eb13d98222a57bc6df323a308
that was about this bug.
And indeed in openSUSE Leap-15 (kdepim-runtime 17.12.3),
there are much fewer duplicates than before.

There is now a related bug about unreachable duplicates
that has already been reported as #388869.

(In reply to Tobias Leupold from comment #214)
> I think one should rather wonder if this even can be fixed 
> inside this Akonadi monster I personally wouldn't even dare to look at 
> the code, or if one should reconsider the whole thing.

kmail/akonadi handle my > 15GB maildir gracefully,
which is quite a performance. Congratulations !

So it is not perfect, but to me it is an amazing piece of software.
Please consider that it is very hard for maintainers
to fix a bug that is not reproducible on their setup.
Comment 218 ederag 2019-04-23 16:40:20 UTC
With a working link: bug #388869
Comment 219 Peter Humphrey 2019-04-23 17:10:17 UTC
> Please consider that it is very hard for maintainers
> to fix a bug that is not reproducible on their setup.

True, but there's nothing to stop them from enlisting help from knowledgeable users (at least a few have offered, including me). They won't do it though; I can only speculate as to why not.
Comment 220 linuxfan 2020-05-08 20:24:05 UTC
Version KMail 5.13.3 (19.12.3)
OS  Gentoo Linux

I have been plagued by this bug for a very long time. Today I found out, that for ages there has been an error message in my mysql/mariadb log-file:

[ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set ...

This is the first time I saw this error, as there is no other application than akonadi using mariadb on my pc and akonadi was never complaining. This mariadb-error could be fixed by running mysql_upgrade (google it to find an explanation, this is due to an incomplete upgrade of the mariadb-package)

After fixing the database, kmail has not been showing duplicated mails any longer. Also filters have been working flawlessly.
Comment 221 ederag 2020-05-09 07:50:50 UTC
(In reply to linuxfan from comment #220)
> for ages there has been an error message in my mysql/mariadb log-file:
> 
> [ERROR] Incorrect definition of table mysql.event: expected column
> 'sql_mode' at position 14 to have type set ...

Thanks for sharing.
In which log file exactly did you find this error ?

/var/log/mysql/ is empty here, so

pgrep -af mysqld
/usr/sbin/mysqld --defaults-file=/home/ederag/.local/share/akonadi/mysql.conf --datadir=/home/ederag/.local/share/akonadi/db_data/ --socket=/tmp/akonadi-ederag.Yqlbg1/mysql.socket --pid-file=/tmp/akonadi-ederag.Yqlbg1/mysql.pid


grep -1 "error log" /home/ederag/.local/share/akonadi/mysql.conf

# # error log file name, relative to datadir (default:hostname.err)
log_error=mysql.err

grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
<nothing>

That was with the "1 s" delay filter active,
just deactivated; we'll see.
Comment 222 Peter Humphrey 2020-05-09 08:32:49 UTC
(In reply to ederag from comment #221)

> grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
> <nothing>

Here, I see:

$ grep -i error .local/share/akonadi/db_data/mysql.err
2020-05-08 13:12:10 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2020-05-08 13:12:10 9 [ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not found.

I don't know how long that's been happening, because I haven't had any duplicates for quite a long time. I remember, though, that when I was getting them, that error log had numerous errors of various types.
Comment 223 BingMyBong 2020-05-09 10:18:15 UTC
(In reply to Peter Humphrey from comment #222)
> (In reply to ederag from comment #221)
> 
> > grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
> > <nothing>
> 
> Here, I see:
> 
> $ grep -i error .local/share/akonadi/db_data/mysql.err
> 2020-05-08 13:12:10 0 [ERROR] Can't open and lock privilege tables: Table
> 'mysql.servers' doesn't exist
> 2020-05-08 13:12:10 9 [ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not
> found.
> 
> I don't know how long that's been happening, because I haven't had any
> duplicates for quite a long time. I remember, though, that when I was
> getting them, that error log had numerous errors of various types.

I've been using postgresql and i've hardly seen any since but as of today i got duplicates by the bucket load.

opensuse:tumbleweed:20200506
Qt: 5.14.1 KDE Frameworks: 5.69.0 - KDE Plasma:  5.18.4 - kwin 5.18.4
kmail2 5.14.0 (20.04.0) - akonadiserver 5.14.0 (20.04.0) -
Comment 224 linuxfan 2020-05-09 10:31:40 UTC
the log file is /var/log/mysql/mysqld.err

However, digging deeper, the problem might be related to two instances of mariadb being present.

pgrep -af mysqld yields

2402 /usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf
2881 /usr/sbin/mysqld --defaults-file=/home/linuxfan/.local/share/akonadi/mysql.conf --datadir=/home/linuxfan/.local/share/akonadi/db_data/ --socket=/run/user/10001/akonadi/mysql.socket --pid-file=/run/user/1000/akonadi/mysql.pid

While /var/log/mysql/mysqld.err is now clear, in 

/home/linuxfan/.local/share/akonadi/db_data/mysql.err

I can see two further errors and some warnings:

[ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
 [Warning] Can't open and lock time zone table: Table 'mysql.time_zone_leap_second' doesn't exist trying to live without them
 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
 [Note] Reading of all Master_info entries succeeded
 [Note] Added new Master_info '' to hash table
 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.4.12-MariaDB'  socket: '/run/user/1000/akonadi/mysql.socket'  port: 0  Source distribution
[ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not found.

what is more important, I believe, is that akonadi seems to start the second mysql instance but some of the data is actually going to my main mysql server instead and its data directory /home/mysql_data/ while other parts are ending up in the akonadi-instance of mysql. 

After disabling the stand-alone server, I found that  ~/.config/akonadi/akonadiserverrc

still contained the line

Options="UNIX_SOCKET=/var/run/mysql/mysqld.socket"

pointing to the main server socket. Clearing this up, everything seems to end up in the akonadi-version of mysql (with the above warnings/errors, not sure what that means). I am unsure, why some of the data still went to the akonadi-version of mysql, though.

From my point of view, I seem to have had some mess in the config files as I have been dragging/upgrading my /home/ directory with me since 1997 .... (and yes, I have been using KDE ever since the original beta1)
Comment 225 ederag 2020-05-09 19:41:05 UTC
Well, only half a day before a duplicate is back.
(again, from lists.gnu.org)
Same status as in comment #221.
And nothing suspicious in mysql.err (no ERROR).
Comment 226 Alexandre Bonneau 2020-05-10 20:24:30 UTC
(In reply to Alexandre Bonneau from comment #210)
> On my end, I'm currently using Kmail 5.9.3 (Debian buster), and am seeing no
> duplicates using imap only.
> Some very rare times I can see a few duplicates, but they disappear by
> themselves pretty quicly.

For info (I posted that info in another related issue already), I found out that I only get duplicates when I have 2 computers running akonadi-server.

For instance it happens when I'm on my main rig, then casually open my laptop. Then mails are duplicated a lot...until I close one of the akonadi server (usually the one from the laptop).
Comment 227 ederag 2021-01-16 23:32:25 UTC
(In reply to Alexandre Bonneau from comment #226)
> For info (I posted that info in another related issue already), I found out
> that I only get duplicates when I have 2 computers running akonadi-server.

Here also, two computers once concurrently ran akonadi 
with the same configuration (synchronized with unison).
But duplicates do appear, even when the other akonadi is stopped.
(pgrep akonadi yields nothing on the other computer)
[akonadictl 5.16.1 (20.12.0)]
Comment 228 r.kunschke 2022-04-27 08:18:45 UTC
The problem is still present in kmail for a long time. I have filters for using bogofilter in kmail. 
Is there any workaround or something? Or a filter for kmail which removes all duplicates?
And is there any explanation why this is happenning?
Comment 229 ederag 2022-04-27 08:57:26 UTC
(In reply to r.kunschke from comment #228)
> Is there any workaround or something? 

Adding this filter helped a lot here:
- match all messages
- Executed Command: /usr/bin/sleep 1,
Comment 230 Christian Hilberg 2022-04-27 09:08:24 UTC
Created attachment 148406 [details]
signature.asc

Am Mittwoch, 27. April 2022, 10:57:26 CEST schrieb ederag:
> https://bugs.kde.org/show_bug.cgi?id=283682
> 
> --- Comment #229 from ederag <edera@gmx.fr> ---
> (In reply to r.kunschke from comment #228)
> > Is there any workaround or something? 
> 
> Adding this filter helped a lot here:
> - match all messages
> - Executed Command: /usr/bin/sleep 1,

Depending on how quick the IMAP server is, this had
little to no effect for me when I tried, so YMMV.

(bye)^2
Comment 231 BingMyBong 2022-04-27 16:05:47 UTC
(In reply to r.kunschke from comment #228)
> The problem is still present in kmail for a long time. I have filters for
> using bogofilter in kmail. 
> Is there any workaround or something? Or a filter for kmail which removes
> all duplicates?
> And is there any explanation why this is happenning?

There is an option on the "Folder" menu to remove duplicates that works reasonably well.
I used to have this issue but its now pretty much gone since i changed from 1. POP3 to Imap and 2.  Mysql to Postgresql for the backend, its very very rare for me to get a duplicate now
Comment 232 Christoph Thielecke 2022-05-24 14:07:51 UTC
Still valid in akonadi 21.12.3 on Kubuntu 22.04.

I think akonadi should automatically handle this problem.
Comment 233 Peter Humphrey 2022-08-24 07:26:45 UTC
New symptom.

I have a local server that fetches POP3 mail from my ISP and serves it as IMAP4. I've had only rare duplicates since setting this arrangement up a few years ago.

I discovered yesterday that creating a KMail POP3 account led immediately to numerous duplicates, as half-expected, but the key point here is that duplicates kept occurring, even after I'd deleted the POP3 account and rebooted.

This is KMail version 5.20.3 (22.04.3) on Gentoo stable, Plasma version 5.24.6-r1. I tried postgres several years ago but it didn't improve anything, so I'm sticking with mariadb.