Version: unspecified (using KDE 4.7.3) OS: Linux When I check my e-mail accounts, sometimes, kmail2 re-downloads some old messages and signs it as "new". The number of duplicated messages may vary from 5 to 30. This happens often if there is a new e-mail on server. Registered accounts are: one POP3 and one IMAP. Most affected is the IMAP account. Reproducible: Sometimes Steps to Reproduce: Check new e-mail on registered accounts with at least one new e-mail to download. Actual Results: When kmail2 doubles messages, most of time is dedicated to deletion of duplicates. Expected Results: If one e-mail (and his attachment) is already present on mailbox, kmail2 should never re-download.
I can confirm a similar problem. To generate a test case for another problem I send 10 mails on my mail server with the following command to my account: for ((i=0; i<10 ; i++)) do echo "Test #`printf %4.4d $i`" | mail -s "Filter Test #`printf %4.4d $i`" thomas done Keep in mind that this command runs really fast and all messages have the same time stamp. I created a filter to move the messages into a Test folder. Every time I start "receive messages" 9 of 10 messages are received again. But they were deleted on the server with the first run. Restarting / crashing kmail2 does not help! Only message #0003 is received only once! But this messages was not filtered as it should. (Moved to Test folder) I added a special filter to included a time stamp 2X-Filter-Test" for the filtering which results in the following: Filter log entry [...] 23:36:04] Evaluating filter rules: (match all of the following) "Subject" <contains> "Filter Test" [23:36:04] 1 = "Subject" <contains> "Filter Test" (Filter Test #0005) [23:36:04] Filter rules have matched. [23:36:04] Applying filter action: Pipe Through "formail -A "X-Filter-Test: `date -R`"" [23:36:04] Evaluating filter rules: (match any of the following) "Subject" <start-with> "Filter" [23:36:04] 1 = "Subject" <start-with> "Filter" (Filter Test #0005) [23:36:04] Filter rules have matched. [23:36:04] Applying filter action: Move Into Folder "322" [...] A message looks after severals receives as following: ---BoM--- From thomas@arend-rhb.de Fri May 18 23: 32:07 2012 Return-Path: <thomas@arend-rhb.de> X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on k2.arend.tksd X-Spam-Status: No, score=-1.3 required=4.5 tests=AWL,BAYES_00,NO_RELAYS autolearn=ham version=3.3.1 X-Spam-Spammy: 0.956-5766--1937h-14398s--0d--HContent-Transfer-Encoding:7bit, 0.955-38--13h-95s--1d--0000 X-Spam-Hammy: 0.000-1987--32095h-1s--0d--H*p:D*arend-rhb.de, 0.001-2--18h-0s--1d--H*UA:mailx X-Original-To: thomas Delivered-To: thomas@arend-rhb.de Received: by k2.arend.tksd (Postfix, from userid 1000) id 6798F28B2F; Fri, 18 May 2012 23:06:23 +0200 (CEST) Date: Fri, 18 May 2012 23:06:23 +0200 To: thomas@arend-rhb.de Subject: Filter Test #0000 User-Agent: Heirloom mailx 12.2 01/07/07 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-ID: <20120518210623.6798F28B2F@k2.arend.tksd> From: Thomas Arend <thomas@arend-rhb.de> X-UIDL: `ZH!!@K3"!AdZ"!od4!! X-Filter-Test: Fri, 18 May 2012 23:35:06 +0200 X-Filter-Test: Fri, 18 May 2012 23:35:44 +0200 X-Filter-Test: Fri, 18 May 2012 23:35:51 +0200 X-Filter-Test: Fri, 18 May 2012 23:35:58 +0200 X-Filter-Test: Fri, 18 May 2012 23:36:03 +0200 X-Filter-Test: Fri, 18 May 2012 23:41:03 +0200 X-Filter-Test: Fri, 18 May 2012 23:42:51 +0200 X-Filter-Test: Fri, 18 May 2012 23:51:25 +0200 Test #0000 ---EoM--- So I conclude the messages are re-received after they were modified by the filter. The messages are received after each restart or crash of kmail2 and every time when the received cycle starts automatically.
Starting and stoping akonadi stoped the re-receiving issue. Unfortunately the behavior was not reproduceable by resending 10 messages.
Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia. Committed on 04/07/2012 at 17:35. Pushed by amantia into branch 'master'. A biggish refactoring of the filter functionality: 1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only). This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list) Might fix 293768. 2) Download only what is really needed for the filter and add safety checks that the mail is not modified if the requested part is missing. So far the distinction was only betwee the body and rest. This should speed up filtering. Also related to 287752, but should fix also 292283, 288109. 3) Rename qDebug->kDebug + fix some includes 4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates with empty content), eg. 275233 All bug reporters: in case you still see the bug, reopen the corresponding one. For the reports that were CC'd, but not closed, please try to reproduce and tell the result. So far the bug is fixed on master, but might be backported later to 4.9. Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 284310, bug 286364, bug 283682 M +7 -11 kmail/kmcommands.cpp M +10 -7 kmail/kmcommands.h M +1 -1 kmail/kmmainwidget.cpp M +1 -5 mailcommon/filter/filteraction.cpp M +6 -5 mailcommon/filter/filteraction.h M +6 -0 mailcommon/filter/filteractionaddheader.cpp M +2 -0 mailcommon/filter/filteractionaddheader.h M +2 -2 mailcommon/filter/filteractionaddtag.cpp M +1 -1 mailcommon/filter/filteractionaddtag.h M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h M +7 -0 mailcommon/filter/filteractionbeep.cpp M +1 -0 mailcommon/filter/filteractionbeep.h M +18 -1 mailcommon/filter/filteractioncopy.cpp M +7 -0 mailcommon/filter/filteractioncopy.h M +6 -0 mailcommon/filter/filteractiondelete.cpp M +1 -0 mailcommon/filter/filteractiondelete.h M +5 -0 mailcommon/filter/filteractionexec.cpp M +1 -0 mailcommon/filter/filteractionexec.h M +5 -0 mailcommon/filter/filteractionforward.cpp M +1 -0 mailcommon/filter/filteractionforward.h M +6 -0 mailcommon/filter/filteractionmove.cpp M +2 -0 mailcommon/filter/filteractionmove.h M +6 -0 mailcommon/filter/filteractionpipethrough.cpp M +1 -0 mailcommon/filter/filteractionpipethrough.h M +6 -0 mailcommon/filter/filteractionplaysound.cpp M +2 -0 mailcommon/filter/filteractionplaysound.h M +5 -0 mailcommon/filter/filteractionredirect.cpp M +1 -0 mailcommon/filter/filteractionredirect.h M +6 -1 mailcommon/filter/filteractionremoveheader.cpp M +1 -0 mailcommon/filter/filteractionremoveheader.h M +6 -0 mailcommon/filter/filteractionreplyto.cpp M +1 -0 mailcommon/filter/filteractionreplyto.h M +6 -0 mailcommon/filter/filteractionrewriteheader.cpp M +1 -0 mailcommon/filter/filteractionrewriteheader.h M +6 -0 mailcommon/filter/filteractionsendfakedisposition.cpp M +1 -0 mailcommon/filter/filteractionsendfakedisposition.h M +8 -2 mailcommon/filter/filteractionsendreceipt.cpp M +1 -0 mailcommon/filter/filteractionsendreceipt.h M +6 -0 mailcommon/filter/filteractionsetidentity.cpp M +1 -0 mailcommon/filter/filteractionsetidentity.h M +6 -0 mailcommon/filter/filteractionsetstatus.cpp M +1 -0 mailcommon/filter/filteractionsetstatus.h M +6 -0 mailcommon/filter/filteractionsettransport.cpp M +1 -0 mailcommon/filter/filteractionsettransport.h M +5 -0 mailcommon/filter/filteractionstatus.cpp M +2 -0 mailcommon/filter/filteractionstatus.h M +5 -2 mailcommon/filter/filteractionunsetstatus.cpp M +2 -1 mailcommon/filter/filteractionunsetstatus.h M +4 -4 mailcommon/filter/filterimporter/filterimporterevolution.cpp M +1 -1 mailcommon/filter/filterimporter/filterimporterprocmail.cpp M +1 -1 mailcommon/filter/filterimporter/filterimportersylpheed.cpp M +2 -2 mailcommon/filter/filtermanager.cpp M +2 -8 mailcommon/filter/filtermanager.h M +8 -2 mailcommon/filter/itemcontext.cpp M +8 -1 mailcommon/filter/itemcontext.h M +10 -17 mailcommon/filter/kmfilterdialog.cpp M +2 -1 mailcommon/filter/kmfilterdialog.h M +25 -8 mailcommon/filter/mailfilter.cpp M +4 -3 mailcommon/filter/mailfilter.h M +48 -20 mailcommon/searchpattern.cpp M +34 -11 mailcommon/searchpattern.h M +1 -1 mailcommon/tests/searchpatterntest.cpp M +78 -47 mailfilteragent/filtermanager.cpp M +10 -8 mailfilteragent/filtermanager.h M +8 -6 mailfilteragent/mailfilteragent.cpp M +3 -0 mailfilteragent/mailfilteragent.h http://commits.kde.org/kdepim/d64d61b470c56b92294f6adee6d74305a217628d
Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia. Committed on 04/07/2012 at 17:35. Pushed by amantia into branch 'KDE/4.9'. (Backport filter fix from master) A biggish refactoring of the filter functionality: 1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only). This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list) Might fix 293768. 2) Download only what is really needed for the filter and add safety checks that the mail is not modified if the requested part is missing. So far the distinction was only betwee the body and rest. This should speed up filtering. Also related to 287752, but should fix also 292283, 288109. 3) Rename qDebug->kDebug + fix some includes 4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates with empty content), eg. 275233 All bug reporters: in case you still see the bug, reopen the corresponding one. For the reports that were CC'd, but not closed, please try to reproduce and tell the result. So far the bug is fixed on master, but might be backported later to 4.9. Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 284310, bug 286364, bug 283682 (cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d) FIXED-IN: 4.9 M +7 -11 kmail/kmcommands.cpp M +10 -7 kmail/kmcommands.h M +1 -1 kmail/kmmainwidget.cpp M +1 -5 mailcommon/filter/filteraction.cpp M +6 -5 mailcommon/filter/filteraction.h M +6 -0 mailcommon/filter/filteractionaddheader.cpp M +2 -0 mailcommon/filter/filteractionaddheader.h M +2 -2 mailcommon/filter/filteractionaddtag.cpp M +1 -1 mailcommon/filter/filteractionaddtag.h M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h M +7 -0 mailcommon/filter/filteractionbeep.cpp M +1 -0 mailcommon/filter/filteractionbeep.h M +18 -1 mailcommon/filter/filteractioncopy.cpp M +7 -0 mailcommon/filter/filteractioncopy.h M +6 -0 mailcommon/filter/filteractiondelete.cpp M +1 -0 mailcommon/filter/filteractiondelete.h M +5 -0 mailcommon/filter/filteractionexec.cpp M +1 -0 mailcommon/filter/filteractionexec.h M +5 -0 mailcommon/filter/filteractionforward.cpp M +1 -0 mailcommon/filter/filteractionforward.h M +6 -0 mailcommon/filter/filteractionmove.cpp M +2 -0 mailcommon/filter/filteractionmove.h M +6 -0 mailcommon/filter/filteractionpipethrough.cpp M +1 -0 mailcommon/filter/filteractionpipethrough.h M +6 -0 mailcommon/filter/filteractionplaysound.cpp M +2 -0 mailcommon/filter/filteractionplaysound.h M +5 -0 mailcommon/filter/filteractionredirect.cpp M +1 -0 mailcommon/filter/filteractionredirect.h M +6 -1 mailcommon/filter/filteractionremoveheader.cpp M +1 -0 mailcommon/filter/filteractionremoveheader.h M +6 -0 mailcommon/filter/filteractionreplyto.cpp M +1 -0 mailcommon/filter/filteractionreplyto.h M +6 -0 mailcommon/filter/filteractionrewriteheader.cpp M +1 -0 mailcommon/filter/filteractionrewriteheader.h M +6 -0 mailcommon/filter/filteractionsendfakedisposition.cpp M +1 -0 mailcommon/filter/filteractionsendfakedisposition.h M +8 -2 mailcommon/filter/filteractionsendreceipt.cpp M +1 -0 mailcommon/filter/filteractionsendreceipt.h M +6 -0 mailcommon/filter/filteractionsetidentity.cpp M +1 -0 mailcommon/filter/filteractionsetidentity.h M +6 -0 mailcommon/filter/filteractionsetstatus.cpp M +1 -0 mailcommon/filter/filteractionsetstatus.h M +6 -0 mailcommon/filter/filteractionsettransport.cpp M +1 -0 mailcommon/filter/filteractionsettransport.h M +5 -0 mailcommon/filter/filteractionstatus.cpp M +2 -0 mailcommon/filter/filteractionstatus.h M +5 -2 mailcommon/filter/filteractionunsetstatus.cpp M +2 -1 mailcommon/filter/filteractionunsetstatus.h M +4 -4 mailcommon/filter/filterimporter/filterimporterevolution.cpp M +1 -1 mailcommon/filter/filterimporter/filterimporterprocmail.cpp M +1 -1 mailcommon/filter/filterimporter/filterimportersylpheed.cpp M +2 -2 mailcommon/filter/filtermanager.cpp M +2 -8 mailcommon/filter/filtermanager.h M +8 -2 mailcommon/filter/itemcontext.cpp M +8 -1 mailcommon/filter/itemcontext.h M +10 -17 mailcommon/filter/kmfilterdialog.cpp M +2 -1 mailcommon/filter/kmfilterdialog.h M +25 -8 mailcommon/filter/mailfilter.cpp M +4 -3 mailcommon/filter/mailfilter.h M +48 -20 mailcommon/searchpattern.cpp M +34 -11 mailcommon/searchpattern.h M +1 -1 mailcommon/tests/searchpatterntest.cpp M +78 -47 mailfilteragent/filtermanager.cpp M +10 -8 mailfilteragent/filtermanager.h M +8 -6 mailfilteragent/mailfilteragent.cpp M +3 -0 mailfilteragent/mailfilteragent.h http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69
Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia. Committed on 06/07/2012 at 11:12. Pushed by amantia into branch 'KDE/4.8'. Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport. An unwanted side-effect might be that more data is downloaded than before, based on the type of filters used. FIXED-IN: 4.8.5 Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 284310, bug 286364, bug 283682 M +0 -1 mailfilteragent/mailfilteragent.cpp http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec
I still see duplicate messages sometimes in 4.9.4. This is with a POP3 account, using a filter to move messages to a separate folder. Unfortunately, this only happens with a particular mailing list which has very light traffic currently, so I'm not able to investigate. I thought I'd try to duplicate the problem with a maildir account as detailed in comment 1, but I've found that automatic filtering doesn't work for me with maildir or mbox accounts (bug 255388 comment 71).
Please, make a control with "akonadi console" Choose "DB Browser", "collectiontable", "Refresh" Get a look to the column "remoteId". Are there any cell(s) with nothing? If yes, let me/us know by copying the raw(s).
I only see empty remoteId in the row with name=Search, which I guess you're not interested in. Is there a way of copying rows to the clipboard? It doesn't look like it.
Yes. Have a look to: http://techbase.kde.org/Projects/PIM/Akonadi/Development_Tools#Access_to_the_Server_Database_with_mysql
I've seen a duplicate message. Looking at Akonadi console, there are still no empty cells in the column "remoteId" (apart from name=Search).
Filter log shows duplicate message twice, so it seems the duplication happens before filtering. The timestamp on the filter log was the same for each message, so it seems the message isn't being fetched twice from the POP3 server. Using 4.10.1.
Another duplicate message after filter moved incoming POP3 message from Inbox to a different folder. Only one message in Maildir cur subdirectory, timestamp 15:01:20.549. Timestamp on filter log is 15:01:20, followed immediately by another entry for the same message, timestamp 15:01:21. Filter rule is "List-Id" <contains> xxx. Using 4.10.2.
Tested it with 4.11.3 This bug is still there. It should be easy to find the reason. Just send 1000 mails to an account and look after Mail several times.
This is still happening in 4.13. I experience the same problem I noted in bug 255388 comment 96.
This bug has only been reported for versions before 4.14, which have been unsupported for at least two years now. Can anyone tell if this bug still present? If noone confirms this bug for a Framework-based version of kmail2 (version 5.0 or later, as part of KDE Applications 15.12 or later), it gets closed in about three months.
Consistently happens to me on kmail2 5.3.0, only on filtered pop3 messages it duplicate one email downloaded per email check
Thanks, lord rel. Someone confirmed it for POP3 in another thread already. Might be more than a coincidence after all...
*** This bug has been marked as a duplicate of bug 283682 ***