Bug 293918

Summary:	kmail2 duplicates some messages
Product:	[Applications] kmail2	Reporter:	aletrev
Component:	general	Assignee:	kdepim bugs <kdepim-bugs>
Status:	RESOLVED DUPLICATE
Severity:	normal	CC:	ariel.rosenfeld.750, aspotashev, bugs, constant, dvratil, kdenis, mmtsales, null, thomas
Priority:	NOR
Version:	5.3.0
Target Milestone:	---
Platform:	Gentoo Packages
OS:	Linux
Latest Commit:		Version Fixed In:
Sentry Crash Report:

Description aletrev 2012-02-12 18:05:40 UTC

Version:           unspecified (using KDE 4.7.3) 
OS:                Linux

When I check my e-mail accounts, sometimes, kmail2 re-downloads some old messages and signs it as "new".
The number of duplicated messages may vary from 5 to 30.
This happens often if there is a new e-mail on server.
Registered accounts are: one POP3 and one IMAP. Most affected is the IMAP account.

Reproducible: Sometimes

Steps to Reproduce:
Check new e-mail on registered accounts with at least one new e-mail to download.

Actual Results:  
When kmail2 doubles messages, most of time is dedicated to deletion of duplicates.

Expected Results:  
If one e-mail (and his attachment) is already present on mailbox, kmail2 should never re-download.

Comment 1 Thomas Arend 2012-05-18 21:55:32 UTC

I can confirm a similar problem. To generate a test case for another problem I send 10 mails on my mail server with the following command to my account:

for ((i=0; i<10 ; i++))
do 

  echo "Test #`printf %4.4d $i`" | mail -s "Filter Test #`printf %4.4d $i`" thomas

done

Keep in mind that this command runs really fast and all messages have the same time stamp.

I created a filter to move the messages into a Test folder.

Every time I start "receive messages" 9 of 10 messages are received again. But they were deleted on the server with the first run. Restarting / crashing kmail2 does not help! 

Only message #0003 is received only once! But this messages was not filtered as it should. (Moved to Test folder)

I added a special filter to included a time stamp 2X-Filter-Test" for the filtering which results in the following:

Filter log entry 
[...]
23:36:04] Evaluating filter rules: (match all of the following) "Subject" <contains> "Filter Test"
[23:36:04] 1 = "Subject" <contains> "Filter Test" (Filter Test #0005)
[23:36:04] Filter rules have matched.
[23:36:04] Applying filter action: Pipe Through "formail -A "X-Filter-Test: `date -R`""
[23:36:04] Evaluating filter rules: (match any of the following) "Subject" <start-with> "Filter"
[23:36:04] 1 = "Subject" <start-with> "Filter" (Filter Test #0005)
[23:36:04] Filter rules have matched.
[23:36:04] Applying filter action: Move Into Folder "322"
[...]

A message looks after severals receives as following:
---BoM---
From thomas@arend-rhb.de  Fri May 18 23: 32:07 2012
Return-Path: <thomas@arend-rhb.de>
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on k2.arend.tksd
X-Spam-Status: No, score=-1.3 required=4.5 tests=AWL,BAYES_00,NO_RELAYS autolearn=ham version=3.3.1
X-Spam-Spammy: 0.956-5766--1937h-14398s--0d--HContent-Transfer-Encoding:7bit, 0.955-38--13h-95s--1d--0000
X-Spam-Hammy: 0.000-1987--32095h-1s--0d--H*p:D*arend-rhb.de, 0.001-2--18h-0s--1d--H*UA:mailx
X-Original-To: thomas
Delivered-To: thomas@arend-rhb.de
Received: by k2.arend.tksd (Postfix, from userid 1000) id 6798F28B2F; Fri, 18 May 2012 23:06:23 +0200 (CEST)
Date: Fri, 18 May 2012 23:06:23 +0200
To: thomas@arend-rhb.de
Subject: Filter Test #0000
User-Agent: Heirloom mailx 12.2 01/07/07
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-ID: <20120518210623.6798F28B2F@k2.arend.tksd>
From: Thomas Arend <thomas@arend-rhb.de>
X-UIDL: `ZH!!@K3"!AdZ"!od4!!
X-Filter-Test: Fri, 18 May 2012 23:35:06 +0200
X-Filter-Test: Fri, 18 May 2012 23:35:44 +0200
X-Filter-Test: Fri, 18 May 2012 23:35:51 +0200
X-Filter-Test: Fri, 18 May 2012 23:35:58 +0200
X-Filter-Test: Fri, 18 May 2012 23:36:03 +0200
X-Filter-Test: Fri, 18 May 2012 23:41:03 +0200
X-Filter-Test: Fri, 18 May 2012 23:42:51 +0200
X-Filter-Test: Fri, 18 May 2012 23:51:25 +0200

Test #0000

---EoM---

So I conclude the messages are re-received after they were modified by the filter.

The messages are received after each restart or crash of kmail2 and every time when the received cycle starts automatically.

Comment 2 Thomas Arend 2012-05-18 22:04:07 UTC

Starting and stoping akonadi stoped the re-receiving issue. Unfortunately the behavior was not reproduceable by resending 10 messages.

Comment 3 András Manţia 2012-07-04 15:36:10 UTC

Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'master'.

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 284310, bug 286364, bug 283682

M  +7    -11   kmail/kmcommands.cpp
M  +10   -7    kmail/kmcommands.h
M  +1    -1    kmail/kmmainwidget.cpp
M  +1    -5    mailcommon/filter/filteraction.cpp
M  +6    -5    mailcommon/filter/filteraction.h
M  +6    -0    mailcommon/filter/filteractionaddheader.cpp
M  +2    -0    mailcommon/filter/filteractionaddheader.h
M  +2    -2    mailcommon/filter/filteractionaddtag.cpp
M  +1    -1    mailcommon/filter/filteractionaddtag.h
M  +9    -3    mailcommon/filter/filteractionaddtoaddressbook.cpp
M  +2    -0    mailcommon/filter/filteractionaddtoaddressbook.h
M  +7    -0    mailcommon/filter/filteractionbeep.cpp
M  +1    -0    mailcommon/filter/filteractionbeep.h
M  +18   -1    mailcommon/filter/filteractioncopy.cpp
M  +7    -0    mailcommon/filter/filteractioncopy.h
M  +6    -0    mailcommon/filter/filteractiondelete.cpp
M  +1    -0    mailcommon/filter/filteractiondelete.h
M  +5    -0    mailcommon/filter/filteractionexec.cpp
M  +1    -0    mailcommon/filter/filteractionexec.h
M  +5    -0    mailcommon/filter/filteractionforward.cpp
M  +1    -0    mailcommon/filter/filteractionforward.h
M  +6    -0    mailcommon/filter/filteractionmove.cpp
M  +2    -0    mailcommon/filter/filteractionmove.h
M  +6    -0    mailcommon/filter/filteractionpipethrough.cpp
M  +1    -0    mailcommon/filter/filteractionpipethrough.h
M  +6    -0    mailcommon/filter/filteractionplaysound.cpp
M  +2    -0    mailcommon/filter/filteractionplaysound.h
M  +5    -0    mailcommon/filter/filteractionredirect.cpp
M  +1    -0    mailcommon/filter/filteractionredirect.h
M  +6    -1    mailcommon/filter/filteractionremoveheader.cpp
M  +1    -0    mailcommon/filter/filteractionremoveheader.h
M  +6    -0    mailcommon/filter/filteractionreplyto.cpp
M  +1    -0    mailcommon/filter/filteractionreplyto.h
M  +6    -0    mailcommon/filter/filteractionrewriteheader.cpp
M  +1    -0    mailcommon/filter/filteractionrewriteheader.h
M  +6    -0    mailcommon/filter/filteractionsendfakedisposition.cpp
M  +1    -0    mailcommon/filter/filteractionsendfakedisposition.h
M  +8    -2    mailcommon/filter/filteractionsendreceipt.cpp
M  +1    -0    mailcommon/filter/filteractionsendreceipt.h
M  +6    -0    mailcommon/filter/filteractionsetidentity.cpp
M  +1    -0    mailcommon/filter/filteractionsetidentity.h
M  +6    -0    mailcommon/filter/filteractionsetstatus.cpp
M  +1    -0    mailcommon/filter/filteractionsetstatus.h
M  +6    -0    mailcommon/filter/filteractionsettransport.cpp
M  +1    -0    mailcommon/filter/filteractionsettransport.h
M  +5    -0    mailcommon/filter/filteractionstatus.cpp
M  +2    -0    mailcommon/filter/filteractionstatus.h
M  +5    -2    mailcommon/filter/filteractionunsetstatus.cpp
M  +2    -1    mailcommon/filter/filteractionunsetstatus.h
M  +4    -4    mailcommon/filter/filterimporter/filterimporterevolution.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimporterprocmail.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimportersylpheed.cpp
M  +2    -2    mailcommon/filter/filtermanager.cpp
M  +2    -8    mailcommon/filter/filtermanager.h
M  +8    -2    mailcommon/filter/itemcontext.cpp
M  +8    -1    mailcommon/filter/itemcontext.h
M  +10   -17   mailcommon/filter/kmfilterdialog.cpp
M  +2    -1    mailcommon/filter/kmfilterdialog.h
M  +25   -8    mailcommon/filter/mailfilter.cpp
M  +4    -3    mailcommon/filter/mailfilter.h
M  +48   -20   mailcommon/searchpattern.cpp
M  +34   -11   mailcommon/searchpattern.h
M  +1    -1    mailcommon/tests/searchpatterntest.cpp
M  +78   -47   mailfilteragent/filtermanager.cpp
M  +10   -8    mailfilteragent/filtermanager.h
M  +8    -6    mailfilteragent/mailfilteragent.cpp
M  +3    -0    mailfilteragent/mailfilteragent.h

http://commits.kde.org/kdepim/d64d61b470c56b92294f6adee6d74305a217628d

Comment 4 András Manţia 2012-07-05 11:40:44 UTC

Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'KDE/4.9'.

(Backport filter fix from master)

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 284310, bug 286364, bug 283682
(cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d)

FIXED-IN: 4.9

M  +7    -11   kmail/kmcommands.cpp
M  +10   -7    kmail/kmcommands.h
M  +1    -1    kmail/kmmainwidget.cpp
M  +1    -5    mailcommon/filter/filteraction.cpp
M  +6    -5    mailcommon/filter/filteraction.h
M  +6    -0    mailcommon/filter/filteractionaddheader.cpp
M  +2    -0    mailcommon/filter/filteractionaddheader.h
M  +2    -2    mailcommon/filter/filteractionaddtag.cpp
M  +1    -1    mailcommon/filter/filteractionaddtag.h
M  +9    -3    mailcommon/filter/filteractionaddtoaddressbook.cpp
M  +2    -0    mailcommon/filter/filteractionaddtoaddressbook.h
M  +7    -0    mailcommon/filter/filteractionbeep.cpp
M  +1    -0    mailcommon/filter/filteractionbeep.h
M  +18   -1    mailcommon/filter/filteractioncopy.cpp
M  +7    -0    mailcommon/filter/filteractioncopy.h
M  +6    -0    mailcommon/filter/filteractiondelete.cpp
M  +1    -0    mailcommon/filter/filteractiondelete.h
M  +5    -0    mailcommon/filter/filteractionexec.cpp
M  +1    -0    mailcommon/filter/filteractionexec.h
M  +5    -0    mailcommon/filter/filteractionforward.cpp
M  +1    -0    mailcommon/filter/filteractionforward.h
M  +6    -0    mailcommon/filter/filteractionmove.cpp
M  +2    -0    mailcommon/filter/filteractionmove.h
M  +6    -0    mailcommon/filter/filteractionpipethrough.cpp
M  +1    -0    mailcommon/filter/filteractionpipethrough.h
M  +6    -0    mailcommon/filter/filteractionplaysound.cpp
M  +2    -0    mailcommon/filter/filteractionplaysound.h
M  +5    -0    mailcommon/filter/filteractionredirect.cpp
M  +1    -0    mailcommon/filter/filteractionredirect.h
M  +6    -1    mailcommon/filter/filteractionremoveheader.cpp
M  +1    -0    mailcommon/filter/filteractionremoveheader.h
M  +6    -0    mailcommon/filter/filteractionreplyto.cpp
M  +1    -0    mailcommon/filter/filteractionreplyto.h
M  +6    -0    mailcommon/filter/filteractionrewriteheader.cpp
M  +1    -0    mailcommon/filter/filteractionrewriteheader.h
M  +6    -0    mailcommon/filter/filteractionsendfakedisposition.cpp
M  +1    -0    mailcommon/filter/filteractionsendfakedisposition.h
M  +8    -2    mailcommon/filter/filteractionsendreceipt.cpp
M  +1    -0    mailcommon/filter/filteractionsendreceipt.h
M  +6    -0    mailcommon/filter/filteractionsetidentity.cpp
M  +1    -0    mailcommon/filter/filteractionsetidentity.h
M  +6    -0    mailcommon/filter/filteractionsetstatus.cpp
M  +1    -0    mailcommon/filter/filteractionsetstatus.h
M  +6    -0    mailcommon/filter/filteractionsettransport.cpp
M  +1    -0    mailcommon/filter/filteractionsettransport.h
M  +5    -0    mailcommon/filter/filteractionstatus.cpp
M  +2    -0    mailcommon/filter/filteractionstatus.h
M  +5    -2    mailcommon/filter/filteractionunsetstatus.cpp
M  +2    -1    mailcommon/filter/filteractionunsetstatus.h
M  +4    -4    mailcommon/filter/filterimporter/filterimporterevolution.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimporterprocmail.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimportersylpheed.cpp
M  +2    -2    mailcommon/filter/filtermanager.cpp
M  +2    -8    mailcommon/filter/filtermanager.h
M  +8    -2    mailcommon/filter/itemcontext.cpp
M  +8    -1    mailcommon/filter/itemcontext.h
M  +10   -17   mailcommon/filter/kmfilterdialog.cpp
M  +2    -1    mailcommon/filter/kmfilterdialog.h
M  +25   -8    mailcommon/filter/mailfilter.cpp
M  +4    -3    mailcommon/filter/mailfilter.h
M  +48   -20   mailcommon/searchpattern.cpp
M  +34   -11   mailcommon/searchpattern.h
M  +1    -1    mailcommon/tests/searchpatterntest.cpp
M  +78   -47   mailfilteragent/filtermanager.cpp
M  +10   -8    mailfilteragent/filtermanager.h
M  +8    -6    mailfilteragent/mailfilteragent.cpp
M  +3    -0    mailfilteragent/mailfilteragent.h

http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69

Comment 5 András Manţia 2012-07-06 09:13:44 UTC

Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia.
Committed on 06/07/2012 at 11:12.
Pushed by amantia into branch 'KDE/4.8'.

Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport.
An unwanted side-effect might be that more data is downloaded than before,
based on the type of filters used.

FIXED-IN: 4.8.5
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 284310, bug 286364, bug 283682

M  +0    -1    mailfilteragent/mailfilteragent.cpp

http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec

Comment 6 Graeme Hewson 2012-12-21 16:50:57 UTC

I still see duplicate messages sometimes in 4.9.4. This is with a POP3 account, using a filter to move messages to a separate folder.

Unfortunately, this only happens with a particular mailing list which has very light traffic currently, so I'm not able to investigate. I thought I'd try to duplicate the problem with a maildir account as detailed in comment 1, but I've found that automatic filtering doesn't work for me with maildir or mbox accounts (bug 255388 comment 71).

Comment 7 Unknown 2013-01-23 10:10:06 UTC

Please, make a control with "akonadi console"
Choose "DB Browser", "collectiontable", "Refresh"
Get a look to the column "remoteId".
Are there any cell(s) with nothing? If yes, let me/us know by copying the raw(s).

Comment 8 Graeme Hewson 2013-01-23 14:13:25 UTC

I only see empty remoteId in the row with name=Search, which I guess you're not interested in.

Is there a way of copying rows to the clipboard? It doesn't look like it.

Comment 9 Unknown 2013-01-23 14:50:07 UTC

Yes.
Have a look to:
  http://techbase.kde.org/Projects/PIM/Akonadi/Development_Tools#Access_to_the_Server_Database_with_mysql

Comment 10 Graeme Hewson 2013-02-04 17:32:27 UTC

I've seen a duplicate message. Looking at Akonadi console, there are still no empty cells in the column "remoteId" (apart from name=Search).

Comment 11 Graeme Hewson 2013-04-02 06:12:50 UTC

Filter log shows duplicate message twice, so it seems the duplication happens before filtering. The timestamp on the filter log was the same for each message, so it seems the message isn't being fetched twice from the POP3 server. Using 4.10.1.

Comment 12 Graeme Hewson 2013-04-29 14:28:23 UTC

Another duplicate message after filter moved incoming POP3 message from Inbox to a different folder. Only one message in Maildir cur subdirectory, timestamp 15:01:20.549. Timestamp on filter log is 15:01:20, followed immediately by another entry for the same message, timestamp 15:01:21. Filter rule is "List-Id" <contains> xxx.

Using 4.10.2.

Comment 13 Thomas Arend 2014-01-11 23:14:55 UTC

Tested it with 4.11.3

This bug is still there. It should be easy to find the reason.

Just send 1000 mails to an account and look after Mail several times.

Comment 14 Graeme Hewson 2014-04-21 09:42:44 UTC

This is still happening in 4.13. I experience the same problem I noted in bug 255388 comment 96.

Comment 15 Denis Kurz 2016-09-24 18:01:41 UTC

This bug has only been reported for versions before 4.14, which have been unsupported for at least two years now. Can anyone tell if this bug still present?

If noone confirms this bug for a Framework-based version of kmail2 (version 5.0 or later, as part of KDE Applications 15.12 or later), it gets closed in about three months.

Comment 16 Ariel Rosenfeld 2016-09-26 19:56:30 UTC

Consistently happens to me on kmail2 5.3.0, only on filtered pop3 messages it duplicate one email downloaded per email check

Comment 17 Denis Kurz 2016-09-28 16:24:33 UTC

Thanks, lord rel. Someone confirmed it for POP3 in another thread already. Might be more than a coincidence after all...

Comment 18 Daniel Vrátil 2017-03-22 00:04:44 UTC


*** This bug has been marked as a duplicate of bug 283682 ***