Bug 275233 - Akonadi creates a lot of empty duplicates/conflicting version
Summary: Akonadi creates a lot of empty duplicates/conflicting version
Status: RESOLVED FIXED
Alias: None
Product: Akonadi
Classification: Frameworks and Libraries
Component: general (show other bugs)
Version: 1.5.3
Platform: openSUSE Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-06-08 21:07 UTC by S. Burmeister
Modified: 2012-10-17 13:42 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
debug output from akonadi console (12.16 KB, text/plain)
2011-07-12 14:04 UTC, S. Burmeister
Details

Note You need to log in before you can comment on or make changes to this bug.
Description S. Burmeister 2011-06-08 21:07:35 UTC
Version:           1.5.3 (using KDE 4.6.3) 
OS:                Linux

Very often I get a notification that there are conflicting versions. Most of the time the left version shows no email text at all, i.e. is empty and just its status is different, e.g. $SEEN or $IGNORED.

This only happens on local/kmail folders. So it might be related to filtering/moving messages from IMAP to local folders.

Reproducible: Sometimes
Comment 1 Malte 2011-06-27 21:37:01 UTC
In my case I get this note when I delete mails from folders with POP3 mails. When I delete messages from my IMAP account it does not occur.
Comment 2 Malte 2011-06-27 21:37:51 UTC
I wanted to add: I can see this note with KMail 2.1.0 on KDE 4.6.4
Comment 3 S. Burmeister 2011-07-12 13:11:35 UTC
I might have some hints towards reproducing this.

- Create filters that move emails from your imap inbox to local folders
- Mark some threads that still receive emails as "ignore" on that box
- change the local folder, e.g. open the trash
- use another computer to access the same imap account (no filters) with kmail and read some of the messages that are going to be marked as ignored on the other box.
(- while the messages are already moved (status unseen) to a local folder on the first box the second one can still access them until the account is checked again, maybe because akonadi cached them) [this is optional]


Second scenario:
- set-up filters to move email to a local folder on one box
- retrieve emails with that box. kmail will only download their headers and show them in the list
- next kmail will try to download the content of the emails to filter them, i.e. the content is not yet available to kmail and clicking on one of them would start to download the content
- make your internet connection fail, i.e. no traffic but no disconnection either
- now restart akonadi or kmail, suspending might already do the trick, maybe even going offline and online.
- next time kmail will try to retrieve the emails it might succeed and retrieve the full email - however there will be a conflict because apparently the email (only headers) does already exist and thus the version conflict window shows the first attempt email as empty, i.e. no data and the second with full content and a date/time-stamp of some minutes later than the first one.

The second example would only make sense if my assumption is correct that the version conflict GUI does not show headers but only the "full payload" including headers.

I have logged what happens if such a conflict comes up.
Comment 4 S. Burmeister 2011-07-12 14:04:25 UTC
Created attachment 61807 [details]
debug output from akonadi console
Comment 5 S. Burmeister 2011-07-17 17:09:29 UTC
Happen as well after kontact crashed.

Honestly, what sense does it make to ask the user whether he wants to keep an empty record? – other than giving him the chance to lose data.
Comment 6 S. Burmeister 2011-07-17 17:10:10 UTC
oh, and it's valid for 4.7.
Comment 7 S. Burmeister 2011-08-03 04:52:50 UTC
Yet another way to reproduce this – without empty versions though.

Go to you imap inbox and select multiple emails (including unread!) while pressing CTRL.

Press DEL to move them to the local trash.

This will cause conflicts between two versions, i.e. one SEEN and one without that flag.
Comment 8 András Manţia 2011-10-20 07:41:13 UTC
Can you reproduce with KDEPIM 4.7.2 or even better, with master?
Comment 9 András Manţia 2012-07-04 15:36:10 UTC
Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'master'.

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 293918, bug 284310, bug 286364, bug 283682

M  +7    -11   kmail/kmcommands.cpp
M  +10   -7    kmail/kmcommands.h
M  +1    -1    kmail/kmmainwidget.cpp
M  +1    -5    mailcommon/filter/filteraction.cpp
M  +6    -5    mailcommon/filter/filteraction.h
M  +6    -0    mailcommon/filter/filteractionaddheader.cpp
M  +2    -0    mailcommon/filter/filteractionaddheader.h
M  +2    -2    mailcommon/filter/filteractionaddtag.cpp
M  +1    -1    mailcommon/filter/filteractionaddtag.h
M  +9    -3    mailcommon/filter/filteractionaddtoaddressbook.cpp
M  +2    -0    mailcommon/filter/filteractionaddtoaddressbook.h
M  +7    -0    mailcommon/filter/filteractionbeep.cpp
M  +1    -0    mailcommon/filter/filteractionbeep.h
M  +18   -1    mailcommon/filter/filteractioncopy.cpp
M  +7    -0    mailcommon/filter/filteractioncopy.h
M  +6    -0    mailcommon/filter/filteractiondelete.cpp
M  +1    -0    mailcommon/filter/filteractiondelete.h
M  +5    -0    mailcommon/filter/filteractionexec.cpp
M  +1    -0    mailcommon/filter/filteractionexec.h
M  +5    -0    mailcommon/filter/filteractionforward.cpp
M  +1    -0    mailcommon/filter/filteractionforward.h
M  +6    -0    mailcommon/filter/filteractionmove.cpp
M  +2    -0    mailcommon/filter/filteractionmove.h
M  +6    -0    mailcommon/filter/filteractionpipethrough.cpp
M  +1    -0    mailcommon/filter/filteractionpipethrough.h
M  +6    -0    mailcommon/filter/filteractionplaysound.cpp
M  +2    -0    mailcommon/filter/filteractionplaysound.h
M  +5    -0    mailcommon/filter/filteractionredirect.cpp
M  +1    -0    mailcommon/filter/filteractionredirect.h
M  +6    -1    mailcommon/filter/filteractionremoveheader.cpp
M  +1    -0    mailcommon/filter/filteractionremoveheader.h
M  +6    -0    mailcommon/filter/filteractionreplyto.cpp
M  +1    -0    mailcommon/filter/filteractionreplyto.h
M  +6    -0    mailcommon/filter/filteractionrewriteheader.cpp
M  +1    -0    mailcommon/filter/filteractionrewriteheader.h
M  +6    -0    mailcommon/filter/filteractionsendfakedisposition.cpp
M  +1    -0    mailcommon/filter/filteractionsendfakedisposition.h
M  +8    -2    mailcommon/filter/filteractionsendreceipt.cpp
M  +1    -0    mailcommon/filter/filteractionsendreceipt.h
M  +6    -0    mailcommon/filter/filteractionsetidentity.cpp
M  +1    -0    mailcommon/filter/filteractionsetidentity.h
M  +6    -0    mailcommon/filter/filteractionsetstatus.cpp
M  +1    -0    mailcommon/filter/filteractionsetstatus.h
M  +6    -0    mailcommon/filter/filteractionsettransport.cpp
M  +1    -0    mailcommon/filter/filteractionsettransport.h
M  +5    -0    mailcommon/filter/filteractionstatus.cpp
M  +2    -0    mailcommon/filter/filteractionstatus.h
M  +5    -2    mailcommon/filter/filteractionunsetstatus.cpp
M  +2    -1    mailcommon/filter/filteractionunsetstatus.h
M  +4    -4    mailcommon/filter/filterimporter/filterimporterevolution.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimporterprocmail.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimportersylpheed.cpp
M  +2    -2    mailcommon/filter/filtermanager.cpp
M  +2    -8    mailcommon/filter/filtermanager.h
M  +8    -2    mailcommon/filter/itemcontext.cpp
M  +8    -1    mailcommon/filter/itemcontext.h
M  +10   -17   mailcommon/filter/kmfilterdialog.cpp
M  +2    -1    mailcommon/filter/kmfilterdialog.h
M  +25   -8    mailcommon/filter/mailfilter.cpp
M  +4    -3    mailcommon/filter/mailfilter.h
M  +48   -20   mailcommon/searchpattern.cpp
M  +34   -11   mailcommon/searchpattern.h
M  +1    -1    mailcommon/tests/searchpatterntest.cpp
M  +78   -47   mailfilteragent/filtermanager.cpp
M  +10   -8    mailfilteragent/filtermanager.h
M  +8    -6    mailfilteragent/mailfilteragent.cpp
M  +3    -0    mailfilteragent/mailfilteragent.h

http://commits.kde.org/kdepim/d64d61b470c56b92294f6adee6d74305a217628d
Comment 10 András Manţia 2012-07-05 11:40:44 UTC
Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'KDE/4.9'.

(Backport filter fix from master)

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 293918, bug 284310, bug 286364, bug 283682
(cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d)

FIXED-IN: 4.9

M  +7    -11   kmail/kmcommands.cpp
M  +10   -7    kmail/kmcommands.h
M  +1    -1    kmail/kmmainwidget.cpp
M  +1    -5    mailcommon/filter/filteraction.cpp
M  +6    -5    mailcommon/filter/filteraction.h
M  +6    -0    mailcommon/filter/filteractionaddheader.cpp
M  +2    -0    mailcommon/filter/filteractionaddheader.h
M  +2    -2    mailcommon/filter/filteractionaddtag.cpp
M  +1    -1    mailcommon/filter/filteractionaddtag.h
M  +9    -3    mailcommon/filter/filteractionaddtoaddressbook.cpp
M  +2    -0    mailcommon/filter/filteractionaddtoaddressbook.h
M  +7    -0    mailcommon/filter/filteractionbeep.cpp
M  +1    -0    mailcommon/filter/filteractionbeep.h
M  +18   -1    mailcommon/filter/filteractioncopy.cpp
M  +7    -0    mailcommon/filter/filteractioncopy.h
M  +6    -0    mailcommon/filter/filteractiondelete.cpp
M  +1    -0    mailcommon/filter/filteractiondelete.h
M  +5    -0    mailcommon/filter/filteractionexec.cpp
M  +1    -0    mailcommon/filter/filteractionexec.h
M  +5    -0    mailcommon/filter/filteractionforward.cpp
M  +1    -0    mailcommon/filter/filteractionforward.h
M  +6    -0    mailcommon/filter/filteractionmove.cpp
M  +2    -0    mailcommon/filter/filteractionmove.h
M  +6    -0    mailcommon/filter/filteractionpipethrough.cpp
M  +1    -0    mailcommon/filter/filteractionpipethrough.h
M  +6    -0    mailcommon/filter/filteractionplaysound.cpp
M  +2    -0    mailcommon/filter/filteractionplaysound.h
M  +5    -0    mailcommon/filter/filteractionredirect.cpp
M  +1    -0    mailcommon/filter/filteractionredirect.h
M  +6    -1    mailcommon/filter/filteractionremoveheader.cpp
M  +1    -0    mailcommon/filter/filteractionremoveheader.h
M  +6    -0    mailcommon/filter/filteractionreplyto.cpp
M  +1    -0    mailcommon/filter/filteractionreplyto.h
M  +6    -0    mailcommon/filter/filteractionrewriteheader.cpp
M  +1    -0    mailcommon/filter/filteractionrewriteheader.h
M  +6    -0    mailcommon/filter/filteractionsendfakedisposition.cpp
M  +1    -0    mailcommon/filter/filteractionsendfakedisposition.h
M  +8    -2    mailcommon/filter/filteractionsendreceipt.cpp
M  +1    -0    mailcommon/filter/filteractionsendreceipt.h
M  +6    -0    mailcommon/filter/filteractionsetidentity.cpp
M  +1    -0    mailcommon/filter/filteractionsetidentity.h
M  +6    -0    mailcommon/filter/filteractionsetstatus.cpp
M  +1    -0    mailcommon/filter/filteractionsetstatus.h
M  +6    -0    mailcommon/filter/filteractionsettransport.cpp
M  +1    -0    mailcommon/filter/filteractionsettransport.h
M  +5    -0    mailcommon/filter/filteractionstatus.cpp
M  +2    -0    mailcommon/filter/filteractionstatus.h
M  +5    -2    mailcommon/filter/filteractionunsetstatus.cpp
M  +2    -1    mailcommon/filter/filteractionunsetstatus.h
M  +4    -4    mailcommon/filter/filterimporter/filterimporterevolution.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimporterprocmail.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimportersylpheed.cpp
M  +2    -2    mailcommon/filter/filtermanager.cpp
M  +2    -8    mailcommon/filter/filtermanager.h
M  +8    -2    mailcommon/filter/itemcontext.cpp
M  +8    -1    mailcommon/filter/itemcontext.h
M  +10   -17   mailcommon/filter/kmfilterdialog.cpp
M  +2    -1    mailcommon/filter/kmfilterdialog.h
M  +25   -8    mailcommon/filter/mailfilter.cpp
M  +4    -3    mailcommon/filter/mailfilter.h
M  +48   -20   mailcommon/searchpattern.cpp
M  +34   -11   mailcommon/searchpattern.h
M  +1    -1    mailcommon/tests/searchpatterntest.cpp
M  +78   -47   mailfilteragent/filtermanager.cpp
M  +10   -8    mailfilteragent/filtermanager.h
M  +8    -6    mailfilteragent/mailfilteragent.cpp
M  +3    -0    mailfilteragent/mailfilteragent.h

http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69
Comment 11 András Manţia 2012-07-06 09:13:44 UTC
Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia.
Committed on 06/07/2012 at 11:12.
Pushed by amantia into branch 'KDE/4.8'.

Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport.
An unwanted side-effect might be that more data is downloaded than before,
based on the type of filters used.

FIXED-IN: 4.8.5
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 293918, bug 284310, bug 286364, bug 283682

M  +0    -1    mailfilteragent/mailfilteragent.cpp

http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec
Comment 12 András Manţia 2012-10-17 13:42:34 UTC
Fixed in 4.9.3/4.10.