Bug 286043

Summary: possible data loss due to filtering (headers or body of email get lost)
Product: [Frameworks and Libraries] Akonadi Reporter: Milian Wolff <mail>
Component: Mail Filter AgentAssignee: kdepim bugs <kdepim-bugs>
Status: RESOLVED FIXED    
Severity: critical CC: amantia, auxsvr, ht990332, joerg.schaible, jospoortvliet, kde.org, krammer, martin.ruessler, oldium.pro, sven.burmeister
Priority: VHI    
Version: 4.8   
Target Milestone: ---   
Platform: Unlisted Binaries   
OS: Linux   
Latest Commit: Version Fixed In: 4.9
Sentry Crash Report:
Attachments: screenshot showing mails with lost headers

Description Milian Wolff 2011-11-07 21:15:30 UTC
Version:           Git (master)
OS:                Linux

I have a pop3 message with lots of filters which either mark new messages as spam or categorize them into a different dimap account.

Right now I see it regularly (every day a few times) that kmail's filter agent displays an error "No Items Found". This might be related to the creation of dead messages in my dimap account. With dead I mean they have *no* body, headers, anything - they just exist as an row in the pimitemtable without any associations in the parttable.

kmail2 does not display these items, I can see them in the database or in kmail1 though.

Also, maybe related, I see it from time to time that new mails are left untouched in the inbox of my pop3 account. If i select these mails and manually "apply all filters" they get sorted properly. I'm not sure though, if this is related.

Reproducible: Didn't try

Steps to Reproduce:
I'm not sure how to reproduce this. I'd try by creating a pop3 account with a filter to move items to a dimap account. maybe it's the spamassassin filters... no clue :(

I can reproduce it all the time though, so if I should try something out, please tell me.


Expected Results:  
filtering should work without showing errors all the time nor should it create dead items or leave items untouched.
Comment 1 Milian Wolff 2011-11-07 22:21:05 UTC
ignore Reproducible: Didn't try,
for me it's reproducible (i.e. it happens frequently) but I have yet to find out how to trigger it by hand (i.e. not randomly)
Comment 2 Milian Wolff 2011-11-08 11:05:30 UTC
Created attachment 65380 [details]
screenshot showing mails with lost headers

I think I just encountered data loss due to this issue, see attached screenshot. I now have about 20 emails without any headers, thankfully nothing important but this looks like a serious issue to me.

I have another idea of what could be the possible issue here: I assumed the mails are not getting filtered (see first comment., my mails sometimes just stay in the pop3 inbox), furthermore due to bug 286095 I have no idea whether a mail actually gets filtered or not. Now I selected some messages and manually triggered the "apply all filters" action. Nothing happened for some time (a few seconds) and then suddenly the mails lost their headers (see screenshot).
Comment 3 Milian Wolff 2011-11-08 11:06:36 UTC
increasing priority as I encountered data loss
Comment 4 Milian Wolff 2011-11-08 11:09:47 UTC
regarding the "no items found" error message: the filter log viewer showed this in such a case:

------------------------------
------------------------------
------------------------------
------------------------------


so probably it does not show any errors there :-/
Comment 5 Milian Wolff 2011-11-10 15:55:41 UTC
starting akonadi from the cli shows some interesting debug output before I see a "no items found" error message:

QFSFileEngine::open: No file name specified
posting retrieval request for item 57572  there are  1  queues and  0  items in mine 
request for item 57572 still pending - waiting 
processing retrieval request for item 57572  parts: ("RFC822")  of resource: "akonadi_imap_resource_2" 
akonadi_imap_resource_2(23147)/libakonadi Akonadi::ResourceBase::itemRetrieved: Item does not provide part "RFC822" 
continuing 
request for item 57572 succeeded 
akonadi_mailfilter_agent(23150): Error while moving items.  103 "Unknown error. (No items found)"  to destinationCollection.id() : 95 
akonadi_nepomuk_feeder(23153) ItemQueue::jobResult: "http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefLabel has a max cardinality of 1. Provided 5 values - "kde-i18n-doc-bounces@kde.org", "kde-i18n-doc-bounces@kde.org", "kde-i18n-doc-bounces@kde.org", "kde-i18n-doc-bounces@kde.org", "kde-i18n-doc-bounces@kde.org"" 
QFSFileEngine::open: No file name specified
QFSFileEngine::open: No file name specified
posting retrieval request for item 57575  there are  1  queues and  0  items in mine 
request for item 57575 still pending - waiting 
processing retrieval request for item 57575  parts: ("RFC822")  of resource: "akonadi_imap_resource_2" 
akonadi_imap_resource_2(23147)/libakonadi Akonadi::ResourceBase::itemRetrieved: Item does not provide part "RFC822" 
continuing 
request for item 57575 succeeded 
akonadi_mailfilter_agent(23150): Error while moving items.  103 "Unknown error. (No items found)"  to destinationCollection.id() : 95
Comment 6 Christophe Marin 2011-11-10 17:17:54 UTC
raising severity.
Comment 7 Oldřich Jedlička 2012-01-15 21:45:55 UTC
Same problem here, I'm trying to figure out how to use KMail2 with my mail. This issue is a show stopper for me as I cannot get to any of my incoming mail.

akonadi-server-1.6.2, kdepim-runtime 4.7.97, using mixedmail resource over the old KMail dir.
Comment 8 Oldřich Jedlička 2012-01-15 21:52:40 UTC
I mean that I have the mentioned "no items found" problem currently.
Comment 9 Oldřich Jedlička 2012-01-15 22:32:44 UTC
There is a code that "smells" to me - I don't know what is Session or KJob, so this might not be prefect analysis:

1. kdepim-runtime/resources/shared/filestore/itemmovejob.cpp in constructor of ItemMoveJob calls session->addJob(this).

2. kdepim-runtime/resources/mixedmaildir/mixedmaildirresource.cpp, method itemMoved:

2a) it calls mStore->moveItem, which creates ItemMoveJob, which adds the job instance to session in the constructor.

2b) it connects to the job's slots.

Is there some confidence that the job doesn't finish before the step 2b is executed? I _guess_ this is multi-threaded application, and from the information that the job isn't explicitly started I _think_ it is done by the session (asynchronously). So it might finish before the signal/slot is connected, so the slot isn't called.

Is there any KMail/Akonadi developer to verify this finding? Thanks.
Comment 10 S. Burmeister 2012-02-10 16:51:01 UTC
bug 293768 could be related.
Comment 11 Kevin Krammer 2012-03-08 14:29:28 UTC
(In reply to comment #9)

> Is there some confidence that the job doesn't finish before the step 2b is
> executed? I _guess_ this is multi-threaded application, and from the
> information that the job isn't explicitly started I _think_ it is done by
> the session (asynchronously). So it might finish before the signal/slot is
> connected, so the slot isn't called.

This is a single threaded application.
The job is started through delayed invocation so all connects have been executed before that happens
Comment 12 jos poortvliet 2012-03-08 15:30:16 UTC
FWIW I also have a LARGE bunch of completely empty mails in my inbox. Yet if you click them you see the content... No idea if it is related but it's quite annoying ;-)
http://susepaste.org/62206930
Comment 13 jos poortvliet 2012-03-08 15:32:41 UTC
BTW it could be related to https://bugs.kde.org/show_bug.cgi?id=295484
Comment 14 Jörg Schaible 2012-03-08 17:16:13 UTC
I have the same problem with KMail 4.7.4 and an IMAP account. No filter configured.
The behavior is different to Bug 295484, because the emails are not touched on the server and can be shown by other email clients. It happens almost only to multipart mails, but not to all. I have mails that look structurally almost equal, but one is shown properly and the other one not at all.
Comment 15 András Manţia 2012-03-17 21:36:28 UTC
Milian, did you see this bug recently? It is very unclear for me what goes on, and some of the comments seems to be unrelated (well, they might looks similar, but they only confirm the result, not the possible cause).
All reporters seeing it: try to reproduce the bug, as right now I can't see anything like this. Also me clear about what filter runs for the messages that get lost (use the filter log from KMail), what kind of mailbox types you use (for inbox and for the destination, e.g imap/maildir/mixedmaildir/mbox - in case of mixedmaildir if it is a maildir or mbox) and if the mail is *really* gone from your hard driver (or from the IMAP server in case of IMAP). 
The best would be to have a reproducible way for the bug.
Comment 16 Milian Wolff 2012-03-19 12:21:50 UTC
I've disabled most of my filters because of this bug and hence didn't see it for some time. I'll try to re-enable the filters and see whether I can still reproduce it.

Note thought that this was not a 100% reproducible issue, only some mails where affected. Anyhow, I'll let you know once I know more.
Comment 17 Jörg Schaible 2012-04-10 20:00:07 UTC
I've updated meanwhile from KMail 4.7.4 to KMail 4.8.1, but I have still those emails whose body is not shown. However, this happens only for some (multi-part) mails that I have received in the first 3 months of this year. It does not happen to new mails. Additionally if I copy such a mail from the .maildir on my server to the server's inbox directory, KMail can suddenly display the body. So it seems that somehow the local cache got corrupt. What can be done to invalidate the cache?
Comment 18 Jörg Schaible 2012-04-13 17:56:19 UTC
One last comment: If I take a copy (with D&D) in KMail itself, the copied email has definitely no body anymore.
Comment 19 András Manţia 2012-04-13 18:09:30 UTC
Is the copy within the same account? What kind of accounts are involved (source/destination)?
Comment 20 Jörg Schaible 2012-04-13 20:52:56 UTC
The same IMAP account (on an intranet server running Courier), server-side subscription, offline-mode activated. I have also direct access on the mail server's file system, Courier saves in maildir format.
Comment 21 András Manţia 2012-07-04 15:36:05 UTC
Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'master'.

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682

M  +7    -11   kmail/kmcommands.cpp
M  +10   -7    kmail/kmcommands.h
M  +1    -1    kmail/kmmainwidget.cpp
M  +1    -5    mailcommon/filter/filteraction.cpp
M  +6    -5    mailcommon/filter/filteraction.h
M  +6    -0    mailcommon/filter/filteractionaddheader.cpp
M  +2    -0    mailcommon/filter/filteractionaddheader.h
M  +2    -2    mailcommon/filter/filteractionaddtag.cpp
M  +1    -1    mailcommon/filter/filteractionaddtag.h
M  +9    -3    mailcommon/filter/filteractionaddtoaddressbook.cpp
M  +2    -0    mailcommon/filter/filteractionaddtoaddressbook.h
M  +7    -0    mailcommon/filter/filteractionbeep.cpp
M  +1    -0    mailcommon/filter/filteractionbeep.h
M  +18   -1    mailcommon/filter/filteractioncopy.cpp
M  +7    -0    mailcommon/filter/filteractioncopy.h
M  +6    -0    mailcommon/filter/filteractiondelete.cpp
M  +1    -0    mailcommon/filter/filteractiondelete.h
M  +5    -0    mailcommon/filter/filteractionexec.cpp
M  +1    -0    mailcommon/filter/filteractionexec.h
M  +5    -0    mailcommon/filter/filteractionforward.cpp
M  +1    -0    mailcommon/filter/filteractionforward.h
M  +6    -0    mailcommon/filter/filteractionmove.cpp
M  +2    -0    mailcommon/filter/filteractionmove.h
M  +6    -0    mailcommon/filter/filteractionpipethrough.cpp
M  +1    -0    mailcommon/filter/filteractionpipethrough.h
M  +6    -0    mailcommon/filter/filteractionplaysound.cpp
M  +2    -0    mailcommon/filter/filteractionplaysound.h
M  +5    -0    mailcommon/filter/filteractionredirect.cpp
M  +1    -0    mailcommon/filter/filteractionredirect.h
M  +6    -1    mailcommon/filter/filteractionremoveheader.cpp
M  +1    -0    mailcommon/filter/filteractionremoveheader.h
M  +6    -0    mailcommon/filter/filteractionreplyto.cpp
M  +1    -0    mailcommon/filter/filteractionreplyto.h
M  +6    -0    mailcommon/filter/filteractionrewriteheader.cpp
M  +1    -0    mailcommon/filter/filteractionrewriteheader.h
M  +6    -0    mailcommon/filter/filteractionsendfakedisposition.cpp
M  +1    -0    mailcommon/filter/filteractionsendfakedisposition.h
M  +8    -2    mailcommon/filter/filteractionsendreceipt.cpp
M  +1    -0    mailcommon/filter/filteractionsendreceipt.h
M  +6    -0    mailcommon/filter/filteractionsetidentity.cpp
M  +1    -0    mailcommon/filter/filteractionsetidentity.h
M  +6    -0    mailcommon/filter/filteractionsetstatus.cpp
M  +1    -0    mailcommon/filter/filteractionsetstatus.h
M  +6    -0    mailcommon/filter/filteractionsettransport.cpp
M  +1    -0    mailcommon/filter/filteractionsettransport.h
M  +5    -0    mailcommon/filter/filteractionstatus.cpp
M  +2    -0    mailcommon/filter/filteractionstatus.h
M  +5    -2    mailcommon/filter/filteractionunsetstatus.cpp
M  +2    -1    mailcommon/filter/filteractionunsetstatus.h
M  +4    -4    mailcommon/filter/filterimporter/filterimporterevolution.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimporterprocmail.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimportersylpheed.cpp
M  +2    -2    mailcommon/filter/filtermanager.cpp
M  +2    -8    mailcommon/filter/filtermanager.h
M  +8    -2    mailcommon/filter/itemcontext.cpp
M  +8    -1    mailcommon/filter/itemcontext.h
M  +10   -17   mailcommon/filter/kmfilterdialog.cpp
M  +2    -1    mailcommon/filter/kmfilterdialog.h
M  +25   -8    mailcommon/filter/mailfilter.cpp
M  +4    -3    mailcommon/filter/mailfilter.h
M  +48   -20   mailcommon/searchpattern.cpp
M  +34   -11   mailcommon/searchpattern.h
M  +1    -1    mailcommon/tests/searchpatterntest.cpp
M  +78   -47   mailfilteragent/filtermanager.cpp
M  +10   -8    mailfilteragent/filtermanager.h
M  +8    -6    mailfilteragent/mailfilteragent.cpp
M  +3    -0    mailfilteragent/mailfilteragent.h

http://commits.kde.org/kdepim/d64d61b470c56b92294f6adee6d74305a217628d
Comment 22 András Manţia 2012-07-05 11:40:39 UTC
Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'KDE/4.9'.

(Backport filter fix from master)

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682
(cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d)

FIXED-IN: 4.9

M  +7    -11   kmail/kmcommands.cpp
M  +10   -7    kmail/kmcommands.h
M  +1    -1    kmail/kmmainwidget.cpp
M  +1    -5    mailcommon/filter/filteraction.cpp
M  +6    -5    mailcommon/filter/filteraction.h
M  +6    -0    mailcommon/filter/filteractionaddheader.cpp
M  +2    -0    mailcommon/filter/filteractionaddheader.h
M  +2    -2    mailcommon/filter/filteractionaddtag.cpp
M  +1    -1    mailcommon/filter/filteractionaddtag.h
M  +9    -3    mailcommon/filter/filteractionaddtoaddressbook.cpp
M  +2    -0    mailcommon/filter/filteractionaddtoaddressbook.h
M  +7    -0    mailcommon/filter/filteractionbeep.cpp
M  +1    -0    mailcommon/filter/filteractionbeep.h
M  +18   -1    mailcommon/filter/filteractioncopy.cpp
M  +7    -0    mailcommon/filter/filteractioncopy.h
M  +6    -0    mailcommon/filter/filteractiondelete.cpp
M  +1    -0    mailcommon/filter/filteractiondelete.h
M  +5    -0    mailcommon/filter/filteractionexec.cpp
M  +1    -0    mailcommon/filter/filteractionexec.h
M  +5    -0    mailcommon/filter/filteractionforward.cpp
M  +1    -0    mailcommon/filter/filteractionforward.h
M  +6    -0    mailcommon/filter/filteractionmove.cpp
M  +2    -0    mailcommon/filter/filteractionmove.h
M  +6    -0    mailcommon/filter/filteractionpipethrough.cpp
M  +1    -0    mailcommon/filter/filteractionpipethrough.h
M  +6    -0    mailcommon/filter/filteractionplaysound.cpp
M  +2    -0    mailcommon/filter/filteractionplaysound.h
M  +5    -0    mailcommon/filter/filteractionredirect.cpp
M  +1    -0    mailcommon/filter/filteractionredirect.h
M  +6    -1    mailcommon/filter/filteractionremoveheader.cpp
M  +1    -0    mailcommon/filter/filteractionremoveheader.h
M  +6    -0    mailcommon/filter/filteractionreplyto.cpp
M  +1    -0    mailcommon/filter/filteractionreplyto.h
M  +6    -0    mailcommon/filter/filteractionrewriteheader.cpp
M  +1    -0    mailcommon/filter/filteractionrewriteheader.h
M  +6    -0    mailcommon/filter/filteractionsendfakedisposition.cpp
M  +1    -0    mailcommon/filter/filteractionsendfakedisposition.h
M  +8    -2    mailcommon/filter/filteractionsendreceipt.cpp
M  +1    -0    mailcommon/filter/filteractionsendreceipt.h
M  +6    -0    mailcommon/filter/filteractionsetidentity.cpp
M  +1    -0    mailcommon/filter/filteractionsetidentity.h
M  +6    -0    mailcommon/filter/filteractionsetstatus.cpp
M  +1    -0    mailcommon/filter/filteractionsetstatus.h
M  +6    -0    mailcommon/filter/filteractionsettransport.cpp
M  +1    -0    mailcommon/filter/filteractionsettransport.h
M  +5    -0    mailcommon/filter/filteractionstatus.cpp
M  +2    -0    mailcommon/filter/filteractionstatus.h
M  +5    -2    mailcommon/filter/filteractionunsetstatus.cpp
M  +2    -1    mailcommon/filter/filteractionunsetstatus.h
M  +4    -4    mailcommon/filter/filterimporter/filterimporterevolution.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimporterprocmail.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimportersylpheed.cpp
M  +2    -2    mailcommon/filter/filtermanager.cpp
M  +2    -8    mailcommon/filter/filtermanager.h
M  +8    -2    mailcommon/filter/itemcontext.cpp
M  +8    -1    mailcommon/filter/itemcontext.h
M  +10   -17   mailcommon/filter/kmfilterdialog.cpp
M  +2    -1    mailcommon/filter/kmfilterdialog.h
M  +25   -8    mailcommon/filter/mailfilter.cpp
M  +4    -3    mailcommon/filter/mailfilter.h
M  +48   -20   mailcommon/searchpattern.cpp
M  +34   -11   mailcommon/searchpattern.h
M  +1    -1    mailcommon/tests/searchpatterntest.cpp
M  +78   -47   mailfilteragent/filtermanager.cpp
M  +10   -8    mailfilteragent/filtermanager.h
M  +8    -6    mailfilteragent/mailfilteragent.cpp
M  +3    -0    mailfilteragent/mailfilteragent.h

http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69
Comment 23 András Manţia 2012-07-06 09:13:36 UTC
Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia.
Committed on 06/07/2012 at 11:12.
Pushed by amantia into branch 'KDE/4.8'.

Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport.
An unwanted side-effect might be that more data is downloaded than before,
based on the type of filters used.

FIXED-IN: 4.8.5
Related: bug 287752, bug 292283, bug 288109, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682

M  +0    -1    mailfilteragent/mailfilteragent.cpp

http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec