| Summary: | possible data loss due to filtering (headers or body of email get lost) | ||
|---|---|---|---|
| Product: | [Frameworks and Libraries] Akonadi | Reporter: | Milian Wolff <mail> |
| Component: | Mail Filter Agent | Assignee: | kdepim bugs <kdepim-bugs> |
| Status: | RESOLVED FIXED | ||
| Severity: | critical | CC: | amantia, auxsvr, ht990332, joerg.schaible, jospoortvliet, kde.org, krammer, martin.ruessler, oldium.pro, sven.burmeister |
| Priority: | VHI | ||
| Version: | 4.8 | ||
| Target Milestone: | --- | ||
| Platform: | Unlisted Binaries | ||
| OS: | Linux | ||
| Latest Commit: | http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69 | Version Fixed In: | 4.9 |
| Sentry Crash Report: | |||
| Attachments: | screenshot showing mails with lost headers | ||
|
Description
Milian Wolff
2011-11-07 21:15:30 UTC
ignore Reproducible: Didn't try, for me it's reproducible (i.e. it happens frequently) but I have yet to find out how to trigger it by hand (i.e. not randomly) Created attachment 65380 [details] screenshot showing mails with lost headers I think I just encountered data loss due to this issue, see attached screenshot. I now have about 20 emails without any headers, thankfully nothing important but this looks like a serious issue to me. I have another idea of what could be the possible issue here: I assumed the mails are not getting filtered (see first comment., my mails sometimes just stay in the pop3 inbox), furthermore due to bug 286095 I have no idea whether a mail actually gets filtered or not. Now I selected some messages and manually triggered the "apply all filters" action. Nothing happened for some time (a few seconds) and then suddenly the mails lost their headers (see screenshot). increasing priority as I encountered data loss regarding the "no items found" error message: the filter log viewer showed this in such a case: ------------------------------ ------------------------------ ------------------------------ ------------------------------ so probably it does not show any errors there :-/ starting akonadi from the cli shows some interesting debug output before I see a "no items found" error message:
QFSFileEngine::open: No file name specified
posting retrieval request for item 57572 there are 1 queues and 0 items in mine
request for item 57572 still pending - waiting
processing retrieval request for item 57572 parts: ("RFC822") of resource: "akonadi_imap_resource_2"
akonadi_imap_resource_2(23147)/libakonadi Akonadi::ResourceBase::itemRetrieved: Item does not provide part "RFC822"
continuing
request for item 57572 succeeded
akonadi_mailfilter_agent(23150): Error while moving items. 103 "Unknown error. (No items found)" to destinationCollection.id() : 95
akonadi_nepomuk_feeder(23153) ItemQueue::jobResult: "http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefLabel has a max cardinality of 1. Provided 5 values - "kde-i18n-doc-bounces@kde.org", "kde-i18n-doc-bounces@kde.org", "kde-i18n-doc-bounces@kde.org", "kde-i18n-doc-bounces@kde.org", "kde-i18n-doc-bounces@kde.org""
QFSFileEngine::open: No file name specified
QFSFileEngine::open: No file name specified
posting retrieval request for item 57575 there are 1 queues and 0 items in mine
request for item 57575 still pending - waiting
processing retrieval request for item 57575 parts: ("RFC822") of resource: "akonadi_imap_resource_2"
akonadi_imap_resource_2(23147)/libakonadi Akonadi::ResourceBase::itemRetrieved: Item does not provide part "RFC822"
continuing
request for item 57575 succeeded
akonadi_mailfilter_agent(23150): Error while moving items. 103 "Unknown error. (No items found)" to destinationCollection.id() : 95
raising severity. Same problem here, I'm trying to figure out how to use KMail2 with my mail. This issue is a show stopper for me as I cannot get to any of my incoming mail. akonadi-server-1.6.2, kdepim-runtime 4.7.97, using mixedmail resource over the old KMail dir. I mean that I have the mentioned "no items found" problem currently. There is a code that "smells" to me - I don't know what is Session or KJob, so this might not be prefect analysis: 1. kdepim-runtime/resources/shared/filestore/itemmovejob.cpp in constructor of ItemMoveJob calls session->addJob(this). 2. kdepim-runtime/resources/mixedmaildir/mixedmaildirresource.cpp, method itemMoved: 2a) it calls mStore->moveItem, which creates ItemMoveJob, which adds the job instance to session in the constructor. 2b) it connects to the job's slots. Is there some confidence that the job doesn't finish before the step 2b is executed? I _guess_ this is multi-threaded application, and from the information that the job isn't explicitly started I _think_ it is done by the session (asynchronously). So it might finish before the signal/slot is connected, so the slot isn't called. Is there any KMail/Akonadi developer to verify this finding? Thanks. bug 293768 could be related. (In reply to comment #9) > Is there some confidence that the job doesn't finish before the step 2b is > executed? I _guess_ this is multi-threaded application, and from the > information that the job isn't explicitly started I _think_ it is done by > the session (asynchronously). So it might finish before the signal/slot is > connected, so the slot isn't called. This is a single threaded application. The job is started through delayed invocation so all connects have been executed before that happens FWIW I also have a LARGE bunch of completely empty mails in my inbox. Yet if you click them you see the content... No idea if it is related but it's quite annoying ;-) http://susepaste.org/62206930 BTW it could be related to https://bugs.kde.org/show_bug.cgi?id=295484 I have the same problem with KMail 4.7.4 and an IMAP account. No filter configured. The behavior is different to Bug 295484, because the emails are not touched on the server and can be shown by other email clients. It happens almost only to multipart mails, but not to all. I have mails that look structurally almost equal, but one is shown properly and the other one not at all. Milian, did you see this bug recently? It is very unclear for me what goes on, and some of the comments seems to be unrelated (well, they might looks similar, but they only confirm the result, not the possible cause). All reporters seeing it: try to reproduce the bug, as right now I can't see anything like this. Also me clear about what filter runs for the messages that get lost (use the filter log from KMail), what kind of mailbox types you use (for inbox and for the destination, e.g imap/maildir/mixedmaildir/mbox - in case of mixedmaildir if it is a maildir or mbox) and if the mail is *really* gone from your hard driver (or from the IMAP server in case of IMAP). The best would be to have a reproducible way for the bug. I've disabled most of my filters because of this bug and hence didn't see it for some time. I'll try to re-enable the filters and see whether I can still reproduce it. Note thought that this was not a 100% reproducible issue, only some mails where affected. Anyhow, I'll let you know once I know more. I've updated meanwhile from KMail 4.7.4 to KMail 4.8.1, but I have still those emails whose body is not shown. However, this happens only for some (multi-part) mails that I have received in the first 3 months of this year. It does not happen to new mails. Additionally if I copy such a mail from the .maildir on my server to the server's inbox directory, KMail can suddenly display the body. So it seems that somehow the local cache got corrupt. What can be done to invalidate the cache? One last comment: If I take a copy (with D&D) in KMail itself, the copied email has definitely no body anymore. Is the copy within the same account? What kind of accounts are involved (source/destination)? The same IMAP account (on an intranet server running Courier), server-side subscription, offline-mode activated. I have also direct access on the mail server's file system, Courier saves in maildir format. Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia. Committed on 04/07/2012 at 17:35. Pushed by amantia into branch 'master'. A biggish refactoring of the filter functionality: 1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only). This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list) Might fix 293768. 2) Download only what is really needed for the filter and add safety checks that the mail is not modified if the requested part is missing. So far the distinction was only betwee the body and rest. This should speed up filtering. Also related to 287752, but should fix also 292283, 288109. 3) Rename qDebug->kDebug + fix some includes 4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates with empty content), eg. 275233 All bug reporters: in case you still see the bug, reopen the corresponding one. For the reports that were CC'd, but not closed, please try to reproduce and tell the result. So far the bug is fixed on master, but might be backported later to 4.9. Related: bug 287752, bug 292283, bug 288109, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682 M +7 -11 kmail/kmcommands.cpp M +10 -7 kmail/kmcommands.h M +1 -1 kmail/kmmainwidget.cpp M +1 -5 mailcommon/filter/filteraction.cpp M +6 -5 mailcommon/filter/filteraction.h M +6 -0 mailcommon/filter/filteractionaddheader.cpp M +2 -0 mailcommon/filter/filteractionaddheader.h M +2 -2 mailcommon/filter/filteractionaddtag.cpp M +1 -1 mailcommon/filter/filteractionaddtag.h M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h M +7 -0 mailcommon/filter/filteractionbeep.cpp M +1 -0 mailcommon/filter/filteractionbeep.h M +18 -1 mailcommon/filter/filteractioncopy.cpp M +7 -0 mailcommon/filter/filteractioncopy.h M +6 -0 mailcommon/filter/filteractiondelete.cpp M +1 -0 mailcommon/filter/filteractiondelete.h M +5 -0 mailcommon/filter/filteractionexec.cpp M +1 -0 mailcommon/filter/filteractionexec.h M +5 -0 mailcommon/filter/filteractionforward.cpp M +1 -0 mailcommon/filter/filteractionforward.h M +6 -0 mailcommon/filter/filteractionmove.cpp M +2 -0 mailcommon/filter/filteractionmove.h M +6 -0 mailcommon/filter/filteractionpipethrough.cpp M +1 -0 mailcommon/filter/filteractionpipethrough.h M +6 -0 mailcommon/filter/filteractionplaysound.cpp M +2 -0 mailcommon/filter/filteractionplaysound.h M +5 -0 mailcommon/filter/filteractionredirect.cpp M +1 -0 mailcommon/filter/filteractionredirect.h M +6 -1 mailcommon/filter/filteractionremoveheader.cpp M +1 -0 mailcommon/filter/filteractionremoveheader.h M +6 -0 mailcommon/filter/filteractionreplyto.cpp M +1 -0 mailcommon/filter/filteractionreplyto.h M +6 -0 mailcommon/filter/filteractionrewriteheader.cpp M +1 -0 mailcommon/filter/filteractionrewriteheader.h M +6 -0 mailcommon/filter/filteractionsendfakedisposition.cpp M +1 -0 mailcommon/filter/filteractionsendfakedisposition.h M +8 -2 mailcommon/filter/filteractionsendreceipt.cpp M +1 -0 mailcommon/filter/filteractionsendreceipt.h M +6 -0 mailcommon/filter/filteractionsetidentity.cpp M +1 -0 mailcommon/filter/filteractionsetidentity.h M +6 -0 mailcommon/filter/filteractionsetstatus.cpp M +1 -0 mailcommon/filter/filteractionsetstatus.h M +6 -0 mailcommon/filter/filteractionsettransport.cpp M +1 -0 mailcommon/filter/filteractionsettransport.h M +5 -0 mailcommon/filter/filteractionstatus.cpp M +2 -0 mailcommon/filter/filteractionstatus.h M +5 -2 mailcommon/filter/filteractionunsetstatus.cpp M +2 -1 mailcommon/filter/filteractionunsetstatus.h M +4 -4 mailcommon/filter/filterimporter/filterimporterevolution.cpp M +1 -1 mailcommon/filter/filterimporter/filterimporterprocmail.cpp M +1 -1 mailcommon/filter/filterimporter/filterimportersylpheed.cpp M +2 -2 mailcommon/filter/filtermanager.cpp M +2 -8 mailcommon/filter/filtermanager.h M +8 -2 mailcommon/filter/itemcontext.cpp M +8 -1 mailcommon/filter/itemcontext.h M +10 -17 mailcommon/filter/kmfilterdialog.cpp M +2 -1 mailcommon/filter/kmfilterdialog.h M +25 -8 mailcommon/filter/mailfilter.cpp M +4 -3 mailcommon/filter/mailfilter.h M +48 -20 mailcommon/searchpattern.cpp M +34 -11 mailcommon/searchpattern.h M +1 -1 mailcommon/tests/searchpatterntest.cpp M +78 -47 mailfilteragent/filtermanager.cpp M +10 -8 mailfilteragent/filtermanager.h M +8 -6 mailfilteragent/mailfilteragent.cpp M +3 -0 mailfilteragent/mailfilteragent.h http://commits.kde.org/kdepim/d64d61b470c56b92294f6adee6d74305a217628d Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia. Committed on 04/07/2012 at 17:35. Pushed by amantia into branch 'KDE/4.9'. (Backport filter fix from master) A biggish refactoring of the filter functionality: 1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only). This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list) Might fix 293768. 2) Download only what is really needed for the filter and add safety checks that the mail is not modified if the requested part is missing. So far the distinction was only betwee the body and rest. This should speed up filtering. Also related to 287752, but should fix also 292283, 288109. 3) Rename qDebug->kDebug + fix some includes 4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates with empty content), eg. 275233 All bug reporters: in case you still see the bug, reopen the corresponding one. For the reports that were CC'd, but not closed, please try to reproduce and tell the result. So far the bug is fixed on master, but might be backported later to 4.9. Related: bug 287752, bug 292283, bug 288109, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682 (cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d) FIXED-IN: 4.9 M +7 -11 kmail/kmcommands.cpp M +10 -7 kmail/kmcommands.h M +1 -1 kmail/kmmainwidget.cpp M +1 -5 mailcommon/filter/filteraction.cpp M +6 -5 mailcommon/filter/filteraction.h M +6 -0 mailcommon/filter/filteractionaddheader.cpp M +2 -0 mailcommon/filter/filteractionaddheader.h M +2 -2 mailcommon/filter/filteractionaddtag.cpp M +1 -1 mailcommon/filter/filteractionaddtag.h M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h M +7 -0 mailcommon/filter/filteractionbeep.cpp M +1 -0 mailcommon/filter/filteractionbeep.h M +18 -1 mailcommon/filter/filteractioncopy.cpp M +7 -0 mailcommon/filter/filteractioncopy.h M +6 -0 mailcommon/filter/filteractiondelete.cpp M +1 -0 mailcommon/filter/filteractiondelete.h M +5 -0 mailcommon/filter/filteractionexec.cpp M +1 -0 mailcommon/filter/filteractionexec.h M +5 -0 mailcommon/filter/filteractionforward.cpp M +1 -0 mailcommon/filter/filteractionforward.h M +6 -0 mailcommon/filter/filteractionmove.cpp M +2 -0 mailcommon/filter/filteractionmove.h M +6 -0 mailcommon/filter/filteractionpipethrough.cpp M +1 -0 mailcommon/filter/filteractionpipethrough.h M +6 -0 mailcommon/filter/filteractionplaysound.cpp M +2 -0 mailcommon/filter/filteractionplaysound.h M +5 -0 mailcommon/filter/filteractionredirect.cpp M +1 -0 mailcommon/filter/filteractionredirect.h M +6 -1 mailcommon/filter/filteractionremoveheader.cpp M +1 -0 mailcommon/filter/filteractionremoveheader.h M +6 -0 mailcommon/filter/filteractionreplyto.cpp M +1 -0 mailcommon/filter/filteractionreplyto.h M +6 -0 mailcommon/filter/filteractionrewriteheader.cpp M +1 -0 mailcommon/filter/filteractionrewriteheader.h M +6 -0 mailcommon/filter/filteractionsendfakedisposition.cpp M +1 -0 mailcommon/filter/filteractionsendfakedisposition.h M +8 -2 mailcommon/filter/filteractionsendreceipt.cpp M +1 -0 mailcommon/filter/filteractionsendreceipt.h M +6 -0 mailcommon/filter/filteractionsetidentity.cpp M +1 -0 mailcommon/filter/filteractionsetidentity.h M +6 -0 mailcommon/filter/filteractionsetstatus.cpp M +1 -0 mailcommon/filter/filteractionsetstatus.h M +6 -0 mailcommon/filter/filteractionsettransport.cpp M +1 -0 mailcommon/filter/filteractionsettransport.h M +5 -0 mailcommon/filter/filteractionstatus.cpp M +2 -0 mailcommon/filter/filteractionstatus.h M +5 -2 mailcommon/filter/filteractionunsetstatus.cpp M +2 -1 mailcommon/filter/filteractionunsetstatus.h M +4 -4 mailcommon/filter/filterimporter/filterimporterevolution.cpp M +1 -1 mailcommon/filter/filterimporter/filterimporterprocmail.cpp M +1 -1 mailcommon/filter/filterimporter/filterimportersylpheed.cpp M +2 -2 mailcommon/filter/filtermanager.cpp M +2 -8 mailcommon/filter/filtermanager.h M +8 -2 mailcommon/filter/itemcontext.cpp M +8 -1 mailcommon/filter/itemcontext.h M +10 -17 mailcommon/filter/kmfilterdialog.cpp M +2 -1 mailcommon/filter/kmfilterdialog.h M +25 -8 mailcommon/filter/mailfilter.cpp M +4 -3 mailcommon/filter/mailfilter.h M +48 -20 mailcommon/searchpattern.cpp M +34 -11 mailcommon/searchpattern.h M +1 -1 mailcommon/tests/searchpatterntest.cpp M +78 -47 mailfilteragent/filtermanager.cpp M +10 -8 mailfilteragent/filtermanager.h M +8 -6 mailfilteragent/mailfilteragent.cpp M +3 -0 mailfilteragent/mailfilteragent.h http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69 Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia. Committed on 06/07/2012 at 11:12. Pushed by amantia into branch 'KDE/4.8'. Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport. An unwanted side-effect might be that more data is downloaded than before, based on the type of filters used. FIXED-IN: 4.8.5 Related: bug 287752, bug 292283, bug 288109, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682 M +0 -1 mailfilteragent/mailfilteragent.cpp http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec |