Bug 302337

Summary:	In IMAP account Kmail2 does not show message body, but only headers and subject
Product:	[Applications] kmail2	Reporter:	GK <gkourtev>
Component:	filtering	Assignee:	kdepim bugs <kdepim-bugs>
Status:	RESOLVED FIXED
Severity:	major	CC:	markotahal
Priority:	NOR
Version:	4.8.4
Target Milestone:	---
Platform:	Ubuntu
OS:	Linux
Latest Commit:	http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69	Version Fixed In:	4.9
Sentry Crash Report:

Description GK 2012-06-22 09:23:23 UTC

After a fresh installation of Kubuntu 12.04, IMAP gmail account works well in Kmail2. After few days the new messages come with not body text nor attachments. Only sender and headers are visible. Restart of Akonadi does not help. No errors are displayed. Same messages are visible through gmail web interface. Old messages are not affected and are with visible text.

Read bugs 289734 and 298962 but there are not 100% same nor offering solutions. 

Reproducible: Always

Steps to Reproduce:
1. Open kmail.
2. Wait for folder synchronization.
3. Click on any new gmail message received.
Actual Results:  
Message header is displayed with no body text and no attachments.

Expected Results:  
Message body text should be visible and attachments too.

Read bugs 289734 and 298962 but there are not 100% same nor offering solutions (one of these dates back to 2009).

Comment 1 GK 2012-06-22 09:30:30 UTC

While just filed the bug, I noticed that in my gmail 'ALL MAIL' folder I have all messages duplicated.  one has the Spamassanin in the header, the other does not. I set up the spam filter to exclude the gmail account and the messages started to appear normally (with body text and attachments). So the issue seems to be on the way the spam filter is treating messages.

Comment 2 András Manţia 2012-07-04 15:36:10 UTC

Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'master'.

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682

M  +7    -11   kmail/kmcommands.cpp
M  +10   -7    kmail/kmcommands.h
M  +1    -1    kmail/kmmainwidget.cpp
M  +1    -5    mailcommon/filter/filteraction.cpp
M  +6    -5    mailcommon/filter/filteraction.h
M  +6    -0    mailcommon/filter/filteractionaddheader.cpp
M  +2    -0    mailcommon/filter/filteractionaddheader.h
M  +2    -2    mailcommon/filter/filteractionaddtag.cpp
M  +1    -1    mailcommon/filter/filteractionaddtag.h
M  +9    -3    mailcommon/filter/filteractionaddtoaddressbook.cpp
M  +2    -0    mailcommon/filter/filteractionaddtoaddressbook.h
M  +7    -0    mailcommon/filter/filteractionbeep.cpp
M  +1    -0    mailcommon/filter/filteractionbeep.h
M  +18   -1    mailcommon/filter/filteractioncopy.cpp
M  +7    -0    mailcommon/filter/filteractioncopy.h
M  +6    -0    mailcommon/filter/filteractiondelete.cpp
M  +1    -0    mailcommon/filter/filteractiondelete.h
M  +5    -0    mailcommon/filter/filteractionexec.cpp
M  +1    -0    mailcommon/filter/filteractionexec.h
M  +5    -0    mailcommon/filter/filteractionforward.cpp
M  +1    -0    mailcommon/filter/filteractionforward.h
M  +6    -0    mailcommon/filter/filteractionmove.cpp
M  +2    -0    mailcommon/filter/filteractionmove.h
M  +6    -0    mailcommon/filter/filteractionpipethrough.cpp
M  +1    -0    mailcommon/filter/filteractionpipethrough.h
M  +6    -0    mailcommon/filter/filteractionplaysound.cpp
M  +2    -0    mailcommon/filter/filteractionplaysound.h
M  +5    -0    mailcommon/filter/filteractionredirect.cpp
M  +1    -0    mailcommon/filter/filteractionredirect.h
M  +6    -1    mailcommon/filter/filteractionremoveheader.cpp
M  +1    -0    mailcommon/filter/filteractionremoveheader.h
M  +6    -0    mailcommon/filter/filteractionreplyto.cpp
M  +1    -0    mailcommon/filter/filteractionreplyto.h
M  +6    -0    mailcommon/filter/filteractionrewriteheader.cpp
M  +1    -0    mailcommon/filter/filteractionrewriteheader.h
M  +6    -0    mailcommon/filter/filteractionsendfakedisposition.cpp
M  +1    -0    mailcommon/filter/filteractionsendfakedisposition.h
M  +8    -2    mailcommon/filter/filteractionsendreceipt.cpp
M  +1    -0    mailcommon/filter/filteractionsendreceipt.h
M  +6    -0    mailcommon/filter/filteractionsetidentity.cpp
M  +1    -0    mailcommon/filter/filteractionsetidentity.h
M  +6    -0    mailcommon/filter/filteractionsetstatus.cpp
M  +1    -0    mailcommon/filter/filteractionsetstatus.h
M  +6    -0    mailcommon/filter/filteractionsettransport.cpp
M  +1    -0    mailcommon/filter/filteractionsettransport.h
M  +5    -0    mailcommon/filter/filteractionstatus.cpp
M  +2    -0    mailcommon/filter/filteractionstatus.h
M  +5    -2    mailcommon/filter/filteractionunsetstatus.cpp
M  +2    -1    mailcommon/filter/filteractionunsetstatus.h
M  +4    -4    mailcommon/filter/filterimporter/filterimporterevolution.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimporterprocmail.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimportersylpheed.cpp
M  +2    -2    mailcommon/filter/filtermanager.cpp
M  +2    -8    mailcommon/filter/filtermanager.h
M  +8    -2    mailcommon/filter/itemcontext.cpp
M  +8    -1    mailcommon/filter/itemcontext.h
M  +10   -17   mailcommon/filter/kmfilterdialog.cpp
M  +2    -1    mailcommon/filter/kmfilterdialog.h
M  +25   -8    mailcommon/filter/mailfilter.cpp
M  +4    -3    mailcommon/filter/mailfilter.h
M  +48   -20   mailcommon/searchpattern.cpp
M  +34   -11   mailcommon/searchpattern.h
M  +1    -1    mailcommon/tests/searchpatterntest.cpp
M  +78   -47   mailfilteragent/filtermanager.cpp
M  +10   -8    mailfilteragent/filtermanager.h
M  +8    -6    mailfilteragent/mailfilteragent.cpp
M  +3    -0    mailfilteragent/mailfilteragent.h

http://commits.kde.org/kdepim/d64d61b470c56b92294f6adee6d74305a217628d

Comment 3 András Manţia 2012-07-05 11:40:44 UTC

Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'KDE/4.9'.

(Backport filter fix from master)

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682
(cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d)

FIXED-IN: 4.9

M  +7    -11   kmail/kmcommands.cpp
M  +10   -7    kmail/kmcommands.h
M  +1    -1    kmail/kmmainwidget.cpp
M  +1    -5    mailcommon/filter/filteraction.cpp
M  +6    -5    mailcommon/filter/filteraction.h
M  +6    -0    mailcommon/filter/filteractionaddheader.cpp
M  +2    -0    mailcommon/filter/filteractionaddheader.h
M  +2    -2    mailcommon/filter/filteractionaddtag.cpp
M  +1    -1    mailcommon/filter/filteractionaddtag.h
M  +9    -3    mailcommon/filter/filteractionaddtoaddressbook.cpp
M  +2    -0    mailcommon/filter/filteractionaddtoaddressbook.h
M  +7    -0    mailcommon/filter/filteractionbeep.cpp
M  +1    -0    mailcommon/filter/filteractionbeep.h
M  +18   -1    mailcommon/filter/filteractioncopy.cpp
M  +7    -0    mailcommon/filter/filteractioncopy.h
M  +6    -0    mailcommon/filter/filteractiondelete.cpp
M  +1    -0    mailcommon/filter/filteractiondelete.h
M  +5    -0    mailcommon/filter/filteractionexec.cpp
M  +1    -0    mailcommon/filter/filteractionexec.h
M  +5    -0    mailcommon/filter/filteractionforward.cpp
M  +1    -0    mailcommon/filter/filteractionforward.h
M  +6    -0    mailcommon/filter/filteractionmove.cpp
M  +2    -0    mailcommon/filter/filteractionmove.h
M  +6    -0    mailcommon/filter/filteractionpipethrough.cpp
M  +1    -0    mailcommon/filter/filteractionpipethrough.h
M  +6    -0    mailcommon/filter/filteractionplaysound.cpp
M  +2    -0    mailcommon/filter/filteractionplaysound.h
M  +5    -0    mailcommon/filter/filteractionredirect.cpp
M  +1    -0    mailcommon/filter/filteractionredirect.h
M  +6    -1    mailcommon/filter/filteractionremoveheader.cpp
M  +1    -0    mailcommon/filter/filteractionremoveheader.h
M  +6    -0    mailcommon/filter/filteractionreplyto.cpp
M  +1    -0    mailcommon/filter/filteractionreplyto.h
M  +6    -0    mailcommon/filter/filteractionrewriteheader.cpp
M  +1    -0    mailcommon/filter/filteractionrewriteheader.h
M  +6    -0    mailcommon/filter/filteractionsendfakedisposition.cpp
M  +1    -0    mailcommon/filter/filteractionsendfakedisposition.h
M  +8    -2    mailcommon/filter/filteractionsendreceipt.cpp
M  +1    -0    mailcommon/filter/filteractionsendreceipt.h
M  +6    -0    mailcommon/filter/filteractionsetidentity.cpp
M  +1    -0    mailcommon/filter/filteractionsetidentity.h
M  +6    -0    mailcommon/filter/filteractionsetstatus.cpp
M  +1    -0    mailcommon/filter/filteractionsetstatus.h
M  +6    -0    mailcommon/filter/filteractionsettransport.cpp
M  +1    -0    mailcommon/filter/filteractionsettransport.h
M  +5    -0    mailcommon/filter/filteractionstatus.cpp
M  +2    -0    mailcommon/filter/filteractionstatus.h
M  +5    -2    mailcommon/filter/filteractionunsetstatus.cpp
M  +2    -1    mailcommon/filter/filteractionunsetstatus.h
M  +4    -4    mailcommon/filter/filterimporter/filterimporterevolution.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimporterprocmail.cpp
M  +1    -1    mailcommon/filter/filterimporter/filterimportersylpheed.cpp
M  +2    -2    mailcommon/filter/filtermanager.cpp
M  +2    -8    mailcommon/filter/filtermanager.h
M  +8    -2    mailcommon/filter/itemcontext.cpp
M  +8    -1    mailcommon/filter/itemcontext.h
M  +10   -17   mailcommon/filter/kmfilterdialog.cpp
M  +2    -1    mailcommon/filter/kmfilterdialog.h
M  +25   -8    mailcommon/filter/mailfilter.cpp
M  +4    -3    mailcommon/filter/mailfilter.h
M  +48   -20   mailcommon/searchpattern.cpp
M  +34   -11   mailcommon/searchpattern.h
M  +1    -1    mailcommon/tests/searchpatterntest.cpp
M  +78   -47   mailfilteragent/filtermanager.cpp
M  +10   -8    mailfilteragent/filtermanager.h
M  +8    -6    mailfilteragent/mailfilteragent.cpp
M  +3    -0    mailfilteragent/mailfilteragent.h

http://commits.kde.org/kdepim/e3a03f1d704838a9ced4ff41358785392774ae69

Comment 4 András Manţia 2012-07-06 09:13:44 UTC

Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia.
Committed on 06/07/2012 at 11:12.
Pushed by amantia into branch 'KDE/4.8'.

Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport.
An unwanted side-effect might be that more data is downloaded than before,
based on the type of filters used.

FIXED-IN: 4.8.5
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364, bug 283682

M  +0    -1    mailfilteragent/mailfilteragent.cpp

http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec