Bug 293039

Summary: KMail fails to parse emails properly in some cases
Product: [Applications] kmail2 Reporter: Moritz <m.o>
Component: generalAssignee: kdepim bugs <kdepim-bugs>
Status: RESOLVED DUPLICATE    
Severity: normal CC: andreas, frederic.coiffier, fresh.road1272, jemand, m.o, m4rkusxxl, marcinkocur, montel, quazgar, sknauss
Priority: NOR    
Version: 4.8   
Target Milestone: ---   
Platform: Arch Linux   
OS: Linux   
Latest Commit: Version Fixed In:
Sentry Crash Report:
Attachments: Sample email in the format KMail stored it.

Description Moritz 2012-02-01 12:19:58 UTC
Version:           4.8 (using KDE 4.8.0) 
OS:                Linux

KMail fails to parse emails properly when (this is the way I see it) there are different message parts in one email, e.g. plaintext and HTML or plaintext and attachement, i.e.: for a lot of mails.

This is the exact output for a mail where parsing (and encoding) fails:
http://pastebin.com/YnqU1FtW

This is the output for a mail consisting of plaintext and an attachement (anonymised):
http://pastebin.com/cQ4miUZ5

These badly displayed emails don't always start with "This is a multi-part message in MIME format.", see the recent Linux Foundation Newsletter:
http://pastebin.com/466LanBS

The same happens when there is a signature:
http://pastebin.com/xzd8PqCj

One more, just the top lines this time:
http://pastebin.com/Jq0JRsgq


I just discovered that it may not perfectly reproducable. I get this newsletter One Word A Day and it was parsed correctly yesterday, but not today.
Yesterday:
http://pastebin.com/eRVpta85

Today:
http://pastebin.com/TaCqWE7L

Reproducible: Sometimes

Steps to Reproduce:
Receive a multi-part email.

Actual Results:  
See pastebin examples.

Expected Results:  
An email properly parsed, readable, with the right encoding, giving you the ability to choose between e.g. plaintext and HTML or allowing you to open an attachement.

This is only happening since 4.8.0.

I'm not sure about the severity of this bug, but since I seem to lose my attachement (it may not be gone but I cannot open it which I would like to in the next days or I won't get my salary :) ), I set the severity to major.
Feel free to reduce severity if this is inappropriate.
Comment 1 Laurent Montel 2012-02-01 12:45:54 UTC
please attach to bug report your email (as mbox if possible)
pastbin will remove soon
Comment 2 Moritz 2012-02-01 12:54:38 UTC
Hm, it says "Expires: Never" on pastebin?
Comment 3 Moritz 2012-02-01 13:16:43 UTC
Created attachment 68399 [details]
Sample email in the format KMail stored it.
Comment 4 Torgny Nyblom 2012-02-01 19:05:49 UTC
Do you have any filters? Some filters sometimes mangle the message so that the mime header in the main header is converted into text/plain.

The attached mail is of text/plain type and should be rendered as shown. The embedded mime parts should only be interpreted as mime parts when the parent says so and in this case it doesn't.
Comment 5 Moritz 2012-02-01 19:25:30 UTC
Yes, I'm using bogofilter (and have been for at least 2-3 years. Last update was on 12th of January and I haven't had this problem before 4.8, so it'd be strange if it was bogofilter's fault).
Besides that I'm using some custom filters using only standard KMail rules like "Subject contains 'orange' -> move to 'fruit'", but that shouldn't be a problem, right?
I'll turn off bogofilter to see if that changes anything.

By the way, in the meantime I've received multi-part emails and they were parsed correctly. Maybe they got a different handling by bogofilter. We'll see.
Comment 6 Moritz 2012-02-01 19:31:40 UTC
I just sent myself a test email containing a PGP signature and this time it was parsed correctly.
If it turns out that there are no mis-parsed emails anymore: is this a bogofilter bug now even though this is only happening since the 4.8 update?
Comment 7 Torgny Nyblom 2012-02-01 19:36:57 UTC
Not sure where this bug is from only that it is related to filters.
Comment 8 Frédéric COIFFIER 2012-02-02 08:14:47 UTC
In my case, the first problem appeared 2 days ago. The same day, I have enabled Bogofilter (but I haven't changed any other filter).
I confirm that I had 2 problems on emails which embeds plain text and HTML version : the content of one of this mail is displayed in Kmail just after this line (in the source) :

X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.2
Comment 9 Frits Spieker 2012-02-02 13:19:02 UTC
Seems to be exactly the same problem I am having. See bug: 292767
Comment 10 Andreas Mahel 2012-02-03 17:45:05 UTC
i have experienced the same behaviour, and I could narrow it down to the "Bogofilter Check" mail filter.
I've sent myself a multi part message from my web mail client, and with the filter turned off, the mail looked as expected, with the filter turned on, it showed the behaviour as above.

Looking at the raw mail content, the primary difference was that in the bogo-filtered message, the Content-Type in the mail header was:
    Content-Type: text/plain; charset="US-ASCII"
whereas in unfiltered message it looked like
    Content-Type: multipart/mixed; boundary="========GMX112281328289219935858"

I checked bogofilter standalone as well, running a "bogofilter -p -e" against an mbox file of the "unfiltered" test mail - the output here shows the correct header Content-Type .

So it looks that somewhere during the bogofilter run the Content-Type is being changed by kmail (or akonadi?).

If needed, I can as well attach the mbox files, or run additional tests.
Comment 11 Torgny Nyblom 2012-02-05 13:13:38 UTC
*** Bug 293359 has been marked as a duplicate of this bug. ***
Comment 12 Moritz 2012-02-09 02:16:53 UTC
*** This bug has been confirmed by popular vote. ***
Comment 13 Markus 2012-02-14 12:04:34 UTC
I added a manual rule to pipe everything through a script:
#!/bin/sh
TEMPFILE=$(mktemp --tmpdir=/tmp/ kmail.test.XXXXXX)
cat > $TEMPFILE
cat $TEMPFILE

You can see, the first time the script gets the original message. The second time every "options" get put in one line each. And somehow the message has doubled the size. (seems to just be copies of the body)
And option may have a different spelling (some letters became upper case).

So kmail seems to parse a mail from a pipe differently than a mail directly from the server.

I couldnt reproduce the missing/wrong "Content-Type" but I do have the same problem with spamassassin.
Comment 14 Markus 2012-02-26 12:26:22 UTC
kmail is still crunching my mails... anybody?!
Is it fixed in trunk? In upcoming 4.8.1?
Comment 15 Markus 2012-03-05 11:23:11 UTC
Perhaps dup of Bug#291171 and fixed in 4.8.1?
Comment 16 Moritz 2012-03-05 11:31:46 UTC
On http://cgbdx.wordpress.com/2012/03/01/kdepim-bug-fixes-in-kde-4-8-1-8/, a bug fix for bug 291171 is mentioned which seems to be this problem, so I'd say: yes, this is fixed.
Comment 17 mkkot 2012-03-08 17:58:13 UTC
Kontact 4.8.1

I just got a notification from facebook. Before I had set up bogofilter back. It came as 2 e-mails, one broken, kmail's browser shows it as:
--b1_2c23527440a52140ce850074979b0adb
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

and the second one which is ok. Here's a source fragment:

X-Bogosity: Unsure, tests=bogofilter, spamicity=0.491833, version=1.2.2
--b1_2c23527440a52140ce850074979b0adb
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
http://www.facebook.com/n/?permalink.php&story_fbid=3D199129653525996&id=

The broken one source fragment:

X-Bogosity: Ham, tests=bogofilter, spamicity=0.042048, version=1.2.2
--b1_2c23527440a52140ce850074979b0adb
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
http://www.facebook.com/n/?permalink.php&story_fbid=3D199129653525996&id=
Comment 18 Frédéric COIFFIER 2012-03-13 16:18:44 UTC
I confirm that the problem still exists in kmail 4.8.1 (Gentoo) : As soon as I enable "Bogofilter" filters, my newly received emails are corrupted.
Comment 19 Moritz 2012-03-15 18:50:50 UTC
Just thought I'd add that on my machine it works just fine with Bogofilter enabled. It does even actually filter spam (which it had stopped doing some time ago).
Only it doesn't mark new spam as unread even though the settings are correct and it crashes twice a day but that's a different story…

Maybe you have to completely remove its filters and run the spam wizard? At least that's what I did.
Comment 20 Frédéric COIFFIER 2012-04-10 07:13:18 UTC
I can't reproduce the problem with kmail 4.8.2.
So, it seems to be fixed.
Comment 21 mkkot 2012-04-10 07:39:08 UTC
Maybe it's because spam filter doesn't work ;)
Comment 22 quazgar 2013-03-28 06:37:11 UTC
Duplicate of bug #292767?
Comment 23 Sandro Knauß 2013-03-28 08:59:23 UTC

*** This bug has been marked as a duplicate of bug 292767 ***