Bug 86302 - KMail cannot handle malformed wrapped header lines
Summary: KMail cannot handle malformed wrapped header lines
Status: RESOLVED FIXED
Alias: None
Product: kmail
Classification: Applications
Component: mime (show other bugs)
Version: 1.10.3
Platform: Debian stable Linux
: NOR wishlist
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords: triaged
: 129032 158284 175965 (view as bug list)
Depends on:
Blocks:
 
Reported: 2004-07-30 18:32 UTC by michael
Modified: 2010-01-20 14:03 UTC (History)
10 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
testcase with 996 recipients that does *not* reproduce the bug in 4.1.1 (mbox file) (25.39 KB, application/mbox)
2008-09-14 22:26 UTC, George Kiagiadakis
Details
Original message from free mail (4.27 KB, message/rfc822)
2008-11-28 14:19 UTC, Oldřich Jedlička
Details
Patch for malformed wrapping of encoded-words (international headers) (2.16 KB, patch)
2010-01-05 22:15 UTC, Oldřich Jedlička
Details
[mimelib-4.3.85] Patch for malformed wrapping of encoded-words (international headers) (2.69 KB, patch)
2010-01-05 23:38 UTC, Oldřich Jedlička
Details
[kmail-4.3.85] Patch for malformed wrapping of encoded-words (international headers) (1.37 KB, patch)
2010-01-06 01:00 UTC, Oldřich Jedlička
Details

Note You need to log in before you can comment on or make changes to this bug.
Description michael 2004-07-30 18:32:41 UTC
Version:            (using KDE KDE 3.2.2)
Installed from:    Debian stable Packages
OS:                Linux

This morning after running kmail from debian woody packages on download.kde.org for 3.2.2, I started getting a bunch of corrupted emails.. Most had no subject, etc..

eg.

Delivered-To: michael@linuxmagic.com
X-AntiVirus: scanned for viruses by AMaViS 0.2.1 (http://amavis.org/)
Status: R
X-Status: N
X-KMail-EncryptionState:  
X-KMail-SignatureState:  
X-KMail-MDN-Sent: 

@itomig.de>
Reply-To: stefan.munz@itomig.de
Organization: ITOMIG
To: wine-users@winehq.org
Subject: Re: [Wine]regedit not working
User-Agent: KMail/1.5.4


Have been getting random problems with this, but now it seems much more pronounced..  This is jsut a quick report, I need to get another message which hasn't been pulled down, so I can send in a comparison report.
Comment 1 Olivier Goffart 2005-06-28 23:12:21 UTC
I got the same problem on KDE 3.4.1

KMail added his own header at the wrong place, before the end of normal header
(in that case, bad added headers was:
Status: R
X-Status: NC
X-KMail-EncryptionState:  
X-KMail-SignatureState:  
X-KMail-MDN-Sent:  
)

Theses header was introduce in the middle of the To:  header. splitting in two the mail of someone.

So i saw the end of the header with the mail in the kmail view, and the html was not parsed.
(It was a hotmail html message full of useless tag, very difficult to read as it)

As it involve personal data, i'll not attach the mail here, but i can provide it in private to someone.

Comment 2 vlado plsek 2006-05-09 14:16:01 UTC
I have the same problem.
kmail 1.9.1 on upgraded suse10.

Comment 3 Pino Toscano 2007-04-20 23:34:35 UTC
*** Bug 129032 has been marked as a duplicate of this bug. ***
Comment 4 H.H. 2007-05-23 21:23:18 UTC
I have the same problem with kmail 1.9.1 , kde-3.5.7 on suse-10. My spam-filter-settings do not apply anymore, because everything from "Date: Wed..." is recognized as mail-body. Example-extract from the spam-mail-header with blank line:


X-Status: NC
X-KMail-EncryptionState:  
X-KMail-SignatureState:  
X-KMail-MDN-Sent:  

Date: Wed, 23 May 2007 14:41:04 -0500
Message-Id: <10186.downtown@incorporable>
X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on
Comment 5 yupa 2008-02-02 22:11:32 UTC
I observed that Kmail places these headers (Status, X-Status, X-KMail-EncryptionState, X-KMail-SignatureState, X-KMail-MDN-Sent) splitting the list of recipients (To: header) after 994 characters.

This behaviour is still present in kmail 3.5.8 both in Debian and Opensuse 10.3

It is harmless in most cases, but if the list of recipients is very large all the headers after go corrupted.
Comment 6 Thomas McGuire 2008-02-27 21:56:38 UTC
*** Bug 158284 has been marked as a duplicate of this bug. ***
Comment 7 George Kiagiadakis 2008-09-14 22:23:51 UTC
I cannot reproduce this with kmail 1.10.1 (kde 4.1.1). What I did was that I sent a test email to myself, saved the source, removed the kmail headers, programmatically added 995 new recipients, added an mbox-style From line in the first line and did "cat testcase > /var/mail/gkiagia", then checked mail for my configured local account. Although the to list was far bigger than 994 characters (it was 996 lines!), the kmail headers were correctly added at the end of the headers list and the mail appeared correctly.

I will attach the testcase I used. It would be useful someone to double-check that this file causes problems in 3.5.8 but not in 4.1.1.
Comment 8 George Kiagiadakis 2008-09-14 22:26:00 UTC
Created attachment 27415 [details]
testcase with 996 recipients that does *not* reproduce the bug in 4.1.1 (mbox file)
Comment 9 yupa 2008-09-14 23:03:06 UTC
Just tried your test, but it doesn't reproduce the bug neither in kmail 3.5.9... 
Comment 10 Jaime Torres 2008-09-17 20:11:39 UTC
Also works for me in kmail 1.10.1. 
Comment 11 Christophe Marin 2008-11-24 10:13:47 UTC
*** Bug 175965 has been marked as a duplicate of this bug. ***
Comment 12 Christophe Marin 2008-11-24 10:14:10 UTC
Reopen due to bug 175965
Comment 13 kablooie 2008-11-28 00:19:12 UTC
same problem here as described in bug 175965... I also use yahoo mail, maybe that's the point?
Comment 14 Oldřich Jedlička 2008-11-28 13:52:29 UTC
I see this problem for quite a long time, so now I think it is a good time to get involved.

The problem is really in split header fields. I have free mail and KMail reading from it. The source mail (on free mail) contains this:

...
X-Mailer: Apple Mail (2.919.2)
X-Mailman-Approved-At: Thu, 13 Nov 2008 16:45:38 +0100
X-Content-Filtered-By: Mailman/MimeDel 2.1.9
Subject: =?us-ascii?Q?=5BMPlayer=2Ddev=2Deng=5D=20development=20project=20=22mplayer=20encapsulation=20for
=09Apple=20TV=22?=
X-Beenthere: mplayer-dev-eng@mplayerhq.hu
X-Mailman-Version: 2.1.9
...

But the KMail's message suddenly contains:

...
X-Mailer: Apple Mail (2.919.2)
X-Mailman-Approved-At: Thu, 13 Nov 2008 16:45:38 +0100
X-Content-Filtered-By: Mailman/MimeDel 2.1.9
Subject: =?us-ascii?Q?=5BMPlayer=2Ddev=2Deng=5D=20development=20project=20=22mplayer=20encapsulation=20for
X-UID: 
Status: RO
X-Status: RC
X-KMail-EncryptionState: N
X-KMail-SignatureState: N
X-KMail-MDN-Sent:  

=09Apple=20TV=22?=
X-Beenthere: mplayer-dev-eng@mplayerhq.hu
X-Mailman-Version: 2.1.9
...

So the problem is KMail adding it's headers in the middle of other header field content.
Comment 15 Oldřich Jedlička 2008-11-28 14:07:24 UTC
Note: the "Subject:" line is split onto two lines only, not as you can see here - the "=?us-ascii?Q?=..." string is on the same line.
Comment 16 Oldřich Jedlička 2008-11-28 14:19:59 UTC
Created attachment 28877 [details]
Original message from free mail
Comment 17 Cartwright 2008-12-02 15:34:19 UTC
I confirm the problem with the current svn snapshot.
Comment 18 Johannes 2008-12-02 15:41:08 UTC
I experience the same problem with the KMail version installed with Kubuntu (Intrepid Ibex).
Comment 19 Cletus 2008-12-10 09:22:12 UTC
I see exactly the same behaviour with Kubuntu Intrepid Ibex packages, KDE 4.1.3, KMail 1.10.3. I would suggest to change the title of this bug since it has been proven that long To: lists are not the reason for corrupted emails.
Comment 20 Oldřich Jedlička 2008-12-10 10:51:58 UTC
It would be interresting if all the original messages (before KMail breaks them) have the same problem: multi-line international header in this form:

<Header>: =?<charset>?Q?<some characters><new-line>
<some characters>?=

The problem I see here is that the second line is not space/tab indented (it should be for header continuation). Anyway, the end can be found by searching for "?=".

As far as I can tell the mailman web interface doesn't have this problem - compare the message at

  http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/2008-November/058997.html

with the "Original message from free mail" attachment and the KMail's one in comment #14.
Comment 21 Cletus 2008-12-10 13:23:11 UTC
Oldrich: I don't think that this is the common denominator of concerned emails. 
If you look at the example email in bug 175965, there is no multi-line international header. 
Comment 22 Oldřich Jedlička 2008-12-10 13:46:06 UTC
(In reply to comment #21)
> Oldrich: I don't think that this is the common denominator of concerned emails. 
> If you look at the example email in bug 175965, there is no multi-line
> international header. 

It looks like there is also problem with header continuation - the second line (the line comment) doesn't start with space/tab character. But for this the original message (before KMail processed it) would be needed for final check.

I've also tried to look at source files, but I'm unable to make head or tail of it.
Comment 23 Torgny Nyblom 2009-09-09 12:08:04 UTC
Marking as a wish since the example emails are malformed.
The wish is to have KMail make an intelligent guess when a header should end and not rely on the RFC (http://tools.ietf.org/html/rfc5322#section-2.2.3).
Comment 24 Oldřich Jedlička 2010-01-05 22:15:16 UTC
Created attachment 39599 [details]
Patch for malformed wrapping of encoded-words (international headers)
Comment 25 Oldřich Jedlička 2010-01-05 22:17:47 UTC
If anybody here has problem with kmail messages, I think I can have a look at your particular parsing problem (I had a look at version 4.3.85 - beta2). But I will need the message (original one) before it got corrupted.
Comment 26 Oldřich Jedlička 2010-01-05 22:25:34 UTC
If I'm right, kmail uses mimelib from kdepim package. This is where my fix goes. There exists also a kmime package that is without a fix, but that one is not used by kmail anyway.

Maybe it is not needed to get the original message, it is quite clear which headers were added by kmail. Just attach your message (save it to mbox format from kmail first) and I will have a look at it.
Comment 27 Oldřich Jedlička 2010-01-05 22:37:09 UTC
I went through my corrupted emails and the same line wrapping problem can be seen in CC and To fields, so it might be fixed by my fix too. But it depends on your particular message.

What I fix is the line starting with "=09", which is apparently a tab character wrongly encoded into a printable ASCII string - that should be a header continuation character.
Comment 28 Oldřich Jedlička 2010-01-05 23:38:04 UTC
Created attachment 39601 [details]
[mimelib-4.3.85] Patch for malformed wrapping of encoded-words (international headers)

Better patch that finds both =09 (tab) and =20 (space) as continuation chracters.
Comment 29 Oldřich Jedlička 2010-01-06 00:03:22 UTC
I've found that the message list still has some wrong subjects, so the patch
needs to be improved.
Comment 30 Oldřich Jedlička 2010-01-06 01:00:22 UTC
Created attachment 39602 [details]
[kmail-4.3.85] Patch for malformed wrapping of encoded-words (international headers)

Here is the patch for kmail for correct subject parsing.
Comment 31 Thomas McGuire 2010-01-08 15:47:04 UTC
> There exists also a kmime package that is without a fix, but that one is
> not used by kmail anyway.

KMail 1 uses mimelib, but KMail 2 and KNode use KMime. We're currently working on KMail 2 in the akonadi-ports branch, which will shortly be merged back to trunk.

It would be good to have this fix, including a unit test, for  KMime as well.

Can you please post your patch to reviewboard.kde.org, usually more people check patches there, since a bugs.kde.org notification mail can drown in the noise. See http://techbase.kde.org/Contribute/Send_Patches.

Thanks for the patch :)
Comment 32 Oldřich Jedlička 2010-01-09 17:34:29 UTC
(In reply to comment #31)
> > There exists also a kmime package that is without a fix, but that one is
> > not used by kmail anyway.
> 
> KMail 1 uses mimelib, but KMail 2 and KNode use KMime. We're currently working
> on KMail 2 in the akonadi-ports branch, which will shortly be merged back to
> trunk.
> 
> It would be good to have this fix, including a unit test, for  KMime as well.

I will have a look at that during next week. I've already checked-out branches/work/akonadi-ports for this purpose. I've updated KMime first, but kmail didn't use it, so I was little bit upset afterwards :-) The KMime patch needs to be split into two pieces anyway, I will do my best to do this with Subversion (thanks God KDE is moving to Git).

> Can you please post your patch to reviewboard.kde.org, usually more people
> check patches there, since a bugs.kde.org notification mail can drown in the
> noise. See http://techbase.kde.org/Contribute/Send_Patches.

Ok, I will send it (next week).

> Thanks for the patch :)

Thanks for your thanks :-)
Comment 33 Oldřich Jedlička 2010-01-12 20:48:39 UTC
I've sent the patch for kmime on work/akonadi-ports, let's see what is the opinion there.
Comment 34 Oldřich Jedlička 2010-01-12 22:26:46 UTC
Does it make sense to send the patches for 4.4 too?
Comment 35 Thomas McGuire 2010-01-16 13:51:42 UTC
I have commented on the review request for the KMime patch.

About the mimelib + KMail 1 patches: How sure are you those don't introduce additional regressions, like crashes because the array bounds are not checked? If you think the patches are stable enough, please post them to reviewboard as well, so we can commit them to the KDE 4.4 branch.
Comment 36 Oldřich Jedlička 2010-01-17 22:17:49 UTC
(In reply to comment #35)
> I have commented on the review request for the KMime patch.

I've updated the patch.

> About the mimelib + KMail 1 patches: How sure are you those don't introduce
> additional regressions, like crashes because the array bounds are not checked?
> If you think the patches are stable enough, please post them to reviewboard as
> well, so we can commit them to the KDE 4.4 branch.

The patches should not cause any regression AFAICT, I can imagine only some assert coming when comparing value at the end of the string. I will have a look at it during the week and prepare the patch for 4.4 branch.
Comment 37 Thomas McGuire 2010-01-17 23:31:32 UTC
SVN commit 1076273 by tmcguire:

Handle malformed multi-line headers in a better way, some clients seem to send those.

Patch by Oldrich Jedlicka <oldium.pro at seznam dot cz>, thank you!

reviewboard.kde.org/r/2561
BUG: 86302


 M  +2 -32     kmime_header_parsing.cpp  
 M  +71 -24    kmime_util.cpp  
 M  +8 -0      kmime_util_p.h  
 M  +20 -1     tests/kmime_util_test.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1076273
Comment 38 Oldřich Jedlička 2010-01-18 21:16:42 UTC
I've added review request for patches for kmail-4.4. My opinion is that there should be no regression and no additional assert().
Comment 39 Thomas McGuire 2010-01-20 14:03:14 UTC
SVN commit 1077597 by tmcguire:

Handle malformed multiline headers correctly in this branch as well.

http://reviewboard.kde.org/r/2644/
CCBUG: 86302


 M  +8 -0      kmail/kmfoldermaildir.cpp  
 M  +8 -0      kmail/kmfoldermbox.cpp  
 M  +9 -1      mimelib/entity.cpp  
 M  +31 -4     mimelib/headers.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1077597