Bug 368003

Summary: CATENATE might produce invalid MIME messages
Product: [Applications] trojita Reporter: Jan Kundrát <jkt>
Component: Message ComposerAssignee: Trojita default assignee <trojita-bugs>
Status: RESOLVED FIXED    
Severity: normal    
Priority: NOR    
Version First Reported In: git   
Target Milestone: ---   
Platform: Other   
OS: Linux   
Latest Commit: http://commits.kde.org/trojita/d9a728d49cef4457afdf0696942a848e82951815 Version Fixed/Implemented In:
Sentry Crash Report:

Description Jan Kundrát 2016-08-30 10:26:34 UTC 
When using drag-and-drop to attach a part stored on an IMAP server, the composer migh produce malformed MIME messages if CATENATE is available and used.

ImapPartAttachmentItem::suggestedCTE() makes this easy to see because it hardcodes "Content-Transfer-Encoding: base64" for anything but message/rfc822. So when attaching something like message/delivery-status or essentially anything which is stored without any CTE on the IMAP server, Trojita's composer writes out a CTE header which identifies the content as base64-encoded. When it comes to adding the actual data, Trojita uses an IMAP URL mechanism which points to a raw MIME part as-is, which in this case means a part stored with no CTE which in turns is 7bit.

One approach at fixing this is to use the CTE header from the IMAP server, and to hope that the original message was not malformed. This, however, might cause trouble if we are about to send a message which is, say, in CTE: 8bit over an SMTP server which does not support the CTE: 8bit. In that case, we will have to transcode that part. If we are transcoding, there's a question on how a copy of message which we are storing to the sent mailbox should look like -- either it can use the most-efficient format, the CTE: 8bit, saving both bandwidth and disk space, or it should match the outgoing message exactly, bite-wise, which means that it must not use CATENATE in that case. Stuff gets messy because right now, we prepare the full MIME structure beforehand, before even connecting to the SMTP server which makes a lot of sense due to BURL. Maybe we should detect that we cannot submit our message in this way, remove the previously stored copy, upload a new one with no CTE: 8bit et al, and then use this new 7bit version as an outgoing message in that case...

Or TL;DR, just disable CATENATE for now?
Comment 1 Thomas Lübking 2016-08-30 22:49:56 UTC 
> should match the outgoing message exactly, bite-wise
+100

> detect that we cannot submit our message in this way, remove the previously stored copy, upload a new one with no CTE: 8bit et al
Sounds not too efficient?

> disable CATENATE for now?
If either a fix is located in Farfaraway or fixing it "right" would add more overhead than CATENATE removes, that would be a viable option.
Comment 2 Jan Kundrát 2016-09-05 15:53:39 UTC 
Git commit 3501e8190d48735c2076f6a1c33607c100e4fa0c by Jan Kundrát.
Committed on 01/09/2016 at 13:43.
Pushed by gerrit into branch 'master'.

tests: Attaching IMAP parts with varying CTE

Just some refactoring to make the test more generic. In future, we'll
actually add more data (and fix bug 368003, too).

Change-Id: I1b7e4ff42c2b43cd8a8131dbb1d5c6753ac0a141

M  +48   -10   tests/Composer/test_Composer_Submission.cpp
M  +2    -1    tests/Composer/test_Composer_Submission.h

http://commits.kde.org/trojita/3501e8190d48735c2076f6a1c33607c100e4fa0c
Comment 3 Jan Kundrát 2016-09-05 15:54:38 UTC 
Git commit d9a728d49cef4457afdf0696942a848e82951815 by Jan Kundrát.
Committed on 01/09/2016 at 18:48.
Pushed by gerrit into branch 'master'.

Use correct CTE when attaching IMAP parts

Because we are making a heavy use of the CATENATE extension which uses
the raw, undecoded data from the IMAP server without any transformation,
let's make sure that the attached part inherits its
Content-Transfer-Encoding from the original part. This ensures that the
attachment is added as-is, with no additional transformations and is
still correctly marked.

A potential problem is that when attaching a part which is
malformed/invalid or if it contains data which are not permitted by a
relying MTA, we might hit problems. However, in that case it would be a
garbage-in, garbage-out problem. There's a hypothetical scenario where
an IMAP message contains a binary, non-encoded part, which this patch
happily ignores. However, it isn't a regression but rather an
improvement -- at least such part is properly marked with a CTE that
reflects its actual content.

This patch still doesn't enforce a 100% byte-by-byte match of the
outgoing message (as submitted over SMTP) and the copy stored in the
Sent folder. If CATENATE is available (it is with Dovecot for example)
and BURL is not available (BURL is still a bit exotic today), and if
attaching something which is already available on the IMAP server, there
might be subtle differences such as the wrapping of the base64 content,
etc. That's the price one pays when offloading stuff to the IMAP server,
and I do not see any possibility of avoiding it (except maybe tying BURL
and CATENATE together and only ever using both or none of them...).

Change-Id: I53a2a186f308a9acab3b1d5f4e3e821e29bbb3ce

M  +30   -7    src/Composer/ComposerAttachments.cpp
M  +1    -0    src/Composer/ComposerAttachments.h
M  +9    -1    src/Composer/MessageComposer.cpp
M  +39   -3    tests/Composer/test_Composer_Submission.cpp

http://commits.kde.org/trojita/d9a728d49cef4457afdf0696942a848e82951815