Bug 417206 - KMail wrong date in IMAP APPEND command
Summary: KMail wrong date in IMAP APPEND command
Status: RESOLVED FIXED
Alias: None
Product: Akonadi
Classification: Frameworks and Libraries
Component: IMAP resource (show other bugs)
Version: 5.14.3
Platform: Gentoo Packages Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-02-05 20:09 UTC by 0xc0ffee
Modified: 2021-01-05 16:18 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
patch that makes sure internal date is used in changeitem-task (654 bytes, patch)
2020-09-13 09:49 UTC, Erik Quaeghebeur
Details

Note You need to log in before you can comment on or make changes to this bug.
Description 0xc0ffee 2020-02-05 20:09:26 UTC
SUMMARY

When copying an email from one account's folder (e.g. Inbox) to another account's folder (e.g. Inbox again) the wrong email date is supplied in the IMAP APPEND command. At present, KMail provides the current date/time string. This is contrary to RFC 3501 section 6.3.11 which states:

    If a date-time is specified, the internal date SHOULD be set in
    the resulting message; otherwise, the internal date of the
    resulting message is set to the current date and time by default

suggesting that, if possible, the message's internal date/time string should take precedence.

STEPS TO REPRODUCE

1. Enable logging: "$ export KIMAP_LOGFILE=$HOME/kimap/kimap.log"
2. Copy (e.g. old) email from a folder in account A to a folder in account B
3. Observe logged IMAP communication, notably the APPEND command as logged


OBSERVED RESULT

For an email with internal date (i.e. has "Date" header) of "01-Jan-2010 12:34:56 +0000" the APPEND command, as logged, looks something like:

    C: A000004 APPEND "INBOX" (\Seen) "05-Feb-2020 10:20:30 +0000" {104666}

where the timestamp of the IMAP command is the time that the command was submitted to the mail server, followed by the email source.


EXPECTED RESULT

The correct APPEND command under said conditions should instead look something like:

    C: A000004 APPEND "INBOX" (\Seen) "01-Jan-2010 12:34:56 +0000" {104666}

since "01-Jan-2010 12:34:56 +0000" is the internal date available in the message headers.


SOFTWARE/OS VERSIONS

KMail Version: 5.12.3
Linux/KDE Plasma: Gentoo Linux
KDE Plasma Version: 5.17.4
KDE Frameworks Version:  5.64.0
Qt Version: 5.13.2

also present in:

KMail Version: 5.12.3
Linux/KDE Plasma: Debian Testing
KDE Plasma Version: 5.14.5
KDE Frameworks Version:  5.62.0
Qt Version: 5.12.5


ADDITIONAL INFORMATION

The above issue appears to affect multi-part messages such as those containing HTML as well as PGP encrypted messages. As far as I could tell, it does not seem to affect plain text messages, though it may need further verification.


WHY IS THIS A PROBLEM

Most mail clients (including KMail) will show the timestamp of the internal Date header. However, some web-based clients (such as GMail, as a popular example) store and index messages based on the timestamp provided as per the IMAP APPEND command.

This can make migrated email messages show up with incorrect timestamps an in the wrong order which can also mess up with conversation view grouping. Another problem is if KMail's message filtering is used to automatically PGP encrypt incoming messages which are IMAP APPEND-ed post-encryption to the mail server, thereby resulting in the same problem as with migrated emails.

In contrast, other mail clients (e.g. Thunderbird) will use the internal timestamp in every case, as per the RFC.
Comment 1 Erik Quaeghebeur 2020-08-24 12:46:37 UTC
I can confirm this. I hit this issue when filtering using the "Pipe through" action. Upon APPEND, the INTERNAL DATE of the filtered message should be set to the one of the original one.
Comment 2 0xc0ffee 2020-09-06 21:18:15 UTC
@Erik,

Thanks for the update to the report. I was beginning to lose hope that this would ever be picked up and am glad somebody else has also noticed this behaviour.

A quick update, as of KMail v5.14.3 (plasma v5.18.5, apps v20.04.3, frameworks v5.70.0) the issue is still present.
Comment 3 Erik Quaeghebeur 2020-09-08 08:48:49 UTC
(In reply to kernel_panic from comment #2)
> […] I was beginning to lose hope that this
> would ever be picked up […]
I think that if we want this to be picked up, it would help if we can at least point out in the code where the issue lies. This is filed in the product kmail2, but it could be an akonadi or kimap issue.

I think kimap can be excluded, because it clearly provides functionality for setting the INTERNAL DATE:

https://invent.kde.org/pim/kimap/-/blob/release/20.04/src/appendjob.cpp

(This was also apparent from your initial report.)

AFAICT from the Akonadi database structure, it only has a concept of
  * last access time
  * datetime, which seems to correspond to last *modification* time (I do not know which modifications count: flags, tags, …?). For sure changing the message's MIME structure counts, for the issue I encounter.
So if I'm correct it has no concept that maps to INTERNAL DATE. That means that upon append to an IMAP store from an IMAP store, the code that appends must explicitly include a call to the original IMAP store to get the INTERNAL DATE and use it upon append.

Further investigation is necessary.
Comment 4 0xc0ffee 2020-09-08 11:31:32 UTC
I remember I was looking at the same "appendjob.cpp" file when I first reported the issue. At the time I was hoping to be able to pinpoint the issue and even make a small patch that fixes it. Unfortunately, I'm not at all familiar with KDE's codebase and there seem to be a number of different components at play.

But I think you're absolutely right. From what I understand in the way that KDE works, KMail connects to Akonadi which actually does the handling of all PIM-related tasks. So even though the issue manifests itself in KMail, it may in fact be related to the way that Akonadi handles IMAP resources in which case this report ought to be filed under Akonadi.

In either case, if Akonadi has no sense of what the internal date is, this doesn't explain why single part plaintext emails are not affected - at least not in the scenarios I provided. Unless, as you suggest, it's fetching the date from IMAP on-the-fly when the operation is performed.

I will have another look at the source code when I get a chance.
Comment 5 Erik Quaeghebeur 2020-09-08 12:06:00 UTC
(In reply to kernel_panic from comment #4)
> […], this doesn't explain why single part plaintext emails are not affected
> - at least not in the scenarios I provided. […]
TBH, I cannot reproduce your issue. When I move messages between IMAP accounts on different servers, they keep their INTERNAL DATE. I've tried with related(plain,image) and alternative(plain,related(html,image,image,image)) messages. So I'm afraid we may be hitting different bugs both related to INTERNALDATE handling.

The fact that copying between different servers works (in some cases) actually makes me hopeful, as that means that it is possible without having to add a database column to Akonadi.
Comment 6 Erik Quaeghebeur 2020-09-08 12:52:15 UTC
I think the line where the INTERNAL DATE is set is

https://invent.kde.org/pim/kdepim-runtime/-/blob/master/resources/imap/additemtask.cpp#L56

In case you agree, we should modify the product/component of this bug.

So next, to find out what determines msg->date()->dateTime(). My guess is it is Akonadi's PimItemTable.datetime, which is the last modified date, which is (in general) not the internal date of the message on the (originating) server. That would be against the RFC and therefore a clear bug.
Comment 7 Erik Quaeghebeur 2020-09-08 13:02:49 UTC
(In reply to Erik Quaeghebeur from comment #6)
> I think the line where the INTERNAL DATE is set is […]
I shouldn't be so quick to point to this instance only.

Similar (for when ‘replacing’ a message):

https://invent.kde.org/pim/kdepim-runtime/-/blob/master/resources/imap/replacemessagejob.cpp#L36

Non-similar as setInternalDate is *not* called here (for when ‘changing an item’): 

https://invent.kde.org/pim/kdepim-runtime/-/blob/master/resources/imap/changeitemtask.cpp#L53

In this last case, the INTERNAL DATE is probably set by the server to the current date. It may be that this is where my issue originates.
Comment 8 0xc0ffee 2020-09-08 13:56:01 UTC
(In reply to Erik Quaeghebeur from comment #5)
> TBH, I cannot reproduce your issue. When I move messages between IMAP
> accounts on different servers, they keep their INTERNAL DATE. I've tried
> with related(plain,image) and
> alternative(plain,related(html,image,image,image)) messages.

I think whether the issue is de-facto observed is also dependent on the implementation of the server and its RFC compliance.

For example, GMail uses the timestamp provided in the APPEND command to store, index, and display messages in its web interface. This timestamp may be different to the one provided in the message header. However, it appears that this timestamp is stored separately (presumably in some indexing database) since the original message's Date header is left intact and showing the correct timestamp. Consequently, the messages with their correct timestamps in a mail client (e.g. KMail or Thunderbird) which use the Date headers for indexing, but are completely messed up in GMail's web interface.

On my private mail server, which uses Postfix and a simple Mbox format for storage and indexing, the message's internal Date header is always used. I don't recall explicitly configuring this but it appears that it is interpreting the "SHOULD" clauses in the RFC slightly differently.

Going back to the RFC section quoted above, I think I may have misinterpreted what it actually means. Notably, it only concerns mail /servers/ and not mail clients. It seems that if a timestamp has been specified in the APPEND command the server SHOULD use that timestamp and it SHOULD supersede any internal message timestamp (e.g. Date header). Moreover, it SHOULD overwrite said internal timestamp. Likewise, if a timestamp is not provided, the server SHOULD use the current date/time and set the message's internal timestamp to that one.

So, in that regard, it would appear GMail's implementation is at least partially conformant wrt its web interface and how it indexes the messages there.

But I don't see any section of the RFC that concerns clients issuing the APPEND command. In either case, my view is that it would make sense for clients, by default, to use the message's internal timestamp - if present - in all instances so as to avoid the above issue. From my mini-analysis, Thunderbird does that. And KMail also does this in most cases. So it seems it's only the small set of cases where this isn't done that are of concern.

I'll see if I can narrow down when else this might occur. So far, the easiest way to consistently reproduce the issue is with filters (e.g. PGP encrypt and/or decrypt) which pushes an encrypted copy of the message to the IMAP server before deleting the old copy from the IMAP server.
Comment 9 0xc0ffee 2020-09-08 14:08:11 UTC
(In reply to Erik Quaeghebeur from comment #7)
> Non-similar as setInternalDate is *not* called here (for when ‘changing an
> item’): 
> 
> https://invent.kde.org/pim/kdepim-runtime/-/blob/master/resources/imap/
> changeitemtask.cpp#L53
> 
> In this last case, the INTERNAL DATE is probably set by the server to the
> current date. It may be that this is where my issue originates.

I think you're right in this case. I'm hesitant to overgeneralise, but it appears it's a selection of individual "Task" implementations that do not include the internal timestamp and the server will use the current one when storing the message. Which makes sense, as the KIMAP clearly does have support for supplying the correct timestamp.

(In reply to Erik Quaeghebeur from comment #6)
> In case you agree, we should modify the product/component of this bug.
> 
Agreed. I think it's the underlying PIM runtime task implementations which KMmail ultimately uses that may need to be corrected. We should probably update this to "kdepim" or "kdepimlibs". I suspect the latter would be more appropriate, what do you think?
Comment 10 Erik Quaeghebeur 2020-09-08 14:11:17 UTC
(In reply to kernel_panic from comment #8)
> For example, GMail uses the timestamp provided in the APPEND command to
> store, index, and display messages in its web interface. This timestamp may
> be different to the one provided in the message header. However, it appears
> that this timestamp is stored separately (presumably in some indexing
> database) since the original message's Date header is left intact and
> showing the correct timestamp. […]

I think there is a misunderstanding. The INTERNAl DATE is *not* the same as the Date header value and it should not be according to the RFC. (Usually, it will be close, but can differ significantly[*] if message delivery is delayed.) Please read https://tools.ietf.org/html/rfc3501#section-2.3.3 for a description. 

GMail does follow the RFC here AFAICT and uses the ‘REFS’ threading option. (As does, e.g., Fastmail.)

[*] So that order of mails in a thread differs based on INTERNAL DATE or Date header.
Comment 11 Erik Quaeghebeur 2020-09-08 14:24:47 UTC
(In reply to kernel_panic from comment #9)
> […] We should probably
> update this to "kdepim" or "kdepimlibs". I suspect the latter would be more
> appropriate, what do you think?
Akonadi contains the IMAP resource, so that.

We may need to go back to kmail2 in case it is kmail that determines the value of msg->date()->datetime().
Comment 12 0xc0ffee 2020-09-08 14:32:05 UTC
(In reply to Erik Quaeghebeur from comment #10)
> (In reply to kernel_panic from comment #8)
> > For example, GMail uses the timestamp provided in the APPEND command to
> > store, index, and display messages in its web interface. This timestamp may
> > be different to the one provided in the message header. However, it appears
> > that this timestamp is stored separately (presumably in some indexing
> > database) since the original message's Date header is left intact and
> > showing the correct timestamp. […]
> 
> I think there is a misunderstanding. The INTERNAl DATE is *not* the same as
> the Date header value and it should not be according to the RFC. (Usually,
> it will be close, but can differ significantly[*] if message delivery is
> delayed.) Please read https://tools.ietf.org/html/rfc3501#section-2.3.3 for
> a description. 
> 
> GMail does follow the RFC here AFAICT and uses the ‘REFS’ threading option.
> (As does, e.g., Fastmail.)
> 
> [*] So that order of mails in a thread differs based on INTERNAL DATE or
> Date header.

I was aware of the difference, though admittedly I may have been using "internal" a bit more loosely then I should have - it has been a while since I last properly referred to the RFC. Thanks for pointing this out. In any case, as I mentioned, the RFC does not seem to specify how clients ought to pick the timestamp they supply to the APPEND command - if they wish to do so.

I suspect, from an IMAP client perspective, the "internal" date can only be inferred from the "Received" header for the final destination. Alternatively, the "Date" header ought to suffice.
Comment 13 Erik Quaeghebeur 2020-09-13 09:49:50 UTC
Created attachment 131589 [details]
patch that makes sure internal date is used in changeitem-task

This patch fixed my issue (limited testing, however).

For (current stable) Gentoo users: place it beneath /etc/portage/kde-apps/kdepim-runtime-20.04.3/ and re-emerge kdepim-runtime. Restart akonadi and check whether or not this solves your issue. (It should apply as well to earlier versions and current 20.08.z versions. I'm hoping to get it into a later 20.08.z version.)
Comment 14 Erik Quaeghebeur 2020-09-13 10:18:18 UTC
(In reply to Erik Quaeghebeur from comment #13)
> For (current stable) Gentoo users: place it beneath
> /etc/portage/kde-apps/kdepim-runtime-20.04.3/ and re-emerge kdepim-runtime.

Typo alert. It should be

/etc/portage/patches/kde-apps/kdepim-runtime-20.04.3/

Note the ‘patches’.
Comment 15 Erik Quaeghebeur 2020-09-16 14:57:58 UTC
(In reply to Erik Quaeghebeur from comment #13)
> I'm hoping to get it into a later 20.08.z version.

It should appear in the 20.08 branch in 20.08.2: https://invent.kde.org/pim/kdepim-runtime/-/commit/40797e87f8a772ee0bb5df301b41824b1b5007b3

I've asked the maintainers to add it to master as well, so that it'll be present in 20.12 and beyond.

@kernel_panic: Do you have the opportunity to test the patch? If it solves your issue as well, we can close this bug.
Comment 16 Bug Janitor Service 2020-10-01 04:33:13 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 17 Bug Janitor Service 2020-10-16 04:33:16 UTC
This bug has been in NEEDSINFO status with no change for at least
30 days. The bug is now closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

Thank you for helping us make KDE software even better for everyone!
Comment 18 0xc0ffee 2021-01-05 16:18:22 UTC
Sorry for the [very] late update. Following some tests, I can confirm that your patch fixes both the issue that you raised as well as the original issue that I had reported.

I have tested it both against 20.04.x (with the patch) as well as the upstream 20.08.x release where the patch is already included.

Thanks for the insight and the patch!