Bug 259151

Taking the liberty of setting this to Major, as it makes Akonadi very hard to use for people with servers without UIDPLUS (Exchange 2005, 2007, 2010).

Fixed in r1204522.

Sorry, this still happens if more than one email is moved.

INBOX = 14 emails
INBOX/processed = 1 email

Everything is synced up, kmail2 is idle.

Last SELECT to INBOX/processed:
C: A000072 SELECT "INBOX/processed"
S: * 1 EXISTS
S: * 0 RECENT
S: * FLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )
S: * OK Permanent flags [ PERMANENTFLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )  ]
S: * OK UIDVALIDITY value [ UIDVALIDITY 1074  ]
S: * OK The next unique identifier value [ UIDNEXT 8555  ]
S: A000072 OK SELECT completed. [ READ-WRITE  ]

Move 2 emails from INBOX to INBOX/processed:
C: A000522 UID COPY 30082 "INBOX/processed"
S: A000522 OK COPY completed.
C: A000523 UID STORE 30082 +FLAGS (\Deleted)
S: * 12 FETCH ( FLAGS (\Seen \Deleted) )
S: A000523 OK STORE completed.
C: A000524 SELECT "INBOX/processed"
S: * 2 EXISTS
S: * 1 RECENT
S: * FLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )
S: * OK Permanent flags [ PERMANENTFLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )  ]
S: * OK UIDVALIDITY value [ UIDVALIDITY 1074  ]
S: * OK The next unique identifier value [ UIDNEXT 8556  ]
S: A000524 OK SELECT completed. [ READ-WRITE  ]
C: A000525 UID SEARCH HEADER Message-ID <201012081039.oB8AdJgL005803@mgw-sa01.nokia.com>
S: * SEARCH 8555
S: A000525 OK SEARCH completed.
C: A000526 SELECT "INBOX"
S: * 14 EXISTS
S: * 0 RECENT
S: * FLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )
S: * OK Permanent flags [ PERMANENTFLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )  ]
S: * OK Is the first unseen message [ UNSEEN 14  ]
S: * OK UIDVALIDITY value [ UIDVALIDITY 553  ]
S: * OK The next unique identifier value [ UIDNEXT 30085  ]
S: A000526 OK SELECT completed. [ READ-WRITE  ]
C: A000527 UID COPY 30083 "INBOX/processed"
S: A000527 OK COPY completed.
C: A000528 UID STORE 30083 +FLAGS (\Deleted)
S: * 13 FETCH ( FLAGS (\Seen \Deleted) )
S: A000528 OK STORE completed.
C: A000529 SELECT "INBOX/processed"
S: * 3 EXISTS
S: * 1 RECENT
S: * FLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )
S: * OK Permanent flags [ PERMANENTFLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )  ]
S: * OK UIDVALIDITY value [ UIDVALIDITY 1074  ]
S: * OK The next unique identifier value [ UIDNEXT 8557  ]
S: A000529 OK SELECT completed. [ READ-WRITE  ]
C: A000530 UID SEARCH HEADER Message-ID <201012081030.oB8AUQIZ026905@mgw-sa02.nokia.com>
S: * SEARCH 8553 8556
S: A000530 OK SEARCH completed.

Now synchronise INBOX/processed.
C: A000005 SELECT "INBOX/processed"
S: * 3 EXISTS
S: * 0 RECENT
S: * FLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )
S: * OK Permanent flags [ PERMANENTFLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )  ]
S: * OK UIDVALIDITY value [ UIDVALIDITY 1074  ]
S: * OK The next unique identifier value [ UIDNEXT 8557  ]
S: A000005 OK SELECT completed. [ READ-WRITE  ]

The agent prints:
akonadi_imap_resource_0(30643)/kdepimlibs (kimap) RetrieveItemsTask::triggerExpunge: "INBOX/processed"
akonadi_imap_resource_0(30643)/kdepimlibs (kimap) RetrieveItemsTask::onFinalSelectDone: UIDNEXT check failed (old UIDNEXT = 8556 but current UIDNEXT = 8557 ), refetching mailbox
akonadi_imap_resource_0(30643)/kdepimlibs (kimap) RetrieveCollectionMetadataTask::doStart: "/processed"

C: A000006 FETCH 1:3 (RFC822.SIZE INTERNALDATE BODY.PEEK[] FLAGS UID)
[...]

Note: one message was a copy of the other, that's why they have the same message-id, but the problem happens with two different messages.

Actually, come to think of it, I think the entire code is bogus. There's never a good reason to redownload the entire mailbox as long as the UIDVALIDITY remains the same.

The RFC says (2.3.1.1):
   [...] Unique identifiers
   are assigned in a strictly ascending fashion in the mailbox; [...]

   The unique identifier of a message MUST NOT change during the
   session, and SHOULD NOT change between sessions.  Any change of
   unique identifiers between sessions MUST be detectable using the
   UIDVALIDITY mechanism discussed below.
[...]
         1) Unique identifiers MUST be strictly ascending in the
            mailbox at all times. [...]

It looks to me like the code is checking the wrong parameter. The only valid reason for redownloading everything is a change in UIDVALIDITY. As long as the UIDVALIDITY remains the same, the client should simply download any UIDs it did not know about.

adding to showstoppers

Some observations from researching this:
- the UIDNEXT check will fail if the message-id is non-unique and the server does not support UIDPLUS (see MoveItemTask::onSearchDone()).
- the UIDNEXT check is necessary to catch the case of an equal amount of mails removed and added while we were not looking, the simple check for the message count is not enough for that (the complete download is obviously a bug though)

See https://git.reviewboard.kde.org/r/101142/ for a possible fix.

Git commit 6adf3ac1bf26662d9e81fa43aced445ecb659d4f by Volker Krause.
Committed on 21/04/2011 at 11:50.
Pushed by vkrause into branch 'master'.

Try to recover from UIDNEXT mismatch.

In case the IMAP resource ends up with an mismatch in current and last seen
UIDNEXT values, it tries to recover from that with a full re-download. With
online IMAP that's not too bad, but for the offline case this can be a huge
problem. This patch tries to do the recovery in a less drastic way, given
some conditions are met.

The assumption is that this case can happen because of an equal amount of
emails being added and deleted while we were not looking. Then the message
count check passes (we still have the same amount locally and remotely),
but the UIDNEXT check fails. Given UIDs are strictly ascending, the
maximum amount of changed messages is (newUidnext - oldUidnext). So,
instead of downloading everything, we now only download that many messages,
since download is done by sequence number, we are guaranteed to get the
newest ones and thus we cover all new messages. Deletion of the old ones is
done by the subsequent flag fetch step, just as in the normal case.

Thiago's problem shown in bug 259151 is slightly different, there we do the
same modification locally and remotely but fail to obtain the new UID due
to the server not supporting UIDPLUS and the message-id being non-unique.
However, this should work with the above approach as well.

BUG: 259151
REVIEW: 101142

M  +11   -1    resources/imap/retrieveitemstask.cpp     
M  +35   -0    resources/imap/tests/testretrieveitemstask.cpp     

http://commits.kde.org/kdepim-runtime/6adf3ac1bf26662d9e81fa43aced445ecb659d4f