Bug 320828 - UW-IMAP: broken ESEARCH: mailbox exception: error when synchronizing all messages: uid search response
Summary: UW-IMAP: broken ESEARCH: mailbox exception: error when synchronizing all mess...
Alias: None
Product: trojita
Classification: Applications
Component: IMAP (show other bugs)
Version: git
Platform: openSUSE Linux
: NOR normal (vote)
Target Milestone: ---
Assignee: Trojita default assignee
Depends on:
Reported: 2013-06-06 19:08 UTC by Nicolás Sirolli
Modified: 2014-03-10 13:02 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In:

IMAP protocol log (11.11 KB, text/plain)
2013-06-06 19:50 UTC, Nicolás Sirolli

Note You need to log in before you can comment on or make changes to this bug.
Description Nicolás Sirolli 2013-06-06 19:08:19 UTC
I get the following message when I try to open my Inbox:

MailboxException: Error when synchronizing all messages: server said that there are 586 messages, but UID (E)SEARCH ALL response contains 1672 entries.

Nevertheless, I am able to open other folders, which happen to contain less messages.

Reproducible: Always

Steps to Reproduce:
1. Launch Trojita
2. Open my Inbox
Actual Results:  
I get the message mentioned above.

Expected Results:  
It should display the messages in my Inbox.
Comment 1 Jan Kundrát 2013-06-06 19:38:14 UTC
Nicolas, could you please attach the IMAP protocol log and reopen this bugreport? The log shall pop automatically, if not, it's in IMAP -> Debugging -> Show IMAP Protocol Log. What IMAP server implementation is this?
Comment 2 Nicolás Sirolli 2013-06-06 19:50:00 UTC
Created attachment 80353 [details]
IMAP protocol log
Comment 3 Jan Kundrát 2013-06-06 20:00:59 UTC
Ah, this is bad. The log comes from an ancient version of the UW-IMAP, a server which is not maintained anymore. The server is sending invalid data in violation of the ESEARCH extension, unfortunately. We *might* add a kludge into Trojita which will try to request an old-fashioned SEARCH instead (and blacklist ESEARCH at the same time), but that will take some time (UW-IMAPD is, thankfully, not that popular anymore). In the meanwhile, you can add ESEARCH into IMAP -> Settings -> IMAP -> Blacklisted Extensions.

For a bit more context, this is how I described this problem in my thesis:

At the time of the ESEARCH adoption, the imap-protocol mailing list witnessed
a disagreement on how exactly the sequence-set shall be interpreted. Mark Crispin, the author of the original IMAP protocol (but not of the ESEARCH extension) implemented ESEARCH in a different manner. He chose to take an advantage of the RFC3501-style definition of UID sequences where the RFC mandates that servers shall treat non-existent UIDs given in sequence sets as if they weren’t referenced from the command at all. For example, if the mailbox contained just UIDs 3, 5 and 10, a client using the 3:10 construct has to be interpreted as if it requested sequence-set 3,5,10. Doing so present certain optimization opportunities to the servers, for example when the client already knows the UID mapping and performs a server-side search for messages matching certain criteria and the result set accurately matches an adjacent range
of messages, the server could take advantage of this adjacency a return a sequence-set in the form of 10:150, even though the mailbox contains only a few UIDs from this range [26]. Furthermore, his another point is that the clients already have two other ways of obtaining the UID mapping, either through the UID SEARCH ALL command or via an explicit UID FETCH 1:*. Needless to say, such a reasoning fails to take into account potential bandwidth savings which can be rather substantial on “reasonable” mailboxes. In the end, the authors of the RFC 4731 disagreed with Crispin [27] [28].

[26] http://www.ietf.org/mail-archive/web/imapext/current/msg00477.html
[27] http://www.ietf.org/mail-archive/web/imapext/current/msg00482.html
[28] http://www.ietf.org/mail-archive/web/imapext/current/msg00472.html
Comment 4 Nicolás Sirolli 2013-06-06 20:21:19 UTC
Thanks! Adding ESEARCH to the blacklisted extensions solved the problem.
Comment 5 Jan Kundrát 2013-06-13 06:50:53 UTC
Git commit 0dba37f7367ae20d3ed3f87b2a7a6f3e063a00b0 by Jan Kundrát.
Committed on 09/06/2013 at 14:42.
Pushed by jkt into branch 'master'.

IMAP: do not doubly-encode backslashes when sending mailbox names

I've noticed this in an unrelated bugreport (thanks to Nicolas Sirolli
<nmsirolli@gmail.com> in bug 320828).

The encodeImapFolderName shall only convert the Unicode data into a proper
modified-utf-7 thing, not escape the backslashes. This escaping is done by the
Parser when queueing commands. Looks like this bug must have been around for
ages, literally.

M  +1    -1    src/Imap/Parser/3rdparty/rfccodecs.cpp
M  +20   -0    tests/tests/test_Imap_Tasks_ListChildMailboxes/test_Imap_Tasks_ListChildMailboxes.cpp
M  +4    -1    tests/tests/test_Imap_Tasks_ListChildMailboxes/test_Imap_Tasks_ListChildMailboxes.h

Comment 6 Jan Kundrát 2014-03-10 13:02:55 UTC
Looks like noone has time for adding the heuristic of "broken ESEARCH response -> retry with just SEARCH". Given that the server in question is not maintained anymore, I'm closing this bug now. Patches are still welcome, it's just that I don't think any of the existing developers is willing to write them.