| Summary: | [Office365][Imap][akonadi] failure to sync/pull all messages from an Inbox with 10GB of e-mails. (infinite loop) | ||
|---|---|---|---|
| Product: | [Frameworks and Libraries] Akonadi | Reporter: | Boris Shtrasman <boris.sh.1983+kde.bugzilla> |
| Component: | IMAP resource | Assignee: | kdepim bugs <pim-bugs-null> |
| Status: | CONFIRMED --- | ||
| Severity: | normal | CC: | carl, dvratil |
| Priority: | NOR | ||
| Version First Reported In: | 6.3.3 | ||
| Target Milestone: | --- | ||
| Platform: | Fedora RPMs | ||
| OS: | Linux | ||
| Latest Commit: | Version Fixed/Implemented In: | ||
| Sentry Crash Report: | |||
|
Description
Boris Shtrasman
2025-04-08 12:29:34 UTC
It's not really a loop. In the past we had problems with GMail having really broken support for RFC4551/CONDSTORE extension (https://datatracker.ietf.org/doc/html/rfc4551), which allows us to ask the email server to give us all messages that have changed since the last time we checked. Without CONDSTORE, we are able to easily find new messages, but to check a flag change of an existing message (e.g. if you marked some older message as read/unread), we need to list *all* messages in the mailbox and compare their flags with what we have cached in Akonadi. If a massive mailbox, this takes ages.... There are two steps that we should take: 1) Check whether GMail have fixed their CONDSTORE support and re-enable it. This would fix the problem instantly for you. 2) Get rid of the SEARCH - it was an optimization to ensure that we always get a batch of 2000 valid UIDs. Unfortunately on most IMAP servers (incl. Gmail), SEARCH takes ages. We can still do batching for the search, if we do a regular FETCH instead of UID FETCH, and have UIDs included in the response. We can possibly even increase the batch size if we process things more efficiently. That should be much much much faster. Even if CONDSTORE is still broken on Gmail, this should speed up the sync massively. (In reply to Daniel Vrátil from comment #1) > It's not really a loop. In the past we had problems with GMail having > really broken support for RFC4551/CONDSTORE extension > (https://datatracker.ietf.org/doc/html/rfc4551), which allows us to ask the > email server to give us all messages that have changed since the last time > we checked. Without CONDSTORE, we are able to easily find new messages, but > to check a flag change of an existing message (e.g. if you marked some older > message as read/unread), we need to list *all* messages in the mailbox and > compare their flags with what we have cached in Akonadi. If a massive > mailbox, this takes ages.... > > There are two steps that we should take: > 1) Check whether GMail have fixed their CONDSTORE support and re-enable it. > This would fix the problem instantly for you. > 2) Get rid of the SEARCH - it was an optimization to ensure that we always > get a batch of 2000 valid UIDs. Unfortunately on most IMAP servers (incl. > Gmail), SEARCH takes ages. We can still do batching for the search, if we do > a regular FETCH instead of UID FETCH, and have UIDs included in the > response. We can possibly even increase the batch size if we process things > more efficiently. That should be much much much faster. Even if CONDSTORE is > still broken on Gmail, this should speed up the sync massively. But this issue is present on Office365 (not GMail).(In reply to Daniel Vrátil from comment #1) > It's not really a loop. In the past we had problems with GMail having > really broken support for RFC4551/CONDSTORE extension > (https://datatracker.ietf.org/doc/html/rfc4551), which allows us to ask the > email server to give us all messages that have changed since the last time > we checked. Without CONDSTORE, we are able to easily find new messages, but > to check a flag change of an existing message (e.g. if you marked some older > message as read/unread), we need to list *all* messages in the mailbox and > compare their flags with what we have cached in Akonadi. If a massive > mailbox, this takes ages.... > > There are two steps that we should take: > 1) Check whether GMail have fixed their CONDSTORE support and re-enable it. > This would fix the problem instantly for you. > 2) Get rid of the SEARCH - it was an optimization to ensure that we always > get a batch of 2000 valid UIDs. Unfortunately on most IMAP servers (incl. > Gmail), SEARCH takes ages. We can still do batching for the search, if we do > a regular FETCH instead of UID FETCH, and have UIDs included in the > response. We can possibly even increase the batch size if we process things > more efficiently. That should be much much much faster. Even if CONDSTORE is > still broken on Gmail, this should speed up the sync massively. Hi. thank you for looking into it. Just a minor point, that is is not GMail but Office365, and for what I remember the 2000 limitation is a limitation to prevent Office365 to disconnect akonadi. (https://bugs.kde.org/show_bug.cgi?id=339393#c3 and https://bugsfiles.kde.org/attachment.cgi?id=89107 ) . In the trace I had shown above it start from start after a disconnect. Indeed, sorry about the confusion - I saw in the log it did not use CONDSTORE and automatically switched to Gmail in my head (because I remember the pain of trying to debug that :-)). Office365 doesn't support CONDSTORE at all (it's not listed in the CAPABILITY response), so the only solution to speed up sync is to get rid of the SEARCH and just use FETCH (which should also allow for larger batches and generally will be much faster). (In reply to Daniel Vrátil from comment #3) > Indeed, sorry about the confusion - I saw in the log it did not use > CONDSTORE and automatically switched to Gmail in my head (because I remember > the pain of trying to debug that :-)). Office365 doesn't support CONDSTORE > at all (it's not listed in the CAPABILITY response), so the only solution to > speed up sync is to get rid of the SEARCH and just use FETCH (which should > also allow for larger batches and generally will be much faster). I would like to remind that at least in 2014 , there was a limitation on the amount of data you could get in one go. (https://bugs.kde.org/show_bug.cgi?id=339393), I'm not sure if that still applies(*) (In reply to Daniel Vrátil from comment #3) > Indeed, sorry about the confusion - I saw in the log it did not use > CONDSTORE and automatically switched to Gmail in my head (because I remember > the pain of trying to debug that :-)). Office365 doesn't support CONDSTORE > at all (it's not listed in the CAPABILITY response), so the only solution to > speed up sync is to get rid of the SEARCH and just use FETCH (which should > also allow for larger batches and generally will be much faster). And please let me know if there is anything I could help with this issue |