Bug 362896

Summary:	Account freezes for several weeks when processing lots of emails
Product:	[Applications] kmail2	Reporter:	krystalcode <dbozelos>
Component:	message list	Assignee:	kdepim bugs <kdepim-bugs>
Status:	RESOLVED NOT A BUG
Severity:	critical	CC:	Adderess91, capsel+kde
Priority:	NOR
Version:	4.14.10
Target Milestone:	---
Platform:	Fedora RPMs
OS:	Linux
Latest Commit:		Version Fixed In:
Sentry Crash Report:

Description krystalcode 2016-05-10 15:04:42 UTC

Kmail seems not to be able to handle operations on thousands of emails well. I have encountered this in 2 cases.

1) Adding a new account with thousands of emails. The message list showed nothing (0 emails) for almost a month. After the first shock, thinking that my emails were wiped out, I realised that the emails were still on the IMAP server. They showed up locally after several weeks. Then things went back to normal.

2) Later I decided to apply an expiry filter on the Inbox of the same account. I added a rule to move emails to a local folder. First tried moving only 2 months worth of emails, worked well and fast. Then 6 months worth of emails, worked well and fast. Encouraged, I increased the number of emails to move to about 2-3 years worth of emails, and ... I am still waiting. The account went blank (message list shows nothing at all). After 2 weeks, few emails showed up, but the account is still unusable - the vast majority of emails are still hiding, including those that were not supposed to be moved.

I don't know enough of Kmail/akonadi's internals to interpret this bug, but it looks like the tasks for processing the emails are forgotten, and randomly picked up at some point. System activity does not show any CPU/Memory usage apart from 1% for Kmail, which I believe is just normal usage. Even with very low priority, the tasks should have finished by now.

It would be great if some visibility would be provided about the status of tasks. I wouldn't mind forcing processing all emails with high priority and burn my CPU for an hour, but get them done, rather than rendering Kmail unusable for an unpredictable amount of time.

Comment 1 krystalcode 2016-06-19 15:59:34 UTC

Additional information:

After setting a few thousand of emails to expire by date, the whole folder (Inbox) stopped displaying emails and it was empty and unusable for a month or so. It was also blocking other operations - I added another account and emails were not downloaded. Other existing accounts were working.

I'm on a quite slow internet connection. At some point I monitored network traffic and I noticed "akonadi_imap_resource" was downloading a huge amount of data from the IMAP server. I left it all night long, it downloaded about 1.5GB and still going.

Another day I happened to be in a hotel with a decent internet connection. I monitored the network traffic again, it downloaded about 2GB of data within an hour or so, and it finished - all my email was displayed again.

Points of interest:

- Why did Kmail (or akonadi) needed to download so much data? All the email were locally available via the offline use setting, it shouldn't have to download anything, only move them to another folder and delete them from the server.

- Even if it has to download the emails again, the amount of data (2GB+) indicates it was downloading all emails again, not just the ones requested to be moved. That requested emails would be much less.

- It seems like it had to do all operations at one go, otherwise it would start from scratch next time. This was going on for a month, maybe more, there was certainly plenty of time to download everything even with a slow connection. There were probably dozens of GBs of data downloaded over this time with this behaviour. Still, it only worked when it was able to download 2GB+ in one go, otherwise the Inbox was empty and unusable.

When expiring smaller amounts of emails e.g. 100 days, everything works ok.