Bug 280980

Summary:	kmail2 is too slow in importing huge amount of messages from imap account
Product:	[Frameworks and Libraries] Akonadi	Reporter:	Vasily Khoruzhick <anarsoul>
Component:	IMAP resource	Assignee:	Kevin Ottens <ervin>
Status:	RESOLVED FIXED
Severity:	normal	CC:	kdepim-bugs, nathan, vkrause
Priority:	NOR
Version:	4.7
Target Milestone:	---
Platform:	Arch Linux
OS:	Linux
Latest Commit:		Version Fixed In:
Sentry Crash Report:

Description Vasily Khoruzhick 2011-08-29 08:45:32 UTC

Version:           4.7 (using KDE 4.7.0) 
OS:                Linux

I've set up my gmail account in kmail2 (via imap) and it takes too much time (it's working 72 hours, eats almost all my RAM and CPU and it's not finished yet!) to finish fetching (importing?) messages from it. Account has huge amount of messages in it (~50000, ~1.5gb)

Reproducible: Always

Steps to Reproduce:
Setup IMAP account with thousands of messages in it

Actual Results:  
it takes forewer to import messages

Expected Results:  
messages imported at some appropriate amount of time (~10min for 10k would be OK, kmail should not fetch all message, only header, right?)

Comment 1 Vasily Khoruzhick 2011-08-29 15:26:49 UTC

Hmm, it's not a migration from kmail1. I've just set up an imap account in kmail2.

Comment 2 Christophe Marin 2011-08-29 19:07:43 UTC

ok

Comment 3 Nathan O'Brennan 2011-08-29 19:49:49 UTC

I have a Gmail account with many thousands of emails but total size in just under 1GB. I also have a local dovecot mail server in my office which I store mail on. I just upgraded the local dovecot imap server and decided to delete all my old mail on it. Then I setup kmail2 (4.7.00) to access my gmail imap and my local dovecot imap. I created some folders on my local imap and then copied emails from my gmail imap account to the new folders I just created on my local dovecot imap server. Each folder had less than one hundred messages and it seemed to copy within a couple minutes... I do have a slow internet service. After a few minutes it seemed all was good. No status bar ever showed any progress. I left for several hours and came back and shut of my laptop. Today, I turned it on and opened kmail2 and all the messages I copied from my gmail account are completely gone. They are not in my gmail account and not in kmail. I cannot find them anywhere! Sadly, I should have known better and backed them up, but I thought copying them would be safe. I am trying to backup my messages now using thunderbird to a third imap account on a different server. Then I will wipe my user account and start completely over with kde 4.7.0 and kmail2 and try again.

Comment 4 Kevin Ottens 2011-09-17 14:09:07 UTC

Please test again against 4.7.1 (just in case), but more likely against the upcoming 4.8 when it is out. I did some testing yesterday and managed to get around 80k emails from a GMail account in something like an hour. Which sounds fairly good for me.

If it still takes that very large amount of time when 4.8 hits the door for you, please feel free to reopen and I'll investigate further. In such a case, please also provide more information on the settings, like for instance the settings used for your resource and so on.

Comment 5 Vasily Khoruzhick 2011-09-17 15:17:04 UTC

Testing against 4.7.1 now (however on another PC and under Gentoo). akonadi+mysqld still eat all my CPU (c2d t5500). It's a no-go. Mutt eats nothing during fetch, same for trojita. Please, don't use mysql like this, looks like some request optimization is necessary. Should I file another bug about CPU usage? Btw, mutt fetched 149k in ~30min. So you still have space for optimization ;)

Comment 6 Kevin Ottens 2011-09-17 17:18:32 UTC

Well, performances issues and resource consumption as you describe them are more the land of the server in case you'd want to open another report.

Comment 7 Vasily Khoruzhick 2011-09-17 18:09:12 UTC

Two hours spent (or so), it did not finish yet (and there's no progress indicator! so I don't know how much it fetched!), so I'm deleting account from kmail and killing it. Looks like bug is not fixed in 4.7.1. I'll try it again when 4.8.0 is out.