Bug 277597 - Akonadi takes up to 100% CPU and is very slow fetching mail
Summary: Akonadi takes up to 100% CPU and is very slow fetching mail
Status: RESOLVED FIXED
Alias: None
Product: Akonadi
Classification: Frameworks and Libraries
Component: POP3 Resource (show other bugs)
Version: unspecified
Platform: Ubuntu Linux
: NOR crash (vote)
Target Milestone: ---
Assignee: Thomas McGuire
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-07-12 06:26 UTC by kde
Modified: 2011-09-10 13:25 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.7.2


Attachments
Patch to fix slowness of pop3 mail reading (4.58 KB, patch)
2011-08-17 16:34 UTC, Oldřich Jedlička
Details
Patch to fix slowness of pop3 mail reading (git version) (5.39 KB, patch)
2011-08-17 17:24 UTC, Oldřich Jedlička
Details

Note You need to log in before you can comment on or make changes to this bug.
Description kde 2011-07-12 06:26:34 UTC
Version:           unspecified (using KDE 4.6.4) 
OS:                Linux

I have installed the most up to date kdepim packages with the appropriate akonadi packages from Kubuntu experimental.

I use a pop3 mail and the akonadi resource goes to 90-100% and so makes system unusable.

Reproducible: Always

Steps to Reproduce:
having a pop3 mail box defined

Actual Results:  
mail is very slow to download and cpu goes to 100%

Expected Results:  
just started
Comment 1 Oldřich Jedlička 2011-07-13 05:55:34 UTC
Same here, I'm using KDE/KMail2 4.7RC2 (4.6.95), Akonadi 1.5.80. Each mail download takes very long time using 100% of CPU and prevents me synchronizing mail several times per day.
Comment 2 Oldřich Jedlička 2011-07-13 20:27:44 UTC
Akonadi 1.6.0, same issue. Isn't the slowness caused by checking for duplicit mails (which would be unnecessary if it worked - bug 271231)? I have a lot of mails here (1806MB).
Comment 3 subscryer 2011-08-01 15:23:07 UTC
I can confirm the behaviour. KDE 4.7 on gentoo.
I checked with nethogs and akonadi is mostly idle on the network while it keeps the cpu busy. The akonadi console shows long pauses between activity.
At the current rate the sync could take days.
On a side note kmail doesn't show any meaningful progress indication as every folder sync is 100% complete in a few seconds in its opinion then sits there at 100% for the rest of the time needed.
Comment 4 Oldřich Jedlička 2011-08-17 15:29:28 UTC
Is there anything I can help with? Enable some logging, look into code (change something) or so? This problem annoys me a lot (the synchronization is unusable), so I would like to help analysing it. I don't know where to start looking.
Comment 5 Oldřich Jedlička 2011-08-17 16:34:47 UTC
Created attachment 62906 [details]
Patch to fix slowness of pop3 mail reading

The code uses QMap::key(value, default_value) to search for the right message UID. According to Qt documentation, the method is: 

  This function can be slow (linear time), because QMap's internal data structure is optimized for fast lookup by key, not by value.

For lot of emails, the function is really really slow, because it tries to match every value until it finds one.

I will send the patch to reviewboard as soon as I get an account created.
Comment 6 Oldřich Jedlička 2011-08-17 17:24:48 UTC
Created attachment 62910 [details]
Patch to fix slowness of pop3 mail reading (git version)
Comment 7 Thomas McGuire 2011-08-17 21:12:23 UTC
Oldřich, thank you very much for working on this! Looking forward to the reviewboard patch. Seems to be ok at first glance. Good job finding the slowness. When I coded the resource, I didn't code for efficiency and tested only with my usual amount of mails (I don't keep mails on the server), therefore I never noticed the problem.

Did you run the unit test of the POP3 resource? It is actually a complete system test that brings up a test akonadi server and everything, it tests a lot of stuff of the POP3 resource. I hope it still works :)
Comment 8 Oldřich Jedlička 2011-08-18 05:28:11 UTC
(In reply to comment #7)
> Oldřich, thank you very much for working on this! Looking forward to the
> reviewboard patch. Seems to be ok at first glance. Good job finding the
> slowness. When I coded the resource, I didn't code for efficiency and tested
> only with my usual amount of mails (I don't keep mails on the server),
> therefore I never noticed the problem.
> 
> Did you run the unit test of the POP3 resource? It is actually a complete
> system test that brings up a test akonadi server and everything, it tests a lot
> of stuff of the POP3 resource. I hope it still works :)

Um, actually no. I did the change, let the modified Gentoo ebuild do the compilation/installation work and that's it :-)

It is review #102353 under kdepim group.
Comment 9 Thomas McGuire 2011-08-21 17:52:43 UTC
Git commit a5be56cf25949db75c88c2c70f720d8c7d2b5ace by Thomas McGuire, on behalf of Oldřich Jedlička.
Committed on 21/08/2011 at 19:51.
Pushed by tmcguire into branch 'master'.

Fix bad performance when fetching mail with many mails on the server.

With many mails on the server, the usage of QMap::key() was slowing
down things a lot.

BUG: 277597
REVIEW: 102353

M  +15   -1    resources/pop3/jobs.cpp
M  +2    -0    resources/pop3/jobs.h
M  +4    -0    resources/pop3/pop3resource.h
M  +6    -3    resources/pop3/pop3resource.cpp

http://commits.kde.org/kdepim-runtime/a5be56cf25949db75c88c2c70f720d8c7d2b5ace
Comment 10 Oldřich Jedlička 2011-09-09 18:12:14 UTC
This bug fix isn't part of 4.7.1
Comment 11 Thomas McGuire 2011-09-10 13:15:57 UTC
> This bug fix isn't part of 4.7.1

Sorry, right, no one merged it to the 4.7 branch yet.
I did that now with a9f648f4d2c7b0086c256981a9f08f71335b5798, so it will be in 4.7.2.