Bug 260647 - akonadi_mixedmaildir_resource is using too much memory
Summary: akonadi_mixedmaildir_resource is using too much memory
Status: RESOLVED FIXED
Alias: None
Product: Akonadi
Classification: Frameworks and Libraries
Component: Mixed Maildir resource (show other bugs)
Version: 1.4.80
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: Sergio Martins
URL:
Keywords:
: 290355 (view as bug list)
Depends on:
Blocks:
 
Reported: 2010-12-18 18:09 UTC by Jure Repinc
Modified: 2013-11-02 11:00 UTC (History)
15 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.11.1


Attachments
akonadi.out (242.38 KB, text/plain)
2010-12-20 00:24 UTC, Jure Repinc
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jure Repinc 2010-12-18 18:09:28 UTC
Version:           1.4.80 (using Devel) 
OS:                Linux

A couple of weeks ago I installed KDE SC 4.6 from SVN and at this time when I started KMail it converted all the stuff to use Akonadi. Ever since then I see huge memory usage when I start KMail. Th process that uses the most memory is akonadi_mixedmaildir_resource which always uses about 1200 MiB of memory. In addition to this KMail uses 700 MiB. This is a lot more than KMail used to use before using Akonadi. If I remember correctly ibefore the conversion only KMail was using memory and it was at about the same level of usage than KMail now uses

Reproducible: Always

Steps to Reproduce:
1. Convert an old non-Akonadi POP3 accout wit a lot of mail to Akonadi
2. Open KMail

Actual Results:  
akonadi_mixedmaildir_resource uses way too much memory

Expected Results:  
The memory usage should be about the same as before without Akonadi

My inbox folder which is set to open when KMail is opened has about 35000 messages which occupy about 200 MiB of storage (according to the info popup over inbox folder in KMail). I also have more folders with more mail and most of them are set to expire mail older than 1 month. Akonadi and KMail I'm using are compiled on 2010-12-17
Comment 1 Jure Repinc 2010-12-20 00:24:16 UTC
Created attachment 55086 [details]
akonadi.out
Comment 2 Daniel 2011-01-12 19:26:22 UTC
I have similar problem.

Once my system boots up, I am using arch linux, kde 4.5.4, my memory usage is:

450 MB with akonadi enabled
230 MB with akonadi disabled (edit akonadiserverrc and write StartServer=false )

I usually don't care a lot about memory usage of my applications, but wow!, 200 MB is too much, it is about 50% of the DE memory usage, and I guess a lot of people don't even use akonadi applications.

Regards
Jaime
Comment 3 Daniel 2011-01-12 19:35:15 UTC
As my problem is not exactly the OP reported bug, I filled a new bug: 262981
Comment 4 Hal V. Engel 2011-02-04 22:04:17 UTC
I am also seeing very bad memory usage by kmail2 and akonadi.  akonadiserver is actually using more memory on my system than akonadi_mixedmaildir_resource but only by a small margin with both using close to 1.1G of virtual memory.  On top of that kmail2 is using over 500M.  I don't understand why a mail app needs close to 3G of memory.  Using this set of apps makes my system unusable and this is a major issue.
Comment 5 Roman K. 2011-07-29 05:59:24 UTC
I can confirm this issue with KMail 4.7.0 and akonadi 4.7.0 (this is the version number opensuse uses) with KDE 4.7.0, akonadi-runtime package has version 1.6.0.

I have about 5000 mail in my inbox and akonadi_mixedmaildir_resource uses 716 MB of memory.
Comment 6 H.H. 2011-07-30 14:47:54 UTC
for me akonadi_mixedmaildir_resource is using also over 200mb, followed by kmail with 140mb.
Comment 7 Marcel Wiesweg 2011-07-31 18:32:02 UTC
I can confirm this bug. Talking about about 80000 messages spread over three folders. When I open these folders in order, the resource takes 1,5 GB of memory.
Reproducible when killing the resource and starting fresh. Memory does not decrease after some minutes or when closing kontact.
The increase in memory does not seem to be linear with number of mails per folder; the last folder with only 12000 eats more than 600 MB of the 1,5 GB.

If there is need and someone tells me how to start a resource under valgrind, I can produce massif logs.
Comment 8 George L. Emigh 2011-07-31 19:54:19 UTC
%MEM VSZ     RSS
16.6 1615576 1363120 akonadi_mixedmaildir_resource

This is what I would call out of hand, and overall performance is down, help?
Comment 9 George L. Emigh 2011-07-31 19:56:50 UTC
Was so frustrated about the new Kmail I forgot to mention this is with the Kmail as found in KDE 4.7.0
Comment 10 Marcel Wiesweg 2011-08-01 18:08:16 UTC
Note that performance is down by orders of magnitude as soon as the system swaps, which it may do when kmail and the resources use these extreme amounts of memory
Comment 11 Thomas Zell 2011-08-15 19:55:00 UTC
Also the process seems to keep running, even after Kontact is shut down.
Comment 12 Marcus 2011-09-07 00:50:35 UTC
Want to confirm:
Same here after updating to 4.7 on Kubuntu 11.4 yesterday.
Akonadi_mixedmaildir needs almost 450MB RAM.
Process is keeping running until restart the machine. opening any mail folders in kmail and the service does start again and takes too much memory.

Akonadi_nepomuk_email_feeder takes another 720MB...
Out of 2GB on my Laptop.
Comment 13 Marcus 2011-09-08 06:15:41 UTC
went back to 4.6. KDE 4.7 this way is unusable on my laptop.
Comment 14 Sebastian Körner (Mailings) 2011-09-08 17:31:05 UTC
Just wanted to share that I got it down a little bit for me.
In my case some SpamChecker had a fight with Kmail2 (Lots of conflict message) resulting in Mails with MIME bodies duplicated again and again.
When removing all the duplicate MIME Bodies using a little script, I got the memory usage down from 6GB over all to 500MB.
Comment 15 Roman K. 2011-09-09 09:34:41 UTC
still present with kmail 4.7.1
Comment 16 Marcel Wiesweg 2011-09-09 17:02:10 UTC
> When removing all the duplicate MIME Bodies using a little script, I got the
> memory usage down from 6GB over all to 500MB.

Which is still way too much: kontact 4.6 only uses <200 MB of memory
Comment 17 mkkot 2011-09-18 11:12:50 UTC
I have found a strange thing. After starting kmail and loading messages from default folder, akonadi_mixedmaildir_resource takes about 1,2GB of RAM. When kmail is already synced, you can safely killall akonadi_mixedmaildir_resource. Clicking on another folders will result the process to reappear, using about 105MB of memory and less than 10% CPU time when syncing. I have nepomuk enabled. I have also removed my old spam filters and created new ones using kmail's wizard.

When all folders are synced, you can killall it once again safely. Kmail takes then ~100MB of RAM and mysql process even less. Anyway, "Search messages" feature still doesn't work for me...
Comment 18 Gulraj Rijhwani 2012-02-09 14:05:37 UTC
I have a very similar problem.

Just made the jump step from Fedora 14 to 16, and now Kmail (4.7 from 4.4)
insists on Akonadi indexing all the mail, which it simply is not man enough to
do.  This was AFTER having to figure out that Kmail, having performed an
automated "migration" had failed to give Akonadi a path for the local mail
folders, and kept crashing out.

This machine has 2GB of RAM and 2GB of swap - Akonadi grows to use it all until
the the synchronising process trying to regrow the indexes
(/usr/bin/akonadi_mixedmaildir_resource --identifier
akonadi_mixedmaildir_resource_0) is assassinated by the kernel, leaving Kmail
without all of my organised mail folders from the last 5 years.  It's less than
useless - I have lost access to some very important historic messages, and I
cannot even read critical current work email...
Comment 19 Sergio Martins 2013-07-17 07:59:26 UTC
*** Bug 290355 has been marked as a duplicate of this bug. ***
Comment 20 Sergio Martins 2013-07-18 09:12:04 UTC
Git commit 815957a4a9ae567f271453aac222abf4c09900f9 by Sergio Martins.
Committed on 17/07/2013 at 09:45.
Pushed by smartins into branch 'KDE/4.11'.

mixedmaildir: don't use 1.5GB of memory when importing a very big folder

Subjobs pile up too fast because they are cheap to create and
because it's taking around 5 event loops for one to start.
Related: bug 317122, bug 309732, bug 307066, bug 295651

M  +41   -6    resources/mixedmaildir/retrieveitemsjob.cpp
M  +4    -2    resources/mixedmaildir/retrieveitemsjob.h

http://commits.kde.org/kdepim-runtime/815957a4a9ae567f271453aac222abf4c09900f9