Bug 323929

Confirm this bug. I have two imap accounts in kmail. Disabling indexing mail does not work. Also kmail over time uses a lot of memory - about 3GB

Yes, it doesn't feel like it is an indexing problem. More that kmail is asking again and again what is in a few folders.

I can also confirm this bug (Kde 4.11 on Kubuntu 12.10). It takes forever to retrieve an email item be it an IMAP or local maildir item.
Kmail2 takes about 31% of one cpu core and continues to consume RAM by the second. This behaviour does not stop even when exiting the program, you have to kill the process explicitely for it to stop.
I can also confirm the huge error load on the akonadi console, I able to post fragments of a saved log file if needed.

Created attachment 81894 [details]
akonadiconsole logfile

Today I started with a new nepomuk DB. Then I gradually added my POP and one IMAP account and everything worked fine.
The problems reappeared when I added my second IMAP account and I had the same symptoms again (high CPU load of kontact at around 31% + high RAM, emails take forever to be retrieved).
So I would also say it is more an akonadi problem than of kontact.
At this point I am a bit lost on how to pinpoint the culprit. I would appreciate the help of an akonadi DEV at this point.

Who gets this bug try:
mv ~/.local/share/akonadi/ ~/.local/share/akonadi_old
akonadictl restart
And try to start kmail. It works for me.

Hello. Thanks for the workaround, it worked also for me. I am crossing my fingers that this will be a permanent solution.

It doesn't work for me. I destroyed my akonadi configuration and database several times, and the problem always came back.

It seems we are not having exactly the same problem, PeterF: my kmail isn't leaking memory.

So I think it would be better for you to create another bug report, if you have this behaviour again.

I have restarted my machine twice since I applied the workaround and kontact is back to a useable state. Just hoping that Murphy's law doesn't kick in after this post.

If think I have the same problem. Moving ~/.local/share/akonadi/ helped for a while, and then the problem reappeared. In addition kontact is filling the job tracker with endless CollectionFetchJobs. I am not sure if https://bugs.kde.org/show_bug.cgi?id=323068 is related or not.

Just one addition: I can see and read all Mails in the akonadieconsole browser, the problem only appears when I start kontact.

knallio: It is exactly the same for me. That was what made me suspect that the problem was in kmail, not in akonadi.

I had a similar problem and managed to fix it. In my local maildirs, I had both "inbox" (the usual inbox) and "{archive}/inbox" (containing old mails moved out of my inbox). Somehow, KMail considered "{archive}/inbox" to be a special folder as well, and it kept replacing one inbox with the other in its special folders collection.

The fix was to create a new folder "{archive}/inb2", move the contents of "{archive}/inbox" there and then remove "{archive}/inbox". I did this in the Browser tab of akonadiconsole, since KMail was not responsive.

Some details from the debugging session:

(gdb) frame 5
#5  0x00007fa6f0828428 in Akonadi::SpecialCollections::registerCollection (this=0xb39820, type=..., 
    collection=...) at /usr/src/debug/kdepimlibs-4.11.0/akonadi/specialcollections.cpp:238
238         d->emitChanged( resourceId );
(gdb) printq4string resourceId
akonadi_mixedmaildir_resource_0
(gdb) print collection.id()
$1 = 30
(gdb) print oldCollection.id()
$2 = 64

Both these collections (30 and 64) had the same remoteId ("inbox"), but a different parentId.

I'll add the full backtrace as an attachment.

Created attachment 81999 [details]
Backtrace of KMail replacing one "inbox" with another

Not the problem I am having either… no two boxes with the same name.

I can confirm too. I upgraded with 3 imap mailboxes. Seems akonadi is fine, since everything normal (fetches email properly, notifications are ok, feeds nepomuk, etc.) until kmail (or kontact with kmail module enabled) started up.

After startup, kmail starts cycle over all e-mails in all mailbox and never finish. Akonadi console shows that kmail requesting "LSUB 0 INF (MIMETYPE (message/rfc822))" on all mailbox and processing all messsage. When finished, the whole process starts again. 

This endless process eats my cpu by akonadiserver and mysql (i use mysql as backend). It's impossible to work with kmail, because each mail fetching takes 3-10mins.

Since I'm using kde (and kmail) for work, I tried find solution. After a lot of search, I found some suggestion:
- search for same named resources which probably a leftover of bad migration process (didn't have any)
- removed all imap accounts, and added only one.
- removed all data in .local/share/akonadi/
- deleted all data in mysql database

None of above worked for me. Since I have to work, I given up and downgraded all kdepim related ebuilds (I'm using gentoo) to 4.10.5. I didn't downgraded akonadi-server, it's still at 1.10.2-r1.
Now I have some filter and organizer crash, akonadiconsole crashes on startup as well, but at least I can work with kmail.

It's interesting what Maarten ter Huurne wrote. I don't have subfolder named "inbox", but all of my imap mailboxes has "INBOX." prefix. Therefore all of my other special folders (sent, trash, drafts, templates) are subfolders under inbox. That means I have special folder in special folder too. So maybe I'm affected by that "replace and re-parse" bug as well 

Now I think I'll upgrade again and play with these special folders.

*** This bug has been confirmed by popular vote. ***

Hello!

4.11.1 upgrade solved all problems for me.

Thanks.

It seems to have solved the problem here too… but I had to destroy and recreate my akonadi database from scratch.

Wow, I think I have found what happened here. The problem only occurs when I start putting a lot of folders in my favorites. It just happened right now. After 24 hours of kmail working flawlessly (after I destroyed akonadi AND kmail's configuration) I started to add back my favorites to the top-left area, and the problem reappeared after I added one too much (it seems to start at 12 here). Can someone try this ?

I can confirm that adding more than 11 favorite folders in KMail will triggers fetching items from Akonadi. It's extremely weird and I'll try to see what's going on.

I haven't had this bug for 3 weeks now. I just kept 10 favorites.

Created attachment 83913 [details]
Patch to try to fix the 100% of cpu bug

I think the problem is the repeated connect calls to same signal/slot.
With UniqueConnection, it shouldn't happen but it can be a qt bug.

I have a similar problem here.
Can you test this patch for me??
It fixed the problem I had.
If this fix the problem to you I will send the patch to kmail developers.

Duplicates (??)
https://bugs.kde.org/show_bug.cgi?id=322038
https://bugs.kde.org/show_bug.cgi?id=307860

Duplicate too
https://bugs.kde.org/show_bug.cgi?id=325043
https://bugs.kde.org/show_bug.cgi?id=285097 (probably)

Nobody from kmail??
I will commit it if nobody say anything because I think it is a showstopper bug and it is lasting too long.

Sorry, I had to stop using kmail, so I cannot test it… I had something like 1 million files in my akonadi's mysql directory, so it was becoming unusable once more.

If you have a patch. please upload it to reviewboard (git.reviewboard.kde.org), where more developers can check and test it.

Submitted to reviewboard.
https://git.reviewboard.kde.org/r/114341/

Just a bit of feedback. I've built the current head of the 4.11 branch (including Raul Fernandes's patch). Solves the problem completely for me; thanks, Raul!

Isn't this a duplicate of https://bugs.kde.org/show_bug.cgi?id=323068 ? It seems to be the same codepath.

I'm not sure the applied fix actually fixed the actual problem, since the use of Qt::UniqueConnection was IMO just fine. The applied patch just disconnects  slotDefaultCollectionsChanged, which it probably shouldn't as we want to get updates for changed default collections. It of course works around the problem of the loop, but probably introduces a new bug.

Here's the analysis from thiago from 323068:
1) when KMail starts, I suppose MailCommon::Kernel::initFolders is called to initialise the folder listing
2) MailCommon::Kernel::initFolders creates a Akonadi::SpecialMailCollectionsDiscoveryJob job (derived from Akonadi::SpecialCollectionsDiscoveryJob)
3) when Akonadi::SpecialCollectionsDiscoveryJob finishes, Akonadi::SpecialCollectionsDiscoveryJob::slotResult is called
4) Akonadi::SpecialCollectionsDiscoveryJob::slotResult iterates over the result (201 folders). For two folders, it calls:
68                  d->mSpecialCollections->registerCollection(collection.attribute<SpecialCollectionAttribute>()->collectionType(), collection);
  The two folders have mRemoteId = "outbox" and mRemoteId = "caixa de saída" (don't ask me why one is localised and the other isn't).
5) because of those calls, Akonadi::SpecialCollectionsPrivate::emitChanged gets called
6) since those two folders have resourceId == mDefaultResourceId, the defaultCollectionsChanged() signal gets called, which ends up calling MailCommon::Kernel::initFolders
7) repeat ad nauseam

I agree with you.
It is not the right way to fix it but it is a workaround.
At least, it stops the behavior.
The solution is avoid the infinite loop through initFolders().
We are talking about it in reviewboard.
https://git.reviewboard.kde.org/r/114341/

I opened akonadiconsole browser and somehow found I had both a trash and wastebin folder in my local folders (despite using only imap and not the local folders). After deleting the trash folder, restarting the akonadi server and killing off a stuck kmail instance, all the hanging and high cpu usage mysqld process went away. Seems in my case the problem was due to there being multiple wastebins under different names.

Thank you for reporting the bugs and for all your comments and research.

Does this still happen with KDEPIM 4.14.6/7? Daniel, Christian, do you still see this?

Sorry, I am still on KDE 4.14.2, but I have not experienced any problems with high CPU load for a long time now.

I am seeing aknoadiserver going to 100% forever once kmail is started.

Even with kmail off, akonadi keeps working on the IMAP folders like a madman:

/usr/bin/akonadi_control
 \_ akonadiserver
 |   \_ /usr/libexec/mysqld --defaults-file=/home/hobbes/.local/share/akonadi/mysql.conf --datadir=/home/hobbes/.local/share/akonadi/db_data/ --socket=/tmp/akonadi-hobbes.aB9Ele/mysql.socket
 \_ /usr/bin/akonadi_agent_launcher akonadi_akonotes_resource akonadi_akonotes_resource_0
 \_ /usr/bin/akonadi_archivemail_agent --identifier akonadi_archivemail_agent
 \_ /usr/bin/akonadi_baloo_indexer --identifier akonadi_baloo_indexer
 \_ /usr/bin/akonadi_birthdays_resource --identifier akonadi_birthdays_resource
 \_ /usr/bin/akonadi_agent_launcher akonadi_contacts_resource akonadi_contacts_resource_0
 \_ /usr/bin/akonadi_followupreminder_agent --identifier akonadi_followupreminder_agent
 \_ /usr/bin/akonadi_agent_launcher akonadi_ical_resource akonadi_ical_resource_0
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_10
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_11
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_12
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_13
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_14
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_15
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_16
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_17
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_18
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_19
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_2
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_20
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_21
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_3
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_4
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_5
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_6
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_7
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_8
 \_ /usr/bin/akonadi_imap_resource --identifier akonadi_imap_resource_9
 \_ /usr/bin/akonadi_agent_launcher akonadi_maildir_resource akonadi_maildir_resource_0
 \_ /usr/bin/akonadi_maildispatcher_agent --identifier akonadi_maildispatcher_agent
 \_ /usr/bin/akonadi_mailfilter_agent --identifier akonadi_mailfilter_agent
 \_ /usr/bin/akonadi_migration_agent --identifier akonadi_migration_agent
 \_ /usr/bin/akonadi_newmailnotifier_agent --identifier akonadi_newmailnotifier_agent
 \_ /usr/bin/akonadi_notes_agent --identifier akonadi_notes_agent
 \_ /usr/bin/akonadi_sendlater_agent --identifier akonadi_sendlater_agent

This is Fedora 22:

- akonadi-1.13.0-16
- kdepim-runtime-libs-4.14.10-5

How do I confirm the problem?

Please retest with most recent KDEPIM + Akonadi 15.12.

It has a ton of performance fixes that are not going to be backported due to shift from Akonadi from Qt4 to Qt5.

Also please carefully check whether the issue you see is the same as this issue. I already have the impression that this bug contains more than one issue. If at doubt, please rather open a new issue.

Is Akonadi actually fetching mails? If in doubt please open a new mail and include more information. The processlist does not do much to aid in debugging. See for example:

https://techbase.kde.org/Projects/PIM/Akonadi/Development_Tools

At least some information on what Akonadi is actually doing during high load (debugger) would be interesting.

And in case of IMAP issues (you seem to have a huge lot of IMAP accounts configured):
https://techbase.kde.org/Projects/PIM/Akonadi/Debug_IMAP

As performance fixes are unlikely to be backported, I suggest doing these debugging steps only if the issue persists with KDEPIM 15.12 or later.

Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least 15 days. Please provide the requested information as soon as possible and set the bug status as REPORTED. Due to regular bug tracker maintenance, if the bug is still in NEEDSINFO status with no change in 30 days, the bug will be closed as RESOLVED > WORKSFORME due to lack of needed information.

For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please set the bug status as REPORTED so that the KDE team knows that the bug is ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!

Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least 30 days. The bug is now closed as RESOLVED > WORKSFORME due to lack of needed information.

For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

Thank you for helping us make KDE software even better for everyone!