Bug 323068 - Exponential growth of CollectionFetchJobs (infinite loop)
Summary: Exponential growth of CollectionFetchJobs (infinite loop)
Status: RESOLVED FIXED
Alias: None
Product: kmail2
Classification: Applications
Component: folders (show other bugs)
Version: Git (master)
Platform: Compiled Sources Linux
: NOR grave
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-08-01 04:52 UTC by Thiago Macieira
Modified: 2015-05-04 22:39 UTC (History)
13 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.14.6


Attachments
Kmail, freshly started up at 10:44 (39.42 KB, image/png)
2013-11-16 15:55 UTC, Georg Greve
Details
Kmail, after idling for 6 hours (35.54 KB, image/png)
2013-11-16 15:58 UTC, Georg Greve
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Thiago Macieira 2013-08-01 04:52:44 UTC
I don't know what caused this. Downgrading to a previous build of akonadi + kdepim* did not help. This problem happens now with the build from April 11 as well as the latest sources (as of right now).

Whenever Kontact / KMail is running, it is constantly requesting a full folder listing, for whatever reason. Here's what I can tell from debugging and from Akonadi Console:

Job tracker shows that KMail is creating *two* Akonadi::CollectionFetchJobs at a time, two or three times per second. Since each job takes about a quarter second to complete running, there's a build up of jobs. That is, every second, KMail is requesting 1.25 to 1.5 second of work from the Akonadi Server. This does not stop.

The Debugger in Akonadi Console shows an endless stream of:
 [red] 24 LSUB 0 INF (MIMETYPE (message/rfc822)) () 
[blue] * 420 419 (NAME "Common Folders" MIMETYPE (message/rfc822 inode/directory) REMOTEID ".Common Folders" REMOTEREVISION "" RESOURCE "akonadi_imap_resource_3" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT 60 SYNCONDEMAND true LOCALPARTS (ENVELOPE HEAD)) AccessRights "wcdCD" imapacl "anyone lrswipcda %% " collectionannotations "/vendor/cmu/cyrus-imapd/partition default" timestamp "1375301134") 
[a lot more blue]
[blue] 24 OK List completed 

In stderr, kontact is printing these two messages over and over again:
kontact(8751)/libakonadi Akonadi::SpecialCollectionsPrivate::emitChanged: Emitting changed for "akonadi_maildir_resource_0"
kontact(8751)/libakonadi Akonadi::SpecialCollectionsPrivate::emitChanged: Emitting defaultFoldersChanged.

(see below for the lines in LSUB that contained "akonadi_maildir_resource_0")

The backtrace of that function is:
#0  Akonadi::SpecialCollectionsPrivate::emitChanged (this=0xda54f0, resourceId=...)
    at /home/thiago/src/kde4/kde/kdepimlibs/akonadi/specialcollections.cpp:76
#1  0x00007ffff686f5ce in Akonadi::SpecialCollections::registerCollection (this=0xddf780, type=..., collection=...)
    at /home/thiago/src/kde4/kde/kdepimlibs/akonadi/specialcollections.cpp:266
#2  0x00007ffff6880046 in Akonadi::SpecialCollectionsDiscoveryJob::slotResult (this=0x9c7190, job=<optimized out>)
    at /home/thiago/src/kde4/kde/kdepimlibs/akonadi/specialcollectionsdiscoveryjob.cpp:68
#3  0x00007ffff6c7bb2a in QMetaObject::activate (sender=0x16bf8f0, m=0x7ffff711f460, local_signal_index=3, argv=
    0x7fffffffc8d0) at /home/thiago/src/qt/qt-4.8/src/corelib/kernel/qobject.cpp:3547
#4  0x00007ffff6f62442 in KJob::result (this=this@entry=0x16bf8f0, _t1=_t1@entry=0x16bf8f0)
    at /home/thiago/obj/kde4/kdelibs/kdecore/kjob.moc:207
#5  0x00007ffff6f62480 in KJob::emitResult (this=0x16bf8f0) at /home/thiago/src/kde4/kdelibs/kdecore/jobs/kjob.cpp:318

I should note that mDefaultResourceId is "akonadi_maildir_resource_0" (a match) in frame 0.

The next CollectionFetchJob is created from:
#0  Akonadi::CollectionFetchJob::CollectionFetchJob (this=0x18c85d0, collection=..., type=
    Akonadi::CollectionFetchJob::Recursive, parent=0x1951090)
    at /home/thiago/src/kde4/kde/kdepimlibs/akonadi/collectionfetchjob.cpp:103
#1  0x00007ffff687fbce in Akonadi::SpecialCollectionsDiscoveryJob::start (this=0x1951090)
    at /home/thiago/src/kde4/kde/kdepimlibs/akonadi/specialcollectionsdiscoveryjob.cpp:54
#2  0x00007fffa8122ead in MailCommon::Kernel::initFolders (this=0xda3bb0)
    at /home/thiago/src/kde4/kde/kdepim/mailcommon/kernel/mailkernel.cpp:155
#3  0x00007ffff6c7bb2a in QMetaObject::activate (sender=0xdb29c0, m=0x7ffff69581a0, local_signal_index=1, argv=0x0)
    at /home/thiago/src/qt/qt-4.8/src/corelib/kernel/qobject.cpp:3547
#4  0x00007ffff686e188 in Akonadi::SpecialCollectionsPrivate::emitChanged (this=0xdb9250, resourceId=...)
    at /home/thiago/src/kde4/kde/kdepimlibs/akonadi/specialcollections.cpp:86

(note: this is a release build; the frame for slot MailCommon::Kernel::slotDefaultCollectionsChanged is omitted is not present above, but it calls initFolders()).

Conclusion:
1) when KMail starts, I suppose MailCommon::Kernel::initFolders is called to initialise the folder listing
2) MailCommon::Kernel::initFolders creates a Akonadi::SpecialMailCollectionsDiscoveryJob job (derived from Akonadi::SpecialCollectionsDiscoveryJob)
3) when Akonadi::SpecialCollectionsDiscoveryJob finishes, Akonadi::SpecialCollectionsDiscoveryJob::slotResult is called
4) Akonadi::SpecialCollectionsDiscoveryJob::slotResult iterates over the result (201 folders). For two folders, it calls:
68                  d->mSpecialCollections->registerCollection(collection.attribute<SpecialCollectionAttribute>()->collectionType(), collection);
  The two folders have mRemoteId = "outbox" and mRemoteId = "caixa de saída" (don't ask me why one is localised and the other isn't).
5) because of those calls, Akonadi::SpecialCollectionsPrivate::emitChanged gets called
6) since those two folders have resourceId == mDefaultResourceId, the defaultCollectionsChanged() signal gets called, which ends up calling MailCommon::Kernel::initFolders
7) repeat ad nauseam


Reproducible: Always

Steps to Reproduce:
(for me, at least)
1. Start KMail

Actual Results:  
KMail's UI is responsive, but reading any email is impossible after a while, since the message lists do not update and the mail contents do not show.

Expected Results:  
Reading email is possible.

The lines in the LSUB result that contain "akonadi_maildir_resource_0" are:

* 9 4 (NAME "caixa de entrada" MIMETYPE (message/rfc822 inode/directory) REMOTEID "caixa de entrada" REMOTEREVISION "1375310411000" RESOURCE "akonadi_maildir_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT 1 SYNCONDEMAND true LOCALPARTS (ENVELOPE)) SpecialCollectionAttribute "inbox" ENTITYDISPLAY "(\"caixa de entrada\" \"mail-folder-inbox\" \"\" ())" newmailnotifierattribute "") 
* 6 4 (NAME "caixa de saída" MIMETYPE (message/rfc822 inode/directory) REMOTEID "caixa de saída" REMOTEREVISION "0" RESOURCE "akonadi_maildir_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT 1 SYNCONDEMAND true LOCALPARTS (ENVELOPE)) SpecialCollectionAttribute "outbox" ENTITYDISPLAY "(\"caixa de saída\" \"mail-folder-outbox\" \"\" ())") 
* 7 4 (NAME "enviado" MIMETYPE (message/rfc822 inode/directory) REMOTEID "enviado" REMOTEREVISION "1375309940000" RESOURCE "akonadi_maildir_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT 1 SYNCONDEMAND true LOCALPARTS (ENVELOPE)) SpecialCollectionAttribute "sent-mail" ENTITYDISPLAY "(\"enviados\" \"mail-folder-sent\" \"\" ())") 
* 11 4 (NAME "lixo" MIMETYPE (message/rfc822 inode/directory) REMOTEID "lixo" REMOTEREVISION "1368747529000" RESOURCE "akonadi_maildir_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT 1 SYNCONDEMAND true LOCALPARTS (ENVELOPE)) SpecialCollectionAttribute "trash" ENTITYDISPLAY "(\"lixo\" \"user-trash\" \"\" ())") 
* 12 4 (NAME "modelos" MIMETYPE (message/rfc822 inode/directory) REMOTEID "modelos" REMOTEREVISION "" RESOURCE "akonadi_maildir_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT 1 SYNCONDEMAND true LOCALPARTS (ENVELOPE)) SpecialCollectionAttribute "templates" ENTITYDISPLAY "(\"modelos\" \"document-new\" \"\" ())") 
* 5 4 (NAME "outbox" MIMETYPE (message/rfc822 inode/directory) REMOTEID "outbox" REMOTEREVISION "0" RESOURCE "akonadi_maildir_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT 1 SYNCONDEMAND true LOCALPARTS (ENVELOPE)) SpecialCollectionAttribute "outbox" ENTITYDISPLAY "(\"caixa de saída\" \"mail-folder-outbox\" \"\" ())") 
* 10 4 (NAME "rascunhos" MIMETYPE (message/rfc822 inode/directory) REMOTEID "rascunhos" REMOTEREVISION "0" RESOURCE "akonadi_maildir_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT 1 SYNCONDEMAND true LOCALPARTS (ENVELOPE)) SpecialCollectionAttribute "drafts" ENTITYDISPLAY "(\"rascunhos\" \"document-properties\" \"\" ())") 
* 4 0 (NAME "Pastas locais" MIMETYPE (message/rfc822 inode/directory) REMOTEID "/home/thiago/.local/share/local-mail" REMOTEREVISION "" RESOURCE "akonadi_maildir_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT false INTERVAL -1 CACHETIMEOUT 1 SYNCONDEMAND true LOCALPARTS (ENVELOPE)) AccessRights "wcdC" SpecialCollectionAttribute "local-mail" ENTITYDISPLAY "(\"Pastas locais\" \"folder\" \"\" ())")
Comment 1 Thiago Macieira 2013-08-01 04:58:21 UTC
Hmm... note how there's a listing for *both* a localised and a non-localised name of the outbox:

* 6 4 (NAME "caixa de saída" MIMETYPE (message/rfc822 inode/directory) REMOTEID "caixa de saída" REMOTEREVISION "0" RESOURCE "akonadi_maildir_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT 1 SYNCONDEMAND true LOCALPARTS (ENVELOPE)) SpecialCollectionAttribute "outbox" ENTITYDISPLAY "(\"caixa de saída\" \"mail-folder-outbox\" \"\" ())") 
* 5 4 (NAME "outbox" MIMETYPE (message/rfc822 inode/directory) REMOTEID "outbox" REMOTEREVISION "0" RESOURCE "akonadi_maildir_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT 1 SYNCONDEMAND true LOCALPARTS (ENVELOPE)) SpecialCollectionAttribute "outbox" ENTITYDISPLAY "(\"caixa de saída\" \"mail-folder-outbox\" \"\" ())") 

Both folders exist on disk. There is nothing in that "outbox" folder, though. It shouldn't exist, but does.

After removing the unwanted folder from disk (rm -rf) and restarting the maildir agent, the problem seems to go away.
Comment 2 kerub 2013-08-27 21:08:46 UTC
+1

After upgrading to KDE 4.11 (in Arch Linux) Kmail (or Kontact) stops displaying email content very soon after starting. With akonadiconsole I discovered that Kmail/Kontact drops at least two (or up to 7) Akonadi::CollectionFetchJob per timestamp and they keep on waiting for minutes.

As far as I can tell local folders are set up correctly.
Comment 3 Vito De Tullio 2013-08-30 03:57:33 UTC
Same problem here: after 4.11 (on slackware-current + alienBob kde 4.11) reading email content needs 5+ minutes...
Comment 4 Jan 2013-09-13 20:00:09 UTC
Confirm with:
Kubuntu 13.04
KDE 4.11.00
Akonadi Console Version 0.99
Comment 5 Martin 2013-10-31 14:04:56 UTC
I can confirm this (Kubuntu 13.10, KDE 4.11.2). In the akonadi console I could see up to 20 new jobs every second. Right before I shut down akonadi, kmail was consuming more than 1GB of RAM.
Comment 6 Georg Greve 2013-11-15 11:08:42 UTC
Seeing this on two folders on Fedora, installed KDE Red Hat repos:

kdepim-4.11.3-1.fc19.x86_64
kdepim-debuginfo-4.11.3-1.fc19.x86_64
kdepim-libs-4.11.3-1.fc19.x86_64
kdepim-runtime-4.11.3-1.fc19.x86_64
kdepim-runtime-debuginfo-4.11.3-1.fc19.x86_64
kdepim-runtime-libs-4.11.3-1.fc19.x86_64
kdepimlibs-4.11.3-2.fc19.x86_64
kdepimlibs-akonadi-4.11.3-2.fc19.x86_64
kdepimlibs-debuginfo-4.11.3-2.fc19.x86_64
kdepimlibs-kxmlrpcclient-4.11.3-2.fc19.x86_64

Let me know if there is anything I can do to provide debug info.
Comment 7 Daniel Vrátil 2013-11-15 12:01:11 UTC
Georg, can you check in Akonadi Console in Browser tab for duplicated folders in "Local Folders"? If there are any duplicates, remove them. The duplicates can be like "Inbox" and "whatever inbox is in your language"
Comment 8 Georg Greve 2013-11-15 13:37:54 UTC
I've looked into akonadi console, but there are no duplicate folders, also the folder affected is not inbox, it's a subfolder of inbox, and subfolder of that subfolder, so something folder "c" in "j" in the following structure:

inbox/
├── a
├── b
├── c
│   ├── h
│   ├── i
│   ├── j
│   ├── k
│   ├── l
│   ├── m
│   ├── n
│   ├── o
│   └── p
├── d
├── e
├── f
└── g

FWIW, some IMAP folders are marked red in akonadiconsole, including (but not only) the folders affected. Christian Mollekopf has some debug output, although he cannot share it (contains data from my system) but he might be able to look for something suspicious.

If there is anything I can do to help track this down, please shout.
Comment 9 Georg Greve 2013-11-16 15:55:36 UTC
Created attachment 83599 [details]
Kmail, freshly started up at 10:44
Comment 10 Georg Greve 2013-11-16 15:58:00 UTC
Created attachment 83600 [details]
Kmail, after idling for 6 hours

Just to get an idea of the memory consumption/leakage when doing absolutely nothing.

Note this keeps one core of the CPU at 100% (four core system, hence 25%) and since it is a laptop it keeps it from scaling down, ever, which means it burns through battery like nothing and the laptop is becoming quite warm, including all the side effects that brings.
Comment 11 Georg Greve 2013-11-18 07:43:19 UTC
FWIW, another data point: 

Switching Kmail to offline and visiting the affected folder can stop the CPU churn. 

When i then keep it open in one tab and go back online, the CPU churn seems gone for the moment.
Comment 12 Fabian 2013-11-18 07:50:19 UTC
(In reply to comment #7)
> Georg, can you check in Akonadi Console in Browser tab for duplicated
> folders in "Local Folders"? If there are any duplicates, remove them. The
> duplicates can be like "Inbox" and "whatever inbox is in your language"

For me this seems to have solved the problem. I had two local trash folders.  So far akonadi has stopped spamming processes.
Comment 13 Christian Mollekopf 2013-11-20 14:27:04 UTC
(In reply to comment #11)
> FWIW, another data point: 
> 
> Switching Kmail to offline and visiting the affected folder can stop the CPU
> churn. 
> 
> When i then keep it open in one tab and go back online, the CPU churn seems
> gone for the moment.

This problem is unrelated and handled separately in https://bugs.kde.org/show_bug.cgi?id=327865.
Comment 14 Christian Mollekopf 2013-11-20 14:28:40 UTC
(In reply to comment #13)
> 
> This problem is unrelated and handled separately in
> https://bugs.kde.org/show_bug.cgi?id=327865.

I meant to say; Georg's problem is unrelated, so you can ignore his debug information for this issue =)
Comment 15 Martin 2013-11-27 09:06:00 UTC
I had a trash and a wastebin folder. I then let Akonadi replace the local folders with a fresh set and now kmail works again. I am using KDE 4.11.3.
Comment 16 Martin Steigerwald 2015-05-04 14:10:06 UTC
Thiago, thank you for reporting the bugs and for all your comments and research.

Does this still happen with KDEPIM 4.14.6/7?
Comment 17 Thiago Macieira 2015-05-04 15:15:21 UTC
(In reply to Martin Steigerwald from comment #16)
> Thiago, thank you for reporting the bugs and for all your comments and
> research.
> 
> Does this still happen with KDEPIM 4.14.6/7?

I cannot reproduce this anymore.