| Summary: | Performance Regression: Folder synchronisation in Akonadi 16.08 (actually in any release, starting with KDE 4.14) very slow, compared to kMail from KDE 4.13.x | ||
|---|---|---|---|
| Product: | [Frameworks and Libraries] Akonadi | Reporter: | Gunter Ohrner <kdebugs> |
| Component: | IMAP resource | Assignee: | kdepim bugs <kdepim-bugs> |
| Status: | CONFIRMED --- | ||
| Severity: | major | CC: | adeptsmail, b-misc, b.buschinski, dav1dblunk3tt, dhenry, hatl, jacobopantoja, jerome.pouiller, justusranvier, kdepim-bugs, kdudka, leszek.lesner, madf, marco.clemencic, martin.steigerwald, Martin, mefyl, montel, plega-kde, tore, ua_bugz_kde, vkrause |
| Priority: | NOR | ||
| Version: | 5.14.1 | ||
| Target Milestone: | --- | ||
| Platform: | Neon | ||
| OS: | Linux | ||
| Latest Commit: | Version Fixed In: | ||
| Sentry Crash Report: | |||
| Attachments: |
excerpt of query log, clicking mails in open folders for some seconds
excerpt of query log for folder synchronisation after deleting mails CPU graph - KMail tests at 21:30-22:00 and 12:35-13:05 IOOPS graph - KMail tests at 21:30-22:00 and 12:35-13:05 I/O throughput graph - KMail tests at 21:30-22:00 and 12:35-13:05 Load graph - KMail tests at 21:30-22:00 and 12:35-13:05 Temperature graph - KMail tests at 21:30-22:00 and 12:35-13:05 |
||
|
Description
Gunter Ohrner
2014-08-26 10:57:57 UTC
Syncing takes over 70 seconds for the aforementioned folder, running on a Intel(R) Core(TM) i7-3540M CPU @ 3.00GHz with 12 GB RAM and an Samsung 840 EVO SSD. During that time, a mysqld process is maxing out one core. I've noticed that when akonadi is syncing some folders the status for the agent in "Akonadi Console" reports things such as: "Syncing folder 'hs' (1790600%)" Monitoring the MySQL server with MySQL Workbench shows 10MB/s of traffic and ~6000 queries per second being executed when these suspiciously high percentages are reported. I also noticed that the Kubuntu 4.14 package repository (at https://launchpad.net/~kubuntu-ppa/+archive/ubuntu/backports) contains akonadi 1.12.91 and not akonadi 1.13 but I've verified that the problem persists even when I upgraded to 1.13.0 Confirm this. I have around 20k messages in my mailbox, it takes minutes to sync it starting from 4.14.0. And after syncing it starts to sync everything again, and again, and again. I use SQLite as a backend. Here is a backtrace of one thread from akonadiserver that consumes 100% CPU: Thread 13 (Thread 0x7f4ad37fe700 (LWP 29182)): #0 0x00000030a36389b2 in sqlite3VdbeRecordCompare () from /usr/lib64/libsqlite3.so.0 #1 0x00000030a367cf16 in sqlite3VdbeExec () from /usr/lib64/libsqlite3.so.0 #2 0x00000030a367f04e in sqlite3_step () from /usr/lib64/libsqlite3.so.0 #3 0x00007f4b4309140c in sqlite3_blocking_step(sqlite3_stmt*) () from /usr/lib64/qt4/plugins/sqldrivers/libqsqlite3.so #4 0x00007f4b43093960 in QSQLiteResultPrivate::fetchNext(QVector<QVariant>&, int, bool) () from /usr/lib64/qt4/plugins/sqldrivers/libqsqlite3.so #5 0x00000030a64210af in QSqlCachedResult::cacheNext() () from /usr/lib64/qt4/libQtSql.so.4 #6 0x00000030a641100d in QSqlQuery::next() () from /usr/lib64/qt4/libQtSql.so.4 #7 0x0000000000493234 in Akonadi::Server::PimItem::extractResult(QSqlQuery&) () #8 0x000000000055e38d in Akonadi::Server::Merge::parseStream() () #9 0x0000000000539b98 in Akonadi::Server::Connection::slotNewData() () #10 0x000000309b5978fa in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /usr/lib64/qt4/libQtCore.so.4 #11 0x000000309b5978fa in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /usr/lib64/qt4/libQtCore.so.4 #12 0x000000309b0c93ed in QAbstractSocketPrivate::canReadNotification() () from /usr/lib64/qt4/libQtNetwork.so.4 #13 0x000000309b0d2abd in QReadNotifier::event(QEvent*) () from /usr/lib64/qt4/libQtNetwork.so.4 #14 0x000000309b58342c in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /usr/lib64/qt4/libQtCore.so.4 #15 0x000000309b5b1e27 in socketNotifierSourceDispatch(_GSource*, int (*)(void*), void*) () from /usr/lib64/qt4/libQtCore.so.4 #16 0x000000318a44a7ab in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0 #17 0x000000318a44a9c8 in g_main_context_iterate.isra () from /usr/lib64/libglib-2.0.so.0 #18 0x000000318a44aa8c in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0 #19 0x000000309b5b106e in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/qt4/libQtCore.so.4 #20 0x000000309b581f8f in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/qt4/libQtCore.so.4 #21 0x000000309b5822cd in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/qt4/libQtCore.so.4 #22 0x000000309b47bd1f in QThread::exec() () from /usr/lib64/qt4/libQtCore.so.4 #23 0x000000000046d93a in Akonadi::Server::ConnectionThread::run() () #24 0x000000309b47e49f in QThreadPrivate::start(void*) () from /usr/lib64/qt4/libQtCore.so.4 #25 0x00000032f6608333 in start_thread () from /lib64/libpthread.so.0 #26 0x00000032f62eb2cd in clone () from /lib64/libc.so.6 In the akonadiconsole debugger I see thousands of messages like these: akonadi_imap_resource_0 (0x7f4ac401c970) 2841 MERGE (REMOTEID SILENT) 53 0 (\MimeType[message/rfc822] "\\RemoteId[20720]" \SEEN) () akonadi_imap_resource_0 (0x7f4ac401c970) 2841 [UIDNEXT 5169 DATETIME "03-Sep-2014 17:51:07 +0000"] akonadi_imap_resource_0 (0x7f4ac401c970) 2841 OK Merge completed akonadi_imap_resource_0 (0x7f4ac401c970) 2842 MERGE (REMOTEID SILENT) 53 0 (\MimeType[message/rfc822] "\\RemoteId[20721]" \SEEN) () akonadi_imap_resource_0 (0x7f4ac401c970) 2842 [UIDNEXT 5170 DATETIME "03-Sep-2014 17:51:07 +0000"] akonadi_imap_resource_0 (0x7f4ac401c970) 2842 OK Merge completed akonadi_imap_resource_0 (0x7f4ac401c970) 2843 MERGE (REMOTEID SILENT) 53 0 (\MimeType[message/rfc822] "\\RemoteId[20722]" \SEEN) () akonadi_imap_resource_0 (0x7f4ac401c970) 2843 [UIDNEXT 5171 DATETIME "03-Sep-2014 17:51:07 +0000"] akonadi_imap_resource_0 (0x7f4ac401c970) 2843 OK Merge completed I can confirm this bug. After update from KDE 4.13 to 4.14 (akonadi server 1.12.91-0ubuntu1) updating of ~21k IMAP folder (MS Exchange) can take hours (instead of seconds in 4.13). But I can't see any load on mysql side *** This bug has been confirmed by popular vote. *** Here I am having the same behaviour. I use Arch Linux (both x86_64 and armv7h) with KMail 4.14.2 and Akonadi 1.13.0, with my company's email server. I can't confirm the behaviour in 4.13 since I am new to KMail. As an additional test, I have set up my GMail account with the same KMail installation, and GMail IMAP account is *NOT* suffering this problem. Sync works instantly and I can use it without a problem. I have not marked the "Download all messages for offline use" in GMail. Setting or unsetting this option in my company's account does not make a difference (except for the initial sync, which obviously takes a lot longer if the option is checked). If it may help, this is the list of commands available from my company's server (the ones marked with an * are not present in GMail's server commands) (I have preserved the order in which the commands appear in KMail's interface, in case it is relevant, although I don't think so): IMAP4REV1 NAMESPACE * AUTH=CRAM-MD5 * AUTH=LOGIN * AUTH=PLAIN IDLE COMPRESS=DEFLATE * ACL UNSELECT UIDPLUS QUOTA * BINARY XLIST And here are the commands available in Gmail's server (those marked with an * are not present in my company's server): IMAP4REV1 UNSELECT IDLE NAMESPACE QUOTA * ID XLIST * CHILDREN * X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE * ENABLE * MOVE * CONDSTORE * ESEARCH * UTF8=ACCEPT Best regards, JPantoja I'm having this problem too, on FC21/4.14.3. But I note that this bug is marked as 4.13. Shouldn't it be tagged 4.14 ? Laurent Montel changes this last September when he reassigned this bug from kmail2 to Akonadi. There is no version 4.14 for Akonadi here in Bugzilla, so that's probably why. Confirming.
I started from a clean slate (no ~/.config/akonadi, ~/.local/share/akonadi, ~/.kde/share/apps/{kmail,akonadi}*, or ~/.kde/share/config/akonadi* files/dirs present) and added my corporate IMAP account (Gigabit Ethernet all the way to the IMAP server). The default SQLite database type is used.
It sat there for about two hours saying «Syncing folder 'foldername'» for all the folders. The progress was tectonic for folders containing a large amount of messages (say 20k+). I cannot actually read any mail while this folder sync is in progress, even though the KMail UI lets me select folders in the tree list to the left, the main area (where I assume a list of the messages in the selected folder should appear) is stuck displaying a «Welcome to KMail 4.14.4» message.
The process "akonadiserver" was using quite a lot of CPU (~80%, according to "top"). Another process, "akonadi_baloo_indexer", was also figuring high in the list (CPU usage generally in the ~10-30% range), and was also frequently in the non-interruptible sleep state ("D").
I can see that the ~/.local/share/akonadi directory was filled with suspiciously large amounts of data. In particular, the file "akonadi.db" is 612MiB and the directory "file_db_data" is 291MiB. So clearly, it must have downloaded lots of stuff, in spite of the fact that the «Download all messages for offline use» setting is disabled.
Unfortunately, this issue is still current as of kMail 5.0.0 / Akonadi 15.08.0. :-( Any news on this issue? Create an email account, add 10k of mails -> akonadi/kmail stop working. To me, this is a major issue. Any web client can handle this many emails. Thunderbird works fine on the same account while kmail doesn't. I bumped the "version" to "GIT Master" hoping to increase visibility of this bug. I'm currently at Akonadi 15.08.3 and kMail 5.0.3, but could not select those versions from the drop down list... In KDE 4.13 and below, the IMAP sync performance was always great! The first appeared in KDE 4.14, has been around ever since and even seems to get worse rather than better. Now on Kubuntu 16.04 and still no improvement, the bug prevails: $ COLUMNS=200 dpkg -l | egrep 'akonadi|kmail' ii akonadi-backend-mysql 4:15.12.3-0ubuntu6 all MySQL storage backend for Akonadi ii akonadi-server 4:15.12.3-0ubuntu6 amd64 Akonadi PIM storage service ii akonadiconsole 4:15.12.3-0ubuntu1 amd64 management and debugging console for akonadi ii kmail 4:15.12.3-0ubuntu1 amd64 full featured graphical email client ii libakonadi-kde4 4:4.14.10-1ubuntu2 amd64 library for using the Akonadi PIM data server ii libakonadi-kmime4 4:4.14.10-1ubuntu2 amd64 Akonadi MIME handling library ii libakonadiprotocolinternals1 1.13.0-8ubuntu2 amd64 libraries for the Akonadi PIM storage service ii libkf5akonadiagentbase5:amd64 4:15.12.3-0ubuntu1 amd64 Akonadi agent base library ii libkf5akonadicalendar5:amd64 4:15.12.3-0ubuntu1 amd64 library providing calendar helpers for Akonadi items ii libkf5akonadicontact5:amd64 4:15.12.3-0ubuntu1 amd64 Akonadi contacts access library ii libkf5akonadicore-bin 4:15.12.3-0ubuntu1 amd64 Tools for Akonadi core library ii libkf5akonadicore5:amd64 4:15.12.3-0ubuntu1 amd64 Akonadi core library ii libkf5akonadimime5:amd64 4:15.12.3-0ubuntu1 amd64 Akonadi MIME handling library ii libkf5akonadinotes5:amd64 4:15.12.3-0ubuntu1 amd64 Akonadi notes access library ii libkf5akonadiprivate5 4:15.12.3-0ubuntu6 amd64 libraries for the Akonadi PIM storage service ii libkf5akonadisearchdebug5:amd64 4:15.12.3-0ubuntu1 amd64 Akonadi search debug library ii libkf5akonadisearchpim5:amd64 4:15.12.3-0ubuntu1 amd64 Akonadi search library ii libkf5akonadiwidgets5:amd64 4:15.12.3-0ubuntu1 amd64 Akonadi widgets library If I click on a larger folder, the following is logged by Akonadi - each line taking close to one second to appear (Remember: the Dovecot IMAP server runs on localhost in my case and the system uses a Samsung 840 EVO SSD), and "Receiving" is done twice, as you can see above, before kMail actually finishes: $ 39643 Received: 1000 In total: 1000 Wanted: 39643 Received: 1000 In total: 2000 Wanted: 39643 Received: 1000 In total: 3000 Wanted: 39643 Received: 1000 In total: 4000 Wanted: 39643 Received: 1000 In total: 5000 Wanted: 39643 Received: 1000 In total: 6000 Wanted: 39643 Received: 1000 In total: 7000 Wanted: 39643 Received: 1000 In total: 8000 Wanted: 39643 Received: 1000 In total: 9000 Wanted: 39643 Received: 1000 In total: 10000 Wanted: 39643 Received: 1000 In total: 11000 Wanted: 39643 Received: 1000 In total: 12000 Wanted: 39643 Received: 1000 In total: 13000 Wanted: 39643 Received: 1000 In total: 14000 Wanted: 39643 Received: 1000 In total: 15000 Wanted: 39643 Received: 1000 In total: 16000 Wanted: 39643 Received: 1000 In total: 17000 Wanted: 39643 Received: 1000 In total: 18000 Wanted: 39643 Received: 1000 In total: 19000 Wanted: 39643 Received: 1000 In total: 20000 Wanted: 39643 Received: 1000 In total: 21000 Wanted: 39643 Received: 1000 In total: 22000 Wanted: 39643 Received: 1000 In total: 23000 Wanted: 39643 Received: 1000 In total: 24000 Wanted: 39643 Received: 1000 In total: 25000 Wanted: 39643 Received: 1000 In total: 26000 Wanted: 39643 Received: 1000 In total: 27000 Wanted: 39643 Received: 1000 In total: 28000 Wanted: 39643 Received: 1000 In total: 29000 Wanted: 39643 Received: 1000 In total: 30000 Wanted: 39643 Received: 1000 In total: 31000 Wanted: 39643 Received: 1000 In total: 32000 Wanted: 39643 Received: 1000 In total: 33000 Wanted: 39643 Received: 1000 In total: 34000 Wanted: 39643 Received: 1000 In total: 35000 Wanted: 39643 Received: 1000 In total: 36000 Wanted: 39643 Received: 1000 In total: 37000 Wanted: 39643 Received: 1000 In total: 38000 Wanted: 39643 Received: 1000 In total: 39000 Wanted: 39643 Received: 643 In total: 39643 Wanted: 39643 finished 39643 Received: 1000 In total: 1000 Wanted: 39643 Received: 1000 In total: 2000 Wanted: 39643 Received: 1000 In total: 3000 Wanted: 39643 Received: 1000 In total: 4000 Wanted: 39643 Received: 1000 In total: 5000 Wanted: 39643 Received: 1000 In total: 6000 Wanted: 39643 Received: 1000 In total: 7000 Wanted: 39643 Received: 1000 In total: 8000 Wanted: 39643 Received: 1000 In total: 9000 Wanted: 39643 Received: 1000 In total: 10000 Wanted: 39643 Received: 1000 In total: 11000 Wanted: 39643 Received: 1000 In total: 12000 Wanted: 39643 Received: 1000 In total: 13000 Wanted: 39643 Received: 1000 In total: 14000 Wanted: 39643 Received: 1000 In total: 15000 Wanted: 39643 Received: 1000 In total: 16000 Wanted: 39643 Received: 1000 In total: 17000 Wanted: 39643 Received: 1000 In total: 18000 Wanted: 39643 Received: 1000 In total: 19000 Wanted: 39643 Received: 1000 In total: 20000 Wanted: 39643 Received: 1000 In total: 21000 Wanted: 39643 Received: 1000 In total: 22000 Wanted: 39643 Received: 1000 In total: 23000 Wanted: 39643 Received: 1000 In total: 24000 Wanted: 39643 Received: 1000 In total: 25000 Wanted: 39643 Received: 1000 In total: 26000 Wanted: 39643 Received: 1000 In total: 27000 Wanted: 39643 Received: 1000 In total: 28000 Wanted: 39643 Received: 1000 In total: 29000 Wanted: 39643 Received: 1000 In total: 30000 Wanted: 39643 Received: 1000 In total: 31000 Wanted: 39643 Received: 1000 In total: 32000 Wanted: 39643 Received: 1000 In total: 33000 Wanted: 39643 Received: 1000 In total: 34000 Wanted: 39643 Received: 1000 In total: 35000 Wanted: 39643 Received: 1000 In total: 36000 Wanted: 39643 Received: 1000 In total: 37000 Wanted: 39643 Received: 1000 In total: 38000 Wanted: 39643 Received: 1000 In total: 39000 Wanted: 39643 Received: 643 In total: 39643 Wanted: 39643 finished While this is running, there is heavy disk activity and the Akonadi MySQL server process is busy using CPU and IO. See the vmstat logs, if you look at the BO column, you can clearly see me switching to a small IMAP folder (183 mails only, and it caused disk activity for 6 seconds, as you can see!) and back to the large one: # vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd frei buff Cache si so bi bo in cs US SY ID WA st 0 0 0 7473808 296960 1870012 0 0 136 40 111 2568 5 2 93 0 0 0 0 0 7473684 296960 1870020 0 0 0 0 482 4567 3 2 96 0 0 1 0 0 7473684 296960 1870040 0 0 0 0 400 3761 2 2 96 0 0 1 0 0 7473684 296960 1869756 0 0 0 0 390 3846 3 2 95 0 0 0 0 0 7474004 296960 1869308 0 0 0 0 488 4283 2 2 96 0 0 0 0 0 7473312 296960 1870120 0 0 0 0 724 6297 9 2 89 1 0 1 0 0 7473436 296960 1870128 0 0 0 0 457 4284 5 3 92 0 0 0 0 0 7473436 297000 1870068 0 0 0 88 670 6856 10 2 87 0 0 0 0 0 7473312 297016 1870152 0 0 0 92 511 4315 5 2 93 0 0 0 0 0 7473332 297048 1869876 0 0 0 408 498 4833 5 4 90 0 0 0 0 0 7473436 297064 1869716 0 0 0 32 478 4262 5 3 92 0 0 0 0 0 7473312 297080 1869720 0 0 0 32 443 3748 4 3 93 0 0 1 0 0 7473312 297088 1869724 0 0 0 52 449 3879 5 3 91 0 0 0 0 0 7473312 297088 1869752 0 0 0 0 432 4032 4 3 93 0 0 0 0 0 7473312 297088 1869764 0 0 0 0 449 4399 5 3 92 0 0 0 0 0 7473312 297088 1869780 0 0 0 0 477 4537 5 2 93 0 0 0 0 0 7473312 297088 1869792 0 0 0 0 449 4340 4 3 93 0 0 0 0 0 7474292 297088 1869804 0 0 0 40 459 3839 6 2 92 0 0 0 0 0 7474180 297088 1869816 0 0 0 0 467 3853 4 3 92 0 0 2 0 0 7473740 297112 1870680 0 0 0 84 796 24581 26 4 69 0 0 1 0 0 7472808 297240 1870924 0 0 0 560 1067 74335 32 5 61 2 0 1 0 0 7472660 297352 1870784 0 0 0 640 916 82622 23 6 70 1 0 1 0 0 7472536 297480 1870764 0 0 0 656 919 85087 22 5 71 2 0 1 0 0 7472288 297596 1870912 0 0 0 656 898 84111 23 5 70 2 0 1 0 0 7472504 297700 1871096 0 0 0 548 912 86671 24 5 69 2 0 1 0 0 7472272 297812 1871228 0 0 0 600 916 83593 23 6 70 2 0 1 0 0 7472256 297940 1871392 0 0 0 632 927 84065 24 4 70 2 0 4 0 0 7472396 298052 1871524 0 0 0 524 911 84327 24 5 70 2 0 2 0 0 7471792 298156 1871652 0 0 0 512 908 86793 24 5 69 2 0 3 0 0 7471544 298276 1871792 0 0 0 508 927 81204 25 4 70 2 0 1 1 0 7471000 298392 1871932 0 0 0 564 899 83962 23 8 68 1 0 2 0 0 7471140 298488 1872128 0 0 0 480 864 81261 23 5 71 1 0 1 0 0 7470704 298600 1872344 0 0 0 524 924 84347 23 6 70 2 0 1 0 0 7470752 298704 1872440 0 0 0 496 863 80989 23 5 71 2 0 1 0 0 7470428 298824 1872528 0 0 0 556 961 86290 23 5 71 1 0 2 0 0 7470396 298928 1872676 0 0 0 508 898 79754 22 6 70 2 0 1 0 0 7470040 299056 1872816 0 0 0 532 884 83506 23 4 71 2 0 1 0 0 7469776 299160 1872584 0 0 0 460 875 81897 24 5 70 1 0 1 0 0 7469372 299264 1872504 0 0 0 476 910 85993 23 7 69 1 0 1 0 0 7469264 299376 1872620 0 0 0 500 935 86358 24 5 70 1 0 1 0 0 7468644 299480 1873308 0 0 0 512 1141 81078 26 6 66 2 0 1 0 0 7468288 299568 1873364 0 0 0 464 1035 82384 24 5 69 2 0 1 0 0 7468024 299680 1873232 0 0 0 500 970 85633 24 6 70 1 0 1 0 0 7467196 299792 1873740 0 0 0 484 981 79406 25 6 67 2 0 1 0 0 7468456 299896 1872344 0 0 0 512 1029 81500 24 5 70 2 0 1 0 0 7468428 300016 1872432 0 0 0 536 937 85018 23 7 69 1 0 1 0 0 7468112 300128 1872496 0 0 0 596 962 89481 24 5 70 1 0 3 0 0 7468212 300240 1872808 0 0 0 592 950 85107 23 7 69 1 0 1 0 0 7468212 300352 1872940 0 0 0 568 1025 78885 24 6 69 2 0 1 0 0 7468056 300456 1873100 0 0 0 484 937 82608 24 7 68 1 0 1 0 0 7467420 300584 1873168 0 0 0 576 1057 81568 24 5 70 2 0 2 0 0 7467388 300688 1873344 0 0 0 532 989 80096 25 5 68 2 0 2 0 0 7467204 300800 1873452 0 0 0 560 948 84102 24 5 69 2 0 1 0 0 7467064 300904 1873600 0 0 0 540 937 80999 24 6 68 2 0 1 0 0 7466924 301016 1873756 0 0 0 560 975 84549 24 6 68 2 0 1 0 0 7466972 301128 1873852 0 0 0 536 974 87379 23 6 70 2 0 1 0 0 7469316 301224 1874028 0 0 0 516 851 60053 25 4 69 2 0 1 0 0 7467188 301328 1874060 0 0 0 500 943 59489 29 5 65 1 0 1 0 0 7465604 301424 1874192 0 0 0 464 883 69000 25 4 69 1 0 1 0 0 7465452 301544 1874768 0 0 0 592 1000 84488 24 5 69 2 0 2 0 0 7464972 301656 1875452 0 0 0 608 1059 84753 23 6 70 2 0 1 0 0 7464548 301768 1875276 0 0 0 536 923 79290 24 6 69 2 0 1 0 0 7464848 301872 1874660 0 0 0 500 1569 85382 25 4 70 2 0 2 0 0 7464568 301984 1874832 0 0 0 560 974 84714 23 5 70 2 0 1 0 0 7464708 302112 1874940 0 0 0 628 990 81891 25 4 70 1 0 1 0 0 7464136 302216 1875060 0 0 0 520 899 82147 24 5 69 1 0 1 0 0 7464120 302328 1875204 0 0 0 572 973 84707 24 5 69 2 0 2 0 0 7464244 302432 1875352 0 0 0 512 965 84649 24 5 69 1 0 1 0 0 7463700 302544 1875552 0 0 0 632 989 85554 24 6 69 2 0 1 0 0 7463560 302656 1875684 0 0 0 588 932 84810 23 7 69 2 0 1 0 0 7463360 302768 1875820 0 0 0 588 995 83968 23 6 69 2 0 1 0 0 7463252 302896 1875928 0 0 0 628 956 82870 24 6 69 2 0 1 0 0 7463004 303008 1876072 0 0 0 556 969 86364 24 6 69 1 0 1 0 0 7462584 303120 1876260 0 0 0 556 983 84060 22 6 70 2 0 1 0 0 7462740 303224 1876128 0 0 0 524 904 82146 24 6 69 1 0 1 0 0 7462508 303344 1876144 0 0 0 592 991 90109 24 6 69 1 0 1 0 0 7462088 303456 1876336 0 0 0 536 914 84076 24 6 70 1 0 1 0 0 7461872 303568 1876440 0 0 0 532 988 83883 24 6 68 2 0 1 0 0 7461764 303688 1876556 0 0 0 560 948 79729 23 5 70 2 0 1 0 0 7461500 303800 1876728 0 0 0 596 910 79917 22 6 69 2 0 1 0 0 7461000 303920 1877188 0 0 0 560 969 87297 25 4 69 2 0 1 0 0 7461128 304032 1877244 0 0 0 528 901 86156 25 4 69 2 0 0 1 0 7460304 304136 1877404 0 0 0 524 910 86843 24 5 70 1 0 1 0 0 7460460 304264 1877588 0 0 0 696 954 83575 23 6 69 2 0 1 0 0 7460104 304376 1877724 0 0 0 548 951 87647 23 6 70 1 0 3 0 0 7459888 304496 1877820 0 0 0 536 993 77743 23 5 70 2 0 2 0 0 7459624 304608 1877980 0 0 0 580 866 78288 23 6 70 2 0 5 0 0 7459080 304712 1878176 0 0 0 500 1001 83174 24 6 68 2 0 1 0 0 7458632 304824 1878280 0 0 0 536 947 82526 24 7 68 2 0 1 0 0 7458260 304936 1878404 0 0 0 844 904 72641 25 4 69 2 0 1 1 0 7457888 305040 1878552 0 0 0 444 956 78351 23 5 69 2 0 1 0 0 7457780 305144 1878676 0 0 0 432 994 84258 25 5 69 2 0 1 0 0 7457532 305272 1878804 0 0 0 524 910 82728 25 5 69 2 0 1 0 0 7457160 305384 1878952 0 0 0 500 937 86766 24 5 70 1 0 0 0 0 7456212 305456 1879184 0 0 0 316 891 42958 28 3 68 1 0 0 0 0 7456044 305472 1879208 0 0 0 120 411 4181 3 2 95 0 0 0 0 0 7455424 305520 1879212 0 0 0 6488 483 4637 3 3 91 2 0 0 0 0 7455672 305568 1879220 0 0 0 6520 472 5005 2 3 93 2 0 0 0 0 7456168 305616 1879248 0 0 0 6484 557 4733 4 4 91 2 0 1 0 0 7456044 305648 1879260 0 0 0 1464 450 3945 4 2 93 1 0 1 0 0 7456044 305664 1879264 0 0 0 32 420 4425 2 1 97 0 0 0 0 0 7455796 305664 1879300 0 0 0 0 404 4297 3 2 96 0 0 0 0 0 7455796 305664 1879316 0 0 0 0 417 4557 3 1 96 0 0 0 0 0 7455796 305664 1879328 0 0 0 0 381 4425 3 1 96 0 0 0 0 0 7455796 305664 1879344 0 0 0 0 446 4146 3 3 95 0 0 0 0 0 7455796 305664 1879356 0 0 0 0 485 4377 2 1 97 0 0 ^C Just upgraded to Fedora 24 and tried a fresh setup of KMail with two IMAP accounts. Versions: akonadi-1.13.0-102.fc24.x86_64 kmail-15.12.3-1.fc24.x86_64 No improvement is noticeable. Synchronising folders takes hours, and my laptop is completely swamped - everything gets extremely slow. The mysqld and akonadi_indexing_agent processes appears to be the biggest resource hogs. Neither of the accounts have «Download all messages for offline use» selected. In spite of this, after the initial sync has completed, ~/.local/share/akonadi/ contains almost 7 GiB of data. The largest contributors to that are: file_db_data/* ~3 GiB search_db/email/termlist.DB ~1 GiB search_db/email/postlist.DB ~1 GiB db_data/akonadi/parttable.ibd ~1 GiB KMail is essentially unusable for me due to this issue. Tore, two things: 1) AFAIK KMail 15.12 neeeds Akonadi 15.12 or at least any *other* Qt 5 based Akonadi. Akonadi 1.13 to my knowledge is Qt 4 based. So I wonder why it works at all on your system. It might be that in Fedora the Qt5 based Akonadi still has a 1.13 version number, but I wouldn´t know why. 2) With Akonadi 15.12 there should be noticable performance improvements. Yet I recommend testing with KMail + Akonadi 16.04. I have a larger setup: ~/.local/share/akonadi> LANG=C du -sch * | sort -rh | head -4 6.9G total 5.1G db_data 1.7G search_db 117M file_db_data yet, with Akonadi 15.12 I think a file leak with Akonadi´s file_db_data has been fixed. You still seem to have it. On any account: Akonadi 1.13 is too old. Please test more recent versions. They old ones will not be fixed. I am running with 16.04 already, but it was already way faster with 15.12. I'm now on KDE Neon with KDE Frameworks 5.25 and Applications 16.08 (kMail 5.3.0), see below for a detailed package list. It did not get any better, just opening my "sent" folder just took 87 seconds (!) during which kMail synced the folder twice. I guess noone want's to have forced breaks of one-and-a-half minute while working with his email client... As written above, kMail as well as the IMAP server are running locally, the system disk is an SSD and MySQL just get's hammered by Akonadi during that time. Whatever was changed between KDE 4.13 and 4.14 in this respect concerning kMail, Akonadi, IMAP and their cooperation just was no good idea and no proper fix for whatever it was intended to fix... $ COLUMNS=200 dpkg -l | egrep 'akonadi|kmail' ii akonadi-backend-mysql 4:16.08.0+p16.04+git2016082 all MySQL storage backend for Akonadi ii akonadi-contacts-data 4:16.08.0+p16.04+git2016082 all akonadi-contact data files ii akonadi-mime-data 4:16.08.0+p16.04+git2016082 all akonadi-mime - data files ii akonadi-server 4:16.08.0+p16.04+git2016082 amd64 Akonadi PIM storage service ii akonadiconsole 4:16.08.0+p16.04+git2016082 amd64 management and debugging console for akonadi ii kmail 4:16.08.0+p16.04+git2016082 amd64 full featured graphical email client ii libakonadi-kde4 4:4.14.10-1ubuntu2 amd64 library for using the Akonadi PIM data server ii libakonadi-kmime4 4:4.14.10-1ubuntu2 amd64 Akonadi MIME handling library ii libakonadiprotocolinternals1 1.13.0-8ubuntu2 amd64 libraries for the Akonadi PIM storage service ii libkf5akonadiagentbase5:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi agent base library ii libkf5akonadicalendar5:amd64 4:16.08.0+p16.04+git2016082 amd64 library providing calendar helpers for Akonadi items ii libkf5akonadicontact5:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi contacts access library ii libkf5akonadicore5:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi core library ii libkf5akonadimime5:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi MIME handling library ii libkf5akonadinotes5:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi notes access library ii libkf5akonadiprivate5:amd64 4:16.08.0+p16.04+git2016082 amd64 libraries for the Akonadi PIM storage service ii libkf5akonadisearch-bin 4:16.08.0+p16.04+git2016082 amd64 Akonadi search library - runtime binaries ii libkf5akonadisearch-data 4:16.08.0+p16.04+git2016082 all Akonadi search library - data files ii libkf5akonadisearch-plugins:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi search library - runtime plugins ii libkf5akonadisearchcore5:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi search core library ii libkf5akonadisearchdebug5:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi search debug library ii libkf5akonadisearchpim5:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi search library ii libkf5akonadisearchxapian5:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi search xapian library ii libkf5akonadiwidgets5:amd64 4:16.08.0+p16.04+git2016082 amd64 Akonadi widgets library In fact, these versions of kMail and Akonadi feel much more "snappy" while working with mails - just the most serious performance problem was not solved and did not improve, unfortunately. Gunter, how big is your sent folder? How many mails are in there. And, I am not sure whether it has been mentioned already, but my better experience may (!) be related to some manual MySQL tuning. Especially: # memory buffer InnoDB uses to cache data and indexes of its tables (default:128M) # Larger values means less I/O innodb_buffer_pool_size=1024M I have no scientific measurement, but I think I noticed slowdowns as after an upgrade of Akonadi its replaced that file with a newer one. I made a bug report about this quite some time ago, but the a dev declined my suggestion to raise the ridiculously low 64M I think default to something higher with the argument the configuration came from I think Kristian Köhntopp so it must be right. Also, and I get that, there are smaller mail setups as well. From what I understand about MySQL performance tuning, the default value set by Akonadi MySQL standard config is way to low for larger setups. mysqltuner.pl agrees there as well. I suggest you run mysqltuner against the socket of the Akonadi database and then review the suggestions it makes. I am pretty sure it will tell you to raise the InnoDB buffer pool size considerably. Currently slightly more than 41000 mails. It was somewhat less two years ago with kMail from KDE 4.13, but not significantly, and the performance dropped all of a sudden with the update. Thunderbird is quite snappy with a folder that size, but I like kMail more - although the extremely bad performance is definitely getting on my nerves. I tried increasing the innodb buffer size, but this did not result in any significant improvement. My impression is that there's at least one separate query issued for each mail, maybe more. This would also match the observations stated above by another user. Probably a lot less queries were issued in kMail 4.13, which also was already based on Akonadi. (KDE 4.7 was the first release that included the Akonadi-based kMail 2.) Wow, it's much worse than one query per mail... Activating the MySQL query log and just opening my Sent folder leaves me with a log file which is - 190 MByte (!) in size... Admittedly there are also some timestamps in there, but at least half of that is SQL... Apparently, kMail runs the following set of queries *twice* for *each* single mail: 2016-08-26T22:57:43.864393Z 88 Execute SELECT PimItemTable.id, PimItemTable.rev, PimItemTable.remoteId, PimItemTable.remoteRevision, PimItemTable.gid, PimItemTable.collectionId, PimItemTable.mimeTypeId, PimItemTable.datetime, PimItemTable.atime, PimItemTable.dirty, PimItemTable.size FROM PimItemTable WHERE ( collectionId = 602 AND remoteId = '24314' ) 2016-08-26T22:57:43.864535Z 88 Reset stmt 2016-08-26T22:57:43.864561Z 88 Execute SELECT FlagTable.id, FlagTable.name FROM FlagTable INNER JOIN PimItemFlagRelation ON ( PimItemFlagRelation.Flag_id = FlagTable.id ) WHERE ( PimItemFlagRelation.PimItem_id = 653279 ) 2016-08-26T22:57:43.864667Z 88 Reset stmt 2016-08-26T22:57:43.864693Z 88 Execute SELECT FlagTable.id, FlagTable.name FROM FlagTable INNER JOIN PimItemFlagRelation ON ( PimItemFlagRelation.Flag_id = FlagTable.id ) WHERE ( PimItemFlagRelation.PimItem_id = 653279 ) 2016-08-26T22:57:43.864793Z 88 Reset stmt 2016-08-26T22:57:43.864819Z 88 Execute SELECT PartTable.id, PartTable.pimItemId, PartTable.partTypeId, PartTable.data, PartTable.datasize, PartTable.version, PartTable.external FROM PartTable WHERE ( pimItemId = 653279 ) 2016-08-26T22:57:43.864946Z 88 Reset stmt 2016-08-26T22:57:43.864980Z 88 Execute UPDATE PimItemTable SET rev = 1, remoteId = '24314', remoteRevision = NULL, gid = '<201101172158.07416.XxXxXx@example.com>', collectionId = 602, mimeTypeId = 2, datetime = '2016-08-26 22:57:43', atime = '2016-08-27 00:57:43', dirty = 0, size = 967 WHERE ( id = 653279 ) 2016-08-26T22:57:43.865197Z 88 Reset stmt That sums up to 5 queries times 40,000 mails times 2 = 400,000 queries just to *open* a single folder. Whoa. In addition to the sheer number of queries one notes that the following identical query (with same parameters!) is executed twice in this batch, and is even a *join*: SELECT FlagTable.id, FlagTable.name FROM FlagTable INNER JOIN PimItemFlagRelation ON ( PimItemFlagRelation.Flag_id = FlagTable.id ) WHERE ( PimItemFlagRelation.PimItem_id = 653279 ) Hopefully, the MySQL query cache is able to optimize the second execution away but you still lose some time to send the query and retrieve the response. So we could already save 80,000 join executions just by removing the second redundant call and re-using the result the first call returned. That would still leave us with 320,000 queries however. If kMail would scan the folder only once, we would be down to 160,000 queries - still a lot for 40,000 mails, but already about 2.5 times faster than at the moment. All queries properly use indices, but given the sheer number of simple statements and a necessary join, I'm not surprised it's *that* slow. All I know about relational databases would suggest it would be much better to use larger batch-selects and updates instead of tons of individual key lookups. This kind of workload would be more suited for KVP/hash based data stores - and preferrably an in-memory cache, at least for read accesses... Gunther, I am not sure what is going on on your system, cause when I access a 76000 mails folder it takes about 8 seconds to actually see KMail showing the first mails and starting the threading operation. Or another example: About 10 seconds to access debian-devel-changes folder with slightly more than 133000 messages, this time flat view without threading. I don´t see why this couldn´t even be faster… but its quite a bit faster from what you report. This all after akonadictl stop, echo 3 > /proc/sys/vm/drop_caches, akonadictl start, so without any in memory caching on a ThinkPad T520 Sandybridge i5, 2.5 to max 3.2 GHz, machine with Dual SSD BTRFS RAID 1. I use MariaDB 10.0.26, but I don´t think it makes much of a difference to MySQL. mysqld load a mere 60-80% for some of this time. I did not create a query log. How did you create it, I may create one so we can compare whether on my setup Akonadi does something different. Then after the MySQL activity KMail was at about 80% on one core for a few seconds. Hmmm, okay, now I notice there is an important difference. I tested this against a local maildir resource, not with IMAP. I think… I bet that may cause a difference in MySQL load. Do you use disconnected IMAP or regular IMAP? I only have Exchange IMAP at work and it doesn´t make sense to test with it, as with Exchange IMAP implementation its not really possible to access large folders via IMAP consistently, not even with Trojita. From what I remember when I still had a kind of 1-2 week IMAP (maildir on server cleaned by find -delete) access to folders have been quite snappy. I am trying to reproduce it now by recreating the IMAP account. Akonadi still has to synchronize on of the larger folders, so takes a time till I can test. I'm using regular IMAP but with a local IMAP server. Don't ask me why I'm doing it this way, but apparently it's a good setup to detect performance bottlenecks in the IMAP codepaths. ;) Laurent Montel already assigned this report to "IMAP Resource" nearly two years ago, so if he was right, you'll probably not be able to reproduce it with a local Maildir resource. However if a Maildir resource scales much better, this would be an indication that it's actually a "suboptimal behaviour" (i.e. "bug" ;) of the IMAP Resource and not "simply unavoidable" if you use Akonadi. If I open a folder, the list of mails is basically shown immediately - but I cannot display mails. One or two previously cached mails might display, but as soon as I select an uncached mail, all mails become inaccessible until the "Syncing folder 'ABCD'" operation finishes, which takes close to a minute to complete and in most cases is executed twice before the folder becomes operational. (Thus the 90 seconds delay during which the folder is not usable and which happens every time I open the folder.) That's how I temporarily enabled the query log: - connect to the MySQL server using the mysql monitor (MariaDB will probably work the same) - SET GLOBAL log_output = "FILE"; - SET GLOBAL general_log_file = "/var/tmp/querylog.sql"; - SET GLOBAL general_log = 'ON'; Okay, I seriously don´t get why its so different for you: One folder with about 10000 mails about one second with cold caches and one with more than 24000 mails about 2-3 seconds with cold caches. This time on Dovecot based IMAP. Regular IMAP, not disconnected. You do have double the amount of mails than the larger one of my folders, Gunther, but opening your sent folder takes about 30-40 times longer than on my system. So there is hope to fix your issue, I think, once it is clear what is causing the high load on your system. I wonder how to find whats different tough. Care to share more details of your setup? Also… do you have any search folders that target mails of your sent folder? My IMAP account setup is very basic. I just pointed it to my Dovecot IMAP, and then let it autodetect the encryption. Thats it. No customizations so far. So what you could try would be: Use a *new* user, recreate IMAP account from scratch. And try then. Also are there any errors Akonadi shows in ~/.xsession-errors or Akonadi error log. Are there any errors in MySQL log? Okay, just reading your mid air collision comment as well: So you have local IMAP, that should be even faster. I also tried clicking folder and then clicking mails. Its just instant with warm caches, i.e. folder already opened before. Okay, and even with cold caches, after the initial delay of 1-3 seconds and maybe an added second its instant. I click mail and its done. IMAP resource doesn´t synchronize the folder on my setup when clicking mails in a folder that it synchronized initially after starting it. What I know is that it does when deleting mail and this can cause quite some slow downs, but not when reading mails. So are you doing any other changes to the mail other than KMail setting read flag on them? I can reproduce huge delays when entering inbox for example and then hit delete 10-20 times. But but its not that MySQL heavy. It is just that Akonadi IMAP for some reason resynchronizes folder content after deleting a mail. So I think that is a different issue. Okay, will post about query logs in next comment. Created attachment 100800 [details]
excerpt of query log, clicking mails in open folders for some seconds
This is what is almost instant on my machine.
Created attachment 100801 [details]
excerpt of query log for folder synchronisation after deleting mails
I sometimes need to delete quite a bunch of mails to trigger a folder synchronisation that takes long. But I made it. I made 2 second excerpt of the log, and I agree with you, Gunther: The amount of queries made in this short time is excessive.
There seem to be three issues in what you observe: 1) There is folder synchronisation when you enter a folder and click mails in there to read them. I don´t have that and for me this operation is, except for few second initial delay to open the folder (any eventual threading work of KMail not counted), instant or almost instant. That may be something for a different bug report to keep that one to the performance on folder synchronisation itself. For Maildir I did a bug report about IMHO superfluous synchronisation (bug #334209). For any excessive folder synchronisation reading mails, if you really see this, this would be worth a different bug. But maybe what you see is the initial folder synchronisation on after you switched to the folder that you see there. Does it really synchronize the folder again and again when reading mails in it? Then I would report this as a different bug (if not already done by someone). 2) As per the title you choose folder synchronisation itself taking a lot of time. And well I can see this here as well. Its easily taking up half a minute or more. And generating a lot of queries. Its not creating much load on my MySQL, but that may be due to creating a lot of load to my IMAP server. I see both mysqld at about 20% of one core and akonadi_imap_resource at about 30% of one core. Now your IMAP is local, and thus will be faster to access and that I think easily explains the higher MySQL load you see. Actually synchroniszing that 24500 mail inbox folder took as much time as typing all of this second point. Easily more than a minute. 3) During folder synchronisation Akonadi blocks out other operations like deleting or viewing mails. I reported this as: [Akonadi] [Bug 334206] New: While maildir resources synchronrizes a folder KMail blocks on switching to a different folder and will change the title of the bug IMAP resources as well now. I bet it may affect *any* resources. I suggest to keep this report about the actual folder synchronisation speed from now on for clarity. Okay, nope changing the title on the bug report I mentioned on the third issue doesn´t make sense. I don´t see this bug anymore since performance improvements made by Dan and Millian. It makes sense to split out the IMAP case, i.e. report "During folder synchronisation Akonadi blocks out other operations like deleting or viewing mails" for the IMAP case if not already done. I just created Bug 367892 - During folder synchronisation Akonadi blocks out other operations like deleting or viewing mails for blocking out other operations during folder synchronisation. Gunther, if you feel like adding anything there, please do so. Please keep this bug to the actual synchronisation performance from now on as per the title of the bug. (In reply to Martin Steigerwald from comment #28) > 1) There is folder synchronisation when you enter a folder and click mails > in there to read them. No, that's a misunderstanding. Switching between emails inside a single folder is fast. Switching between folders is sloooooow... :) > 2) As per the title you choose folder synchronisation itself taking a lot of > time. And well I can see this here as well. Its easily taking up half a > minute or more. And generating a lot of queries. Its not creating much load That's the issue I'm having trouble with. I'm not sure if this is the issue I have but since switching from kubuntu 14.04 to 16.04 kmail has become extremely slow retrieving emails - it seems to cache a couple but beyond that seems to be constantly going back to the imap server to grab things it has grabbed just two minutes ago - I'm now spending far more time looking at a blank pane show "retrieving folder contents" than actually doing anything. IIRC in the previous version I might have a small delay while a big email was downloaded for the first time but everything was pretty snappy after that. As an example - I'm currently trying to view an email I looked at an hour ago - it still hasn't been shown and I started this bug report <after> I started waiting. Hold on - it's arrived. David, to me this appears an issue that is not directly related to folder synchronisation speed although slow folder synchronisation speeds makes it worse. I think it is a mixture of Bug 367892 - During folder synchronisation Akonadi blocks out other operations like deleting or viewing mails and probably needless folder synchronisations. I reported IMHO needless folder synchronisations for maildir already in: [Akonadi] [Bug 334209] New: synchronizes folder contents with filesystem on switching folders needlessly For IMAP I reported something similar in Bug 367892 mentioned above. Hitting delete key to delete messages from IMAP folder here easily triggered a folder synchronisation (what for?). I think this is also a candidate for a separate bug report. For IMAP servers these folder synchronisations are likely required as other clients can update the IMAP account behind Akonadi. Yet, KMail should not block on these. Please try to separate different issues. I bet what would help you most would be that Akonadi does not block on folder synchronisations. So I suggest you add your finding there. Unfortunately there are quite a bunch of performance related issues still left in Akonadi. (In reply to Martin Steigerwald from comment #33) […] > and probably needless folder synchronisations. I reported IMHO needless > folder synchronisations for maildir already in: > > [Akonadi] [Bug 334209] New: synchronizes folder contents with filesystem on > switching folders needlessly But this one I reported as solved. It still synchronizes needlessly when filtering message into local maildir folders – I am not sure whether I reported that. Its really difficult to tell all those issues apart. (In reply to Gunter Ohrner from comment #31) > (In reply to Martin Steigerwald from comment #28) > > 1) There is folder synchronisation when you enter a folder and click mails > > in there to read them. > > No, that's a misunderstanding. Switching between emails inside a single > folder is fast. > > Switching between folders is sloooooow... :) Is this still the case? > > 2) As per the title you choose folder synchronisation itself taking a lot of > > time. And well I can see this here as well. Its easily taking up half a > > minute or more. And generating a lot of queries. Its not creating much load > > That's the issue I'm having trouble with. And well I think that are *two* issues. 1) Probably needless folder synchronisations on switching folders on IMAP accounts. (I reported this for maildir accounts but there this seems to be fixed). 2) Slow folder synchronisation. I suggest to keep this bug solely for folder synchronisation speed. So if you still see needless folder synchronisations on switching folders on IMAP accounts, I suggest you report another bug about it and link it here. Some information that could help: I've been hit by the horrible read performance bug for years - i.e. opening a folder would take minutes of processing, with akonadi apparently going through all the mails, as if it was the first folder synchronization ever. I had some hunch around 6 months ago and switched my IMAP server from courier-imap to dovecot, and problem solved: folders now synchronize in a matter of milliseconds. I suppose other servers like Cyrus would do too. I can't be sure who's at fault here, but I'm positive that this setup (kmail/akonadi/courier-imap) did work perfectly for years before the apparition of this regression, so some digging around here could be useful. Or in any case, if you have that bug and are using courier-imap, you can try changing the IMAP server as a workaround and finally find some peace of minds wrt to your emails. (In reply to Martin Steigerwald from comment #35) > > Switching between folders is sloooooow... :) > Is this still the case? Absolutely. Unfortunately. (In reply to mefyl from comment #36) > I had some hunch around 6 months ago and switched my IMAP server from > courier-imap to dovecot, and problem solved: folders now synchronize in a > matter of milliseconds. I suppose other servers like Cyrus would do too. I I'm also using (a locally running) Dovecot. As stated and explained above in detail, it's pretty slow and hammering Akonadi's MySQL DB while waiting. Still current as of Akonadi 17.04.2, no improvements whatsoever. This report soon celebrates its third birthday. Following the release of Fedora 26, I decided to test KMail again and compare it with Mozilla Thunderbird, both using a completely empty UNIX account. tl;dr: Compared to Thunderbird, KMail/Akonadi ends up using a absolutely ridiculous amount of time, CPU, disk space, and network traffic to connect to an IMAP account. Main observations: - KMail needed ~30 minutes to perform the initial sync of all folders, while Thunderbird used <5 minutes*: - The system was completely swamped by Akonadi and MySQL processes. Thunderbird was very modest. - KMail exchanged 1.19 GB of data with the IMAP server. Thunderbird exchanged 128 MB. - Afterwards, the home directory of the KMail test account contained 3842 MB of data, Thunderbird's contained 128 MB. * Unlike KMail, Thunderbird only synced INBOX automatically and its UI usable within seconds. The <5 minutes included manually opening and syncing every single IMAP folder. More details: The test system has a quad-core Intel i5-7300U and 16 GB of RAM. The IMAP server is running Dovecot 2.2.22 and requires TLS. The IMAP accounts contains approx 224.500 messages (4.3 GB of data). The largest folder contains about 43.400 messages. When starting KMail at about 12:30 I cancelled the initial Account Assistant wizard, and selected both "Work Offline" and "Use Less Bandwidth" from the File menu. I then created the IMAP account, making sure to uncheck "Download all messages for offline use" and "Interval mail checking". At 12:35 I selected "Work Online" from the File menu and clicked the "Check Mail" button. At this point, KMail goes to work. The status indicator is showing a progress bar titled "Syncing folder 'x'" which cycles through every folder on the IMAP server. While this goes on, I can see the following processes consistently occupying the top-5 spots in the output of top(1). They are sorted according to which position they typically was found at (although it does change around a bit from refresh to refresh): 1) mysqld 2) akonadiserver 3) akonadi_indexing_agent 4) akonadi_imap_resource 5) kmail The load average jumps up to around 5, and the laptop's fan revs up to max speed. At 13:02 the last "Syncing folder 'x'" progress indicator vanishes. At this point, IMAP network traffic stops and only the "akonadi_indexing_agent" process continue to hog a CPU core. At around 13:05 this process also calms down, the load eventually drops to below 1, and the laptop fan silences. As mentioned above, at this point there had been 1,19 GB of network traffic between the client and the IMAP server. Almost exclusively in the server->client direction. This was measured with iftop(8). The test user's (previously empty) home directory had grown to contain 3842 MB of data. This was almost all divided into these three directories: /home/kmailtest/.local/share/akonadi/db_data : 791 MB /home/kmailtest/.local/share/akonadi/file_db_data : 1360 MB /home/kmailtest/.local/share/akonadi/search_db : 1674 MB The Thunderbird process was started at 13:30. As with KMail, I cancelled the initial wizard and entered offline mode. Then I created the IMAP account, making sure to deselect "Keep messages for this account on this computer". At 13:35 I entered online mode and clicked "Get Messages". This caused it to download all the headers in the modestly-sized INBOX folder, which was done in maybe five seconds. This is IMHO a vastly superior approach to the KMail one of syncing all the remote folders immediately. In order to get a comparable test, I then proceeded to manually open every single IMAP folder, one after the other. Upon entering each folder Thunderbird would download the headers of all messages contained in it. I had walked through all the folders on the servers before 13:40. After this /home/tbirdtest had grown to contain 128 MB of data and iftop(8) had observed 210MB of network traffic between the client and the IMAP server. I will be attaching a few Munin graphs gathered from the laptop as these tests were run. The Thunderbird test isn't really visible, while the KMail one clearly is. I also ran this KMail test yesterday at around 21:30-22:00, which also clearly stands out. The KMail/Akonadi versions installed were: kf5-libkdepim-akonadi-17.04.1-1.fc26.x86_64 kf5-akonadi-notes-17.04.1-1.fc26.x86_64 kf5-akonadi-server-mysql-17.04.1-3.fc26.x86_64 kf5-akonadi-mime-17.04.1-1.fc26.x86_64 kf5-akonadi-contacts-17.04.1-1.fc26.x86_64 kf5-akonadi-search-17.04.1-1.fc26.x86_64 kmail-account-wizard-17.04.1-1.fc26.x86_64 kf5-kmailtransport-17.04.1-1.fc26.x86_64 akonadi-1.13.0-103.fc26.x86_64 kmail-17.04.1-3.fc26.x86_64 kf5-akonadi-calendar-17.04.1-1.fc26.x86_64 kf5-mailimporter-akonadi-17.04.1-1.fc26.x86_64 kdepimlibs-akonadi-4.14.10-18.fc26.x86_64 akonadi-import-wizard-17.04.1-1.fc26.x86_64 kf5-akonadi-server-17.04.1-3.fc26.x86_64 kmail-libs-17.04.1-3.fc26.x86_64 kf5-kmailtransport-akonadi-17.04.1-1.fc26.x86_64 kf5-pimcommon-akonadi-17.04.1-1.fc26.x86_64 Tore Created attachment 106916 [details]
CPU graph - KMail tests at 21:30-22:00 and 12:35-13:05
Created attachment 106917 [details]
IOOPS graph - KMail tests at 21:30-22:00 and 12:35-13:05
Created attachment 106918 [details]
I/O throughput graph - KMail tests at 21:30-22:00 and 12:35-13:05
Created attachment 106919 [details]
Load graph - KMail tests at 21:30-22:00 and 12:35-13:05
Created attachment 106920 [details]
Temperature graph - KMail tests at 21:30-22:00 and 12:35-13:05
By the way, if you need an IMAP account to test with, try the IETF mailing list archives, which is available through anonymous IMAP (cf. https://trac.tools.ietf.org/group/tools/trac/wiki/Imap): Server: imap.ietf.org port 143 Username: anonymous Password: anonymous@ietf.org Encryption: none Authentication: Clear text (note: auto-detection does NOT work, as the CAPABILITY output of the server is incorrect) There is a ton of folders on this server and probably several million messages. The fact of the matter is that this bug is quite simply making impossible to use KMail to access this server. "Use Less Bandwidth" or "Download all messages for offline use" doesn't help. Try it and see. Tore, Gunther or someone else who commented here: Could you please check whether you still have those performance issues you reported there with KDEPIM/Akonadi 20.04? If so, it might be good to report it in a new bug since – with my contribution unfortunately, this bug report has so many comments that it may be difficult to make sense out of it for a developer. If so, it would be good if you could provide some performance data again. I may check using the anonymous IMAP from IETF myself if I manage to take time for that. Gunter, sorry for spelling your name wrong. (In reply to Martin Steigerwald from comment #47) > Tore, Gunther or someone else who commented here: Could you please check > whether you still have those performance issues you reported there with > KDEPIM/Akonadi 20.04? I can try. I obviously haven't been using KMail in the last few years, but now I installed a Fedora 32 KDE VM to perform, upgraded it to Rawhide to get KDEPIM/Akonadi 20.04, rebooted it and deleted all files (including hidden) in my home directory from a console login (just to make sure what I am about to report was not a holdover from the previous Akonadi version in Fedora 32) and logged back in. The result - just from logging in - is that ~/.local/share/akonadi/db_data contains 158 MiB of stuff. To be clear: I have *not* started KMail, *not* edited any Akonadi settings, or anything like that. The *only* thing I have done is to log in from a completely empty user account, start Konsole, run 'du' to check disk usage. Yet Akonadi has managed to store 158 MiB of what has to be completely pointless stuff. That is beyond comprehension, and certainly does not give me any hope that this bug is anywhere close to being fixed, quite the opposite. Akonadi's disk and resource usage seems to still be beyond insane. That said, I will test towards an IMAP account later and update this bug with my findings. > If so, it might be good to report it in a new bug since – with my > contribution unfortunately, this bug report has so many comments that it may > be difficult to make sense out of it for a developer. What's the point? The reason why this bug has many comments is that it has languished here or almost six years with no developer taking any interest in actually fixing it. Why assume a new bug report will be treated any differently KMail is unusable with IMAP accounts with more than a few thousand messages, and it's been like this for years. I can only surmise that this is a use case the developers care about (which is fair enough, don't get me wrong, I get what I pay for here). Tore (In reply to Tore Anderson from comment #49) > (In reply to Martin Steigerwald from comment #47) > > Tore, Gunther or someone else who commented here: Could you please check > > whether you still have those performance issues you reported there with > > KDEPIM/Akonadi 20.04? > > I can try. I obviously haven't been using KMail in the last few years, but > now I installed a Fedora 32 KDE VM to perform, upgraded it to Rawhide to get > KDEPIM/Akonadi 20.04, rebooted it and deleted all files (including hidden) > in my home directory from a console login (just to make sure what I am about > to report was not a holdover from the previous Akonadi version in Fedora 32) > and logged back in. > > The result - just from logging in - is that ~/.local/share/akonadi/db_data > contains 158 MiB of stuff. > > To be clear: I have *not* started KMail, *not* edited any Akonadi settings, > or anything like that. The *only* thing I have done is to log in from a > completely empty user account, start Konsole, run 'du' to check disk usage. > > Yet Akonadi has managed to store 158 MiB of what has to be completely > pointless stuff. That is beyond comprehension, and certainly does not give > me any hope that this bug is anywhere close to being fixed, quite the > opposite. Akonadi's disk and resource usage seems to still be beyond insane. > > That said, I will test towards an IMAP account later and update this bug > with my findings. > > > If so, it might be good to report it in a new bug since – with my > > contribution unfortunately, this bug report has so many comments that it may > > be difficult to make sense out of it for a developer. > > What's the point? The reason why this bug has many comments is that it has > languished here or almost six years with no developer taking any interest in > actually fixing it. Why assume a new bug report will be treated any > differently > > KMail is unusable with IMAP accounts with more than a few thousand messages, > and it's been like this for years. I can only surmise that this is a use > case the developers care about (which is fair enough, don't get me wrong, I > get what I pay for here). > > Tore I shouldn't laugh. I've been using mutt for at least five years now, and I can't imagine ever going back to a GUI for my mail client. (In reply to Martin Steigerwald from comment #48) > Gunter, sorry for spelling your name wrong. No problem Martin, and thanks for even noticing it! ;) However, unfortunately I don't think I can be of much help regarding this or similar bugs at the moment. I'm using KDE Neon, and I abandoned kMail quite some time ago when - besides the very bad performance - kMail itself or Akonadi (cannot remember exactly) completely broke for me once again after some update. Unfortunately, it was no rare event to run "apt-get upgrade" on Neon - which you also need to do to get security and maintenance updates - just to find out that you didn't have a working mail client any more...) I also increasingly needed to work with CalDAV and team events, which didn't work at all reliably with kOrganizer. As I couldn't work this way, I switched to Thunderbird, even though I actually liked kMails interface and feature set (if it would work as expected...) better than Thunderbird's. I just was no longer willing to repeatedly have to invest hours of work just to be able to continue reading and writing mails after bugfix and maintenance updates to the OS... Until I abandoned it, performance didn't get any better, though. (I'm not sure when I switched to Thunderbird, but according to my kMail bug reports I used several 5.x versions, at least up to 5.8 and possibly later releases. OK, I have now tested KMail 5.14.1/Akonadi 20.04 in a freshly installed VM running Fedora 33 Rawhide. Then I compared it to the behaviour of Thunderbird. tl;dr: * KMail/Akonadi spends ~63 minutes on opening a large IMAP folder (133k messages) that Thunderbird spends ~8 minutes on. 688% increase with KMail/Akonadi. * KMail/Akonadi generates an overall higher load average than Thunderbird does (even without taking into account that it does so for a much longer period of time) * KMail/Akonadi ends up requiring 1703 MiB of disk space, Thunderbird only 124 MiB. 1273% increase with KMail/Akonadi. So by every single observable metric, KMail/Akonadi is *extremely* inefficient and slow. In fact it manages to use more disk space than Thunderbird does in the end, even *before* any IMAP account has been created (cf. comment #49). That is just ridiculous. Here is the exact test procedure I used. I'm using the IETF public IMAP server, so anyone can try to do the same. 1. Launched KMail, dismissed the initial account setup wizard. 2. Settings -> KMail settings -> Receiving tab -> Add...custom account -> IMAP 3. General tab: * Account name: IETF * IMAP Server: imap.ietf.org * Username: anonymous * Password: anonymous@ietf.org * Uncheck download all messages for offline use * Uncheck regular checking of e-mail Advanced tab: * Encryption: none 4. Pressed OK, waited for folder list to be downloaded 5. Found the folder named just "ietf" in the list, clicked on it as I started a stopwatch (@11:40). This folder has about 133.000 messages in it. 6. Waited until folder sync and threading had completed After ~15 minutes: folder sync 22% completed, 15-min load average 0.52 After ~30 minutes: folder sync 47% completed, 15-min load average 0.88 After ~45 minutes: folder sync 73% completed, 15-min load average 1.39 After ~60 minutes: folder sync 100% completed, 15-min load average 1.63 At this point, KMail is still busy doing stuff. Status bar says «grouped X of Y threads», «processing X of Y messages», etc. After ~63 minutes: KMail finally idles. 15-min load average 1.61 7. Closed KMail 8. du -ms ~/.local/share/akonadi -> 1703 MiB I then compared with Thunderbird's behaviour: 9. Launched Thunderbird (version 68.8.0) 10. In inital account wizard: * E-mail address: anonymous@ietf.org * Password: anonymous Click "Manual settings": * Incoming server name: imap.ietf.org * Incoming SSL: none * Incoming authentication: regular password * Configured a random outgoing SMTP account (since it's required) * Clicked "Test Again" * Clicked "Advanced settings" * Server settings page: Unchecked "Look for new messages on startup" * Server settings page: Unchecked "Look for new messages every 10 minutes" * Synchronisation and storage page: Unchecked "Keep messages in all folder for this account on this computer" Press OK 11. In advanced config editor: set mail.server.default.using_subscription to false 12. Restart Thunderbird (apparently necessary for mail.server.default.using_subscription to take effect), let it load the list of shared folders 13. Clicked the "ietf" folder and started stopwatch Waiting until folder sync was done: After ~5m: 98k of 133k messages processed (~74%), 5-min load average 0.68 After ~8m: completed. 5-min load average 0.62 14: du -ms ~/.thunderbird -> 124 MiB Tore Thank you, Tore. I updated the version number in the bug report from 5.5.2 to 5.14.1. As for the initial size requirement, I think that is just the MariaDB database with InnoDB logfiles. AFAIR it uses 2x64 MiB InnoDB logfiles already. Tore, thank you again for your recent findings. Just to let you know: I was not doubting it. In my experience Akonadi based KMail with IMAP never really worked with folders with more than maybe 10000-20000 mails. > In my experience Akonadi based KMail with IMAP never really worked
Assuming KMail wasn't always «Akonadi based», I would agree with this.
I used to be a happy KMail user years ago, then I upgraded my distro and found it to have become unusable. This is not an exaggeration, it was literally unusable. I had to immediately find another MUA in order to continue to use my e-mail accounts. I've occasionally tried to go back to KMail since then, but there has been no improvement. So I've been using other clients like Evolution, Thunderbird, Claws, etc. - they all work fine.
I don't remember the exact version when it became unusable myself, since it was so long ago. The OP puts it at KDE 4.14, though, which I have no reason to doubt is correct.
Thus, if KMail became «Akonadi based» in KDE 4.14 but was «SomethingElse based» in KDE 4.13, then the Akonadi stuff does indeed appear to be the problem here.
Tore
|