Even after working around [Bug 332684] New: [Maildir] lots of stats calls to /etc/localtime on synchronizing folders by setting an TZ environment variable synchronizing large folders with filesystem contents hogs one CPU core for minutes. Reproducible: Always Steps to Reproduce: 1. Have a large maildir folder. 2. Synchronize it. Actual Results: akonadi_maildir_resource hogs one Sandybridge core for minutes. SSDs are under utilized. MySQL barely visible. Expected Results: Synchronizing large folders is faster. Akonadi stats every file. Is it necessary? For a folder with 250000 mails that are 250000 calls to stat(). While just listing folder contents with martin@merkaba:~/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory> /usr/bin/time find kernel-ml | wc -l 0.21user 0.35system 0:00.68elapsed 82%CPU (0avgtext+0avgdata 59316maxresident)k 13648inputs+0outputs (1major+17920minor)pagefaults 0swaps 250167 is blazingly fast here. We have high CPU usage here as well… but I bet thats due to Linux caching the directory entries and inodes: OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 454480 453722 99% 0,98K 28405 16 454480K btrfs_inode 434616 418562 96% 0,19K 20696 21 82784K dentry So, wouldn´t it be sufficient to only stat() the files that are new or have updated timestamps? martin@merkaba:~/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory> /usr/bin/time find kernel-ml -ls | wc -l 0.70user 0.36system 0:01.07elapsed 99%CPU (0avgtext+0avgdata 59536maxresident)k 32inputs+0outputs (0major+18010minor)pagefaults 0swaps 250167 indicated that also the timestamps can be provided quickly. So I´d: 1) list the fs folder contents for filenames and timestamps (mtime). 2) compare with database. 3) only stat() the files that are new or have been updated meanwhile. Result: Blazingly fast folder sync? Part of the CPU time used I see no activity of akonadi maildir resource in strace. Other time is stat()-ing files like this: [pid 4137] stat("/home/martin/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory/kernel-ml/new/1376733031.R234.merkaba", {st_mode=S_IFREG|0644, st_size=4079, ...}) = 0 [pid 4137] stat("/home/martin/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory/kernel-ml/new/1376733031.R322.merkaba", {st_mode=S_IFREG|0644, st_size=8056, ...}) = 0 [pid 4137] stat("/home/martin/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory/kernel-ml/new/1376733031.R342.merkaba", {st_mode=S_IFREG|0644, st_size=2771, ...}) = 0 [pid 4137] stat("/home/martin/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory/kernel-ml/new/1376733031.R608.merkaba", {st_mode=S_IFREG|0644, st_size=4492, ...}) = 0 [pid 4137] stat("/home/martin/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory/kernel-ml/new/1376733031.R665.merkaba", {st_mode=S_IFREG|0644, st_size=13036, ...}) = 0 [pid 4137] stat("/home/martin/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory/kernel-ml/new/1376733031.R738.merkaba", ^C{st_mode=S_IFREG|0644, st_size=6870, ...}) = 0 Related observations also indicate that Akonadi is doing this work needlessly: Bug 334209 - synchronizes folder contents during runtime needlessly Bug 334216 - synchronizes folder with filesystem after downloading and filtering mails needlessly Again blazingly fast ThinkPad T520 with Sandybridge and Dual SSD BTRFS RAID 1 setup.
I bet that stat()ing files isn´t the only issue here, as find -ls also uses a variant of the stat() call on each file and as demonstrated it way faster: Warm caches: martin@merkaba:~/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory> strace -c find kernel-ml -ls 2>&1 >/dev/null % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 90.93 4.199983 17 250167 newfstatat 6.34 0.292629 770 380 getdents 1.97 0.091028 14 6437 write 0.60 0.027740 51 539 brk 0.04 0.001891 34 56 18 open 0.03 0.001159 27 43 mmap 0.02 0.000866 17 50 close 0.01 0.000650 17 38 fstat 0.01 0.000492 492 1 execve 0.01 0.000458 29 16 mprotect 0.01 0.000334 42 8 8 access 0.01 0.000323 36 9 munmap 0.01 0.000311 24 13 read 0.00 0.000180 45 4 4 connect 0.00 0.000176 10 17 fchdir 0.00 0.000150 21 7 openat 0.00 0.000121 30 4 socket 0.00 0.000095 19 5 lseek 0.00 0.000051 17 3 2 ioctl 0.00 0.000027 27 1 uname 0.00 0.000014 14 1 arch_prctl ------ ----------- ----------- --------- --------- ---------------- 100.00 4.618678 257799 32 total martin@merkaba:~/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory> /usr/bin/time find kernel-ml -ls 2>&1 >/dev/null 0.69user 0.35system 0:01.04elapsed 100%CPU (0avgtext+0avgdata 59536maxresident)k 0inputs+0outputs (0major+18011minor)pagefaults 0swaps After echo 3 > /proc/sys/vm/drop_caches: martin@merkaba:~/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory> /usr/bin/time find kernel-ml -ls 2>&1 >/dev/null 3.40user 9.67system 0:21.56elapsed 60%CPU (0avgtext+0avgdata 59532maxresident)k 780736inputs+0outputs (8major+18001minor)pagefaults 0swaps (still just 20 seconds and well in case of KMail / Akonadi caches are warm, machine has 8 GB of RAM)
Compared to this strace of martin@merkaba:~#16> ps aux | head -1 ; ps aux | grep akonadi_maildir_resource | grep -v grep USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND martin 26830 12.3 4.4 660948 356240 ? Rl 12:49 5:47 /usr/local/bin/akonadi_agent_launcher akonadi_maildir_resource akonadi_maildir_resource_0 after downloading about 50-100 new mails and filtering them. The tasks used up one sandybridge core for several minutes even in this case – OK, granted strace will make it way slower, so what is 5-10 minutes might be one minute without strace maybe. Here strace -c for the whole time of maildir resource occupying the CPU core: merkaba:~> strace -c -f -p 26830 Process 26830 attached with 2 threads - interrupt to quit ^CProcess 26830 detached Process 26831 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 84.29 10.324846 26 395164 111 stat 5.87 0.719352 448 1607 getdents 2.85 0.349666 5 74548 74264 recvmsg 2.77 0.339268 9 37233 459 read 1.70 0.208472 6 37003 poll 1.23 0.150676 4 35434 ioctl 0.39 0.048249 11 4521 write 0.24 0.029460 446 66 rename 0.17 0.021152 18 1178 sendmsg 0.14 0.017625 10 1744 access 0.09 0.010768 21 518 brk 0.08 0.009760 38 259 open 0.03 0.003285 11 309 close 0.03 0.003128 7 442 fstat 0.03 0.003128 45 69 66 lstat 0.02 0.002616 30 87 munmap 0.02 0.002154 43 50 statfs 0.01 0.001812 21 87 mmap 0.01 0.001712 7 259 fcntl 0.01 0.000826 17 50 openat 0.00 0.000411 7 59 lseek 0.00 0.000262 5 58 uname 0.00 0.000078 78 1 restart_syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 12.248706 590746 74900 total Note during this time I saw not any notable activity of MySQL server in atop. Just the maildir resource hogging up the CPU. Also Akonadi server itself is idling. Except for the end where MySQL and Akonadi server show a short burst of activity. After this analysis I get more and more the impression that the time Akonadi maildir resource takes to sort 50-100 mails exceeds the actually necessary effort by a tremendous factor. Maildir resource is doing needless work here. For me there is no other explaination to it. Filtering mails to folders IMHO consists of the following work: 1) Running filters on the mails. These load the mail headers and check various criteria. Should be done in a second or two. 2) Actually moving the files to their respective folders which involves: - changing folder path in database. - mv the file - okay in case of Akonadi this may go through file_db_data? Anyway I cannot see why this ridicolously amount of system calls and CPU usage – as I demontrated in my analysis here – is even remotely necessary for the job. After this analysis I intend to look at the source again. Maybe with reading API docs I understand it a bit better than last time and find something. Akonadi is git 159bfbdf8667bb01b1f4c85ea144ca128ccb3195
Created attachment 86406 [details] callgrind data of 5 minutes cpu busy maildir resource About 5 minute callgrind data with akonadi maildir resource being 100% cpu busy after Akonadi restart. As its still busy I can add another dump later on :) Done as: martin@merkaba:~> env | grep AKONADI AKONADI_VALGRIND=akonadi_maildir_resource AKONADI_VALGRIND_SKIN=callgrind AKONADI_VALGRIND_OPTIONS=--instr-atstart=no --callgrind-out-file=/home/martin/callgrind-vcard.%p martin@merkaba:~> callgrind_control -i on PID 28282: /usr/local/bin/akonadi_agent_launcher akonadi_maildir_resource akonadi_maildir_resource_0 sending command instrumentation on to pid 28282 OK. martin@merkaba:~> date Fr 2. Mai 14:06:14 CEST 2014 martin@merkaba:~> date Fr 2. Mai 14:11:14 CEST 2014 martin@merkaba:~> callgrind_control -d PID 28282: /usr/local/bin/akonadi_agent_launcher akonadi_maildir_resource akonadi_maildir_resource_0
Created attachment 86408 [details] complete callgrind run over a longer time with three dumps one after 5 minutes, another one after 15 minutes (i.e. ten minutes later) and a third one before stopping the thing
Testing with: diff --git a/resources/maildir/libmaildir/maildir.cpp b/resources/maildir/libmaildir/maildir.cpp index 9bd3802..36166ed 100644 --- a/resources/maildir/libmaildir/maildir.cpp +++ b/resources/maildir/libmaildir/maildir.cpp @@ -106,12 +106,14 @@ public: QStringList listNew() const { QDir d( path + QString::fromLatin1( "/new" ) ); + d.setSorting(QDir::NoSort); return d.entryList( QDir::Files ); } QStringList listCurrent() const { QDir d( path + QString::fromLatin1( "/cur" ) ); + d.setSorting(QDir::NoSort); return d.entryList( QDir::Files ); } which disables QDir sorting which was so prominent in callgrind traces. I get the impression it helps the bit, but it does not seem to fix all of the performance issue. Need to take more time to check more thoroughly.
Suggestion was from David Faure and Sergio.
Created attachment 86418 [details] second callgrind of about well at least half an hour, after nodirsorting patch applied This one looks a quite bit different. The resource still hogs a core, partly for minutes still, and KMail. But KMail does respond more often in between. I think KMail reacts after each complete folder synchronisation for a short time. So do you think about this callgrind?
During this callgrind run KMail did respond in between, so I really think folder sync got faster by nodir sorting patch as during last callgrind of also about half an hour or so it didn´t respond once. Yet… no hard numbers yet. I don´t know how to make repeatable tests so far.
Sergio, I tried: 107 QStringList listNew() const 108 { 109 static int count = 0; count++; 110 static int elapsed = 0; 111 112 QDir d( path + QString::fromLatin1( "/new" ) ); 113 d.setSorting(QDir::NoSort); 114 115 QTime t; 116 t.start(); 117 QStringList list = d.entryList( QDir::Files ); 118 elapsed += t.elapsed(); 119 kDebug() << "PERF: listNew NoSort " << count << elapsed << "\n"; 120 return list; 121 122 //return d.entryList( QDir::Files ); 123 } 124 125 QStringList listCurrent() const 126 { 127 static int count = 0; count++; 128 static int elapsed = 0; 129 130 QDir d( path + QString::fromLatin1( "/cur" ) ); 131 d.setSorting(QDir::NoSort); 132 133 QTime t; 134 t.start(); 135 QStringList list = d.entryList( QDir::Files ); 136 elapsed += t.elapsed(); 137 kDebug() << "PERF: listCurrent NoSort " << count << elapsed << "\n"; 138 return list; 139 140 //return d.entryList( QDir::Files ); 141 } as well as your - std::cout << "PERF: listNew NoSort " << count << elapsed << "\n"; + kDebug() << "PERF: listNew NoSort " << count << elapsed << "\n"; I see no output on konsole or in ~/.xsession-errors even with full debug enabled in kdebugdialog before starting akonadi again. Additionally I look at the callgrind files again and I still think they are quite different. The nodirsorting callgrind-vcard.26128.1 has: - QDir::entryList - QAlgorithmsPrivate::qSortHelper - QAlgorithmsPrivate::qSortHelper - QDirSortItemComparator::operator - QString::compare as first 5 out of first 6 entries. And none of it has a very long bar. Yet the old callgrind has - QAlgorithmsPrivat::qSortHelper, lr 145684.95, lr per call as the first of the entries and it has a very long outstanding bar. This very long outstanding bar is basically gone in the new callgrind run. Here as directly from callgrind file: So somewhere QDir may still sort something, but not at the original place anymore. At least thats my bet on looking at the data.
Sergio, I tried: 107 QStringList listNew() const 108 { 109 static int count = 0; count++; 110 static int elapsed = 0; 111 112 QDir d( path + QString::fromLatin1( "/new" ) ); 113 d.setSorting(QDir::NoSort); 114 115 QTime t; 116 t.start(); 117 QStringList list = d.entryList( QDir::Files ); 118 elapsed += t.elapsed(); 119 kDebug() << "PERF: listNew NoSort " << count << elapsed << "\n"; 120 return list; 121 122 //return d.entryList( QDir::Files ); 123 } 124 125 QStringList listCurrent() const 126 { 127 static int count = 0; count++; 128 static int elapsed = 0; 129 130 QDir d( path + QString::fromLatin1( "/cur" ) ); 131 d.setSorting(QDir::NoSort); 132 133 QTime t; 134 t.start(); 135 QStringList list = d.entryList( QDir::Files ); 136 elapsed += t.elapsed(); 137 kDebug() << "PERF: listCurrent NoSort " << count << elapsed << "\n"; 138 return list; 139 140 //return d.entryList( QDir::Files ); 141 } as well as your - std::cout << "PERF: listNew NoSort " << count << elapsed << "\n"; + kDebug() << "PERF: listNew NoSort " << count << elapsed << "\n"; I see no output on konsole or in ~/.xsession-errors even with full debug enabled in kdebugdialog before starting akonadi again. Additionally I look at the callgrind files again and I still think they are quite different. The nodirsorting callgrind-vcard.26128.1 has: - QDir::entryList - QAlgorithmsPrivate::qSortHelper - QAlgorithmsPrivate::qSortHelper - QDirSortItemComparator::operator - QString::compare as first 5 out of first 6 entries. And none of it has a very long bar. Yet the old callgrind-vcard.28282.1 has - QAlgorithmsPrivat::qSortHelper, lr 145684.95, lr per call as the first of the entries and it has a very long outstanding bar. This very long outstanding bar is basically gone in the new callgrind run. Just look at each Just look at these two files with kcachegrind to me they look quite different in the old the sort stuff is dominating everything else by a factor of 10 at least, in the new it looks way more balanced. So somewhere QDir may still sort something, but not at the original place anymore. At least thats my bet on looking at the data. I wonder whether something else is using QDir as well and did not disable sorting and we see this now.
I don´t know whether this is called and what that does, but setting nosort here as well for testing: diff --git a/resources/maildir/libmaildir/keycache.cpp b/resources/maildir/libmaildir/keycache.cpp index f0af9c4..814ce6c 100644 --- a/resources/maildir/libmaildir/keycache.cpp +++ b/resources/maildir/libmaildir/keycache.cpp @@ -75,12 +75,14 @@ bool KeyCache::isNewKey( const QString& dir, const QString& key ) const QSet< QString > KeyCache::listNew( const QString& dir ) const { QDir d( dir + QString::fromLatin1( "/new" ) ); + d.setSorting(QDir::NoSort); return d.entryList( QDir::Files ).toSet(); } QSet< QString > KeyCache::listCurrent( const QString& dir ) const { QDir d( dir + QString::fromLatin1( "/cur" ) ); + d.setSorting(QDir::NoSort); return d.entryList( QDir::Files ).toSet(); }
Created attachment 86428 [details] callgrind with nodirsorting also in libmaildir/keycache.cpp Well this is it. KMail is usable even with make -j4 on kdepim-runtime on a hyperthreaded Sandybridge mobile dual core with that. Akonadi maildir resource still appears with some CPU usage, but I didn´t see it at 100% for even just one 10 second measurement period in atop. callgrind logs show a different bottleneck now: MaildirResource::listRecursive To me these two nosorting patches have such a huge impact on Maildir resource performance that using KMail is a completely different experience now. KMail hardly ever blocks on folder changes anymore for example, although that seems just the case due to synchronisation being much faster now. So this still is good to *fix*.
Review Request: Change libmaildir listNew and listCurrent functions to explicetely not sort QDir.entryList https://git.reviewboard.kde.org/r/117975/
Git commit 4dd5cb7fcb37f9da481d8324421428a0028e9e2d by Martin Steigerwald. Committed on 03/05/2014 at 14:52. Pushed by steigerwald into branch 'master'. Do not sort the directory entry list in listNew() and listCurrent(): According to callgrind dumps at https://bugs.kde.org/show_bug.cgi?id=334218#c4 QDir sorts the list of directory entries unless specified otherwise. After disabling the sorting the callgrind is quite different already: https://bugs.kde.org/show_bug.cgi?id=334218#c7 This already helps shortening the time to synchronize folders visibly, but KMail gets still blocked for half a minute or more. There is another occurence of QDir entry sorting in keycache.cpp which the next commit will address. Thanks to Sergio and David for help and pointing out how to disable the sorting. REVIEW: 117975 DIGEST: Huge performance improvement for POP3 users with large maildir. M +2 -0 resources/maildir/libmaildir/maildir.cpp http://commits.kde.org/kdepim-runtime/4dd5cb7fcb37f9da481d8324421428a0028e9e2d
Git commit 41be4d27dab49d600e3bd93cfc3804a347243257 by Martin Steigerwald. Committed on 03/05/2014 at 14:53. Pushed by steigerwald into branch 'master'. Also do not sort directory entries in listCurrent and listNew in key cache: QDir sorts the list of directory entries unless specified otherwise. The last commit already disabled the sorting in maildir.cpp. This commit completes the work and makes the remaining sorting calls in the callgrind dumps at https://bugs.kde.org/show_bug.cgi?id=334218#c7 go away completely as demonstrated in the callgrind dumps at: https://bugs.kde.org/show_bug.cgi?id=334218#c12 Subjectively this has a huge impact on the performance of KMail with huge maildir. KMail hardly blocks anymore on folder changes and feels much more responsive now. Akonadi maildir resource went from hogging a Sandy Bridge core for minutes to not appearing using 100% of one core in even one averaged 10 second interval in atop. I hardly see it in atop at all anymore, except when accessing very large folders such as one with Linux Kernel mailing list and more than 245000 unread mails. Thanks to Sergio for pointing out the remaining sorting calls in the second callgrind dump and for help. Now the bottleneck appears to be MaildirResource::listRecursive but I do not know whether further optimization is necessary at this point. Both changes tested with kdepimlibs master as of today and KMail 4.12.4 from Debian unstable packages with an enormous maildir including a folder for Linux Kernel mailing list with more than 245000 unread mails. REVIEW: 117975 DIGEST: Huge performance improvement for POP3 users with large maildirs. M +2 -0 resources/maildir/libmaildir/keycache.cpp http://commits.kde.org/kdepim-runtime/41be4d27dab49d600e3bd93cfc3804a347243257
Git commit 1af2704a63c6641867eaf2b67976f5309a994cca by Christian Mollekopf, on behalf of Martin Steigerwald. Committed on 03/05/2014 at 14:52. Pushed by cmollekopf into branch 'kolab/integration/4.13.0'. Do not sort the directory entry list in listNew() and listCurrent(): According to callgrind dumps at https://bugs.kde.org/show_bug.cgi?id=334218#c4 QDir sorts the list of directory entries unless specified otherwise. After disabling the sorting the callgrind is quite different already: https://bugs.kde.org/show_bug.cgi?id=334218#c7 This already helps shortening the time to synchronize folders visibly, but KMail gets still blocked for half a minute or more. There is another occurence of QDir entry sorting in keycache.cpp which the next commit will address. Thanks to Sergio and David for help and pointing out how to disable the sorting. REVIEW: 117975 DIGEST: Huge performance improvement for POP3 users with large maildir. M +2 -0 resources/maildir/libmaildir/maildir.cpp http://commits.kde.org/kdepim-runtime/1af2704a63c6641867eaf2b67976f5309a994cca
Git commit 81f30e18c975f9cc44da7c904ed612df804200da by Christian Mollekopf, on behalf of Martin Steigerwald. Committed on 03/05/2014 at 14:53. Pushed by cmollekopf into branch 'kolab/integration/4.13.0'. Also do not sort directory entries in listCurrent and listNew in key cache: QDir sorts the list of directory entries unless specified otherwise. The last commit already disabled the sorting in maildir.cpp. This commit completes the work and makes the remaining sorting calls in the callgrind dumps at https://bugs.kde.org/show_bug.cgi?id=334218#c7 go away completely as demonstrated in the callgrind dumps at: https://bugs.kde.org/show_bug.cgi?id=334218#c12 Subjectively this has a huge impact on the performance of KMail with huge maildir. KMail hardly blocks anymore on folder changes and feels much more responsive now. Akonadi maildir resource went from hogging a Sandy Bridge core for minutes to not appearing using 100% of one core in even one averaged 10 second interval in atop. I hardly see it in atop at all anymore, except when accessing very large folders such as one with Linux Kernel mailing list and more than 245000 unread mails. Thanks to Sergio for pointing out the remaining sorting calls in the second callgrind dump and for help. Now the bottleneck appears to be MaildirResource::listRecursive but I do not know whether further optimization is necessary at this point. Both changes tested with kdepimlibs master as of today and KMail 4.12.4 from Debian unstable packages with an enormous maildir including a folder for Linux Kernel mailing list with more than 245000 unread mails. REVIEW: 117975 DIGEST: Huge performance improvement for POP3 users with large maildirs. M +2 -0 resources/maildir/libmaildir/keycache.cpp http://commits.kde.org/kdepim-runtime/81f30e18c975f9cc44da7c904ed612df804200da
Git commit afdc4bc8d85dbadfddd5d89169fede0f5a57317d by Martin Steigerwald. Committed on 03/05/2014 at 14:52. Pushed by steigerwald into branch 'KDE/4.13'. Do not sort the directory entry list in listNew() and listCurrent(): According to callgrind dumps at https://bugs.kde.org/show_bug.cgi?id=334218#c4 QDir sorts the list of directory entries unless specified otherwise. After disabling the sorting the callgrind is quite different already: https://bugs.kde.org/show_bug.cgi?id=334218#c7 This already helps shortening the time to synchronize folders visibly, but KMail gets still blocked for half a minute or more. There is another occurence of QDir entry sorting in keycache.cpp which the next commit will address. Thanks to Sergio and David for help and pointing out how to disable the sorting. REVIEW: 117975 DIGEST: Huge performance improvement for POP3 users with large maildir. M +2 -0 resources/maildir/libmaildir/maildir.cpp http://commits.kde.org/kdepim-runtime/afdc4bc8d85dbadfddd5d89169fede0f5a57317d
Git commit ea0b761fbe3c43b42f83c5477018e10209f92ea3 by Martin Steigerwald. Committed on 03/05/2014 at 14:53. Pushed by steigerwald into branch 'KDE/4.13'. Also do not sort directory entries in listCurrent and listNew in key cache: QDir sorts the list of directory entries unless specified otherwise. The last commit already disabled the sorting in maildir.cpp. This commit completes the work and makes the remaining sorting calls in the callgrind dumps at https://bugs.kde.org/show_bug.cgi?id=334218#c7 go away completely as demonstrated in the callgrind dumps at: https://bugs.kde.org/show_bug.cgi?id=334218#c12 Subjectively this has a huge impact on the performance of KMail with huge maildir. KMail hardly blocks anymore on folder changes and feels much more responsive now. Akonadi maildir resource went from hogging a Sandy Bridge core for minutes to not appearing using 100% of one core in even one averaged 10 second interval in atop. I hardly see it in atop at all anymore, except when accessing very large folders such as one with Linux Kernel mailing list and more than 245000 unread mails. Thanks to Sergio for pointing out the remaining sorting calls in the second callgrind dump and for help. Now the bottleneck appears to be MaildirResource::listRecursive but I do not know whether further optimization is necessary at this point. Both changes tested with kdepimlibs master as of today and KMail 4.12.4 from Debian unstable packages with an enormous maildir including a folder for Linux Kernel mailing list with more than 245000 unread mails. REVIEW: 117975 DIGEST: Huge performance improvement for POP3 users with large maildirs. M +2 -0 resources/maildir/libmaildir/keycache.cpp http://commits.kde.org/kdepim-runtime/ea0b761fbe3c43b42f83c5477018e10209f92ea3
Git commit 431ae9b185aa9e74a6b822fbee44e4b93b3f1498 by Martin Steigerwald. Committed on 29/05/2014 at 15:49. Pushed by steigerwald into branch 'master'. Manually sort the QStringList for the readEntryFlags test Otherwise the individual test in this testcase may work on the wrong mails as libmaildir doesn´t sort the entry list anymore. REVIEW: 118400 M +4 -0 resources/maildir/libmaildir/tests/testmaildir.cpp http://commits.kde.org/kdepim-runtime/431ae9b185aa9e74a6b822fbee44e4b93b3f1498
Git commit c6659e74196b74ebe32254cf0b42d9ed4274d17c by Martin Steigerwald. Committed on 31/05/2014 at 18:18. Pushed by steigerwald into branch 'master'. Manually sort the item list so that tagListHash.contains tests work Otherwise the individual tests in this testcase may work on the wrong mails as libmaildir doesn´t sort the entry list anymore. This is related to the nosorting change from review request 117975. This needs to go into 4.13 branch as well before release of 4.13.2. REVIEW: 118441 M +5 -0 resources/mixedmaildir/tests/itemfetchtest.cpp http://commits.kde.org/kdepim-runtime/c6659e74196b74ebe32254cf0b42d9ed4274d17c
Git commit 2b8c7db15c11c6d76a691b831fe83dc0879872ab by Martin Steigerwald. Committed on 29/05/2014 at 15:49. Pushed by steigerwald into branch 'KDE/4.13'. Manually sort the QStringList for the readEntryFlags test Otherwise the individual test in this testcase may work on the wrong mails as libmaildir doesn´t sort the entry list anymore. REVIEW: 118400 M +4 -0 resources/maildir/libmaildir/tests/testmaildir.cpp http://commits.kde.org/kdepim-runtime/2b8c7db15c11c6d76a691b831fe83dc0879872ab
Git commit 6428954599ce87fd3844a43e3581aec47d4033ee by Martin Steigerwald. Committed on 31/05/2014 at 18:18. Pushed by steigerwald into branch 'KDE/4.13'. Manually sort the item list so that tagListHash.contains tests work Otherwise the individual tests in this testcase may work on the wrong mails as libmaildir doesn´t sort the entry list anymore. This is related to the nosorting change from review request 117975. This needs to go into 4.13 branch as well before release of 4.13.2. REVIEW: 118441 M +5 -0 resources/mixedmaildir/tests/itemfetchtest.cpp http://commits.kde.org/kdepim-runtime/6428954599ce87fd3844a43e3581aec47d4033ee
Git commit ceb7bdd423f5ed148b53f422795a167df48ab99e by Martin Steigerwald. Committed on 08/06/2014 at 16:34. Pushed by steigerwald into branch 'master'. Sort by filename, which is RemoteId, instead of item id to make test pass. Review request 118441 introduced sorting of items for ItemFetchJob testcase as due to an performance optimization mixedmaildir resource does not sort anymore. This fixed the testcase for me on BTRFS. However on the Jenkins server the test case was still not fixed and Kevin pointed out that qSort may sort by item id instead of RemoteId. The test relies on sorting by filename, which is basically the RemoteId. REVIEW: 118619 M +8 -1 resources/mixedmaildir/tests/itemfetchtest.cpp http://commits.kde.org/kdepim-runtime/ceb7bdd423f5ed148b53f422795a167df48ab99e
Git commit 79ca2264b5c210ec02f4240b161477d5d9314d25 by Martin Steigerwald. Committed on 08/06/2014 at 16:34. Pushed by steigerwald into branch 'KDE/4.13'. Sort by filename, which is RemoteId, instead of item id to make test pass. Review request 118441 introduced sorting of items for ItemFetchJob testcase as due to an performance optimization mixedmaildir resource does not sort anymore. This fixed the testcase for me on BTRFS. However on the Jenkins server the test case was still not fixed and Kevin pointed out that qSort may sort by item id instead of RemoteId. The test relies on sorting by filename, which is basically the RemoteId. REVIEW: 118619 M +8 -1 resources/mixedmaildir/tests/itemfetchtest.cpp http://commits.kde.org/kdepim-runtime/79ca2264b5c210ec02f4240b161477d5d9314d25
While this may be able to be tuned further the original hogging of a Sandybridge core for minutes is definately gone, currently using Akonadi 1.13 current master, kdepimlibs and kdepim-runtime from 24th of February and KMail 4.14.2. Akonadi 1.13 has the MySQL performance improvements. I think it may still do needless work on maildir synchronisation, but I have other bugs open for that. Thus closing in my attempt at which of my bug reports are still happening and matter in current versions.