Bug 334218

I bet that stat()ing files isn´t the only issue here, as find -ls also uses a variant of the stat() call on each file and as demonstrated it way faster:

Warm caches:

martin@merkaba:~/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory> strace -c find kernel-ml -ls 2>&1 >/dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 90.93    4.199983          17    250167           newfstatat
  6.34    0.292629         770       380           getdents
  1.97    0.091028          14      6437           write
  0.60    0.027740          51       539           brk
  0.04    0.001891          34        56        18 open
  0.03    0.001159          27        43           mmap
  0.02    0.000866          17        50           close
  0.01    0.000650          17        38           fstat
  0.01    0.000492         492         1           execve
  0.01    0.000458          29        16           mprotect
  0.01    0.000334          42         8         8 access
  0.01    0.000323          36         9           munmap
  0.01    0.000311          24        13           read
  0.00    0.000180          45         4         4 connect
  0.00    0.000176          10        17           fchdir
  0.00    0.000150          21         7           openat
  0.00    0.000121          30         4           socket
  0.00    0.000095          19         5           lseek
  0.00    0.000051          17         3         2 ioctl
  0.00    0.000027          27         1           uname
  0.00    0.000014          14         1           arch_prctl
------ ----------- ----------- --------- --------- ----------------
100.00    4.618678                257799        32 total

martin@merkaba:~/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory> /usr/bin/time find kernel-ml -ls 2>&1 >/dev/null
0.69user 0.35system 0:01.04elapsed 100%CPU (0avgtext+0avgdata 59536maxresident)k
0inputs+0outputs (0major+18011minor)pagefaults 0swaps


After echo 3 > /proc/sys/vm/drop_caches:

martin@merkaba:~/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory> /usr/bin/time find kernel-ml -ls 2>&1 >/dev/null
3.40user 9.67system 0:21.56elapsed 60%CPU (0avgtext+0avgdata 59532maxresident)k
780736inputs+0outputs (8major+18001minor)pagefaults 0swaps

(still just 20 seconds and well in case of KMail / Akonadi caches are warm, machine has 8 GB of RAM)

Compared to this

strace of

martin@merkaba:~#16> ps aux | head -1 ; ps aux | grep akonadi_maildir_resource | grep -v grep
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
martin   26830 12.3  4.4 660948 356240 ?       Rl   12:49   5:47 /usr/local/bin/akonadi_agent_launcher akonadi_maildir_resource akonadi_maildir_resource_0

after downloading about 50-100 new mails and filtering them. The tasks used up one sandybridge core for several minutes even in this case – OK, granted strace will make it way slower, so what is 5-10 minutes might be one minute without strace maybe. Here strace -c for the whole time of maildir resource occupying the CPU core:

merkaba:~> strace -c -f -p 26830 
Process 26830 attached with 2 threads - interrupt to quit
^CProcess 26830 detached
Process 26831 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 84.29   10.324846          26    395164       111 stat
  5.87    0.719352         448      1607           getdents
  2.85    0.349666           5     74548     74264 recvmsg
  2.77    0.339268           9     37233       459 read
  1.70    0.208472           6     37003           poll
  1.23    0.150676           4     35434           ioctl
  0.39    0.048249          11      4521           write
  0.24    0.029460         446        66           rename
  0.17    0.021152          18      1178           sendmsg
  0.14    0.017625          10      1744           access
  0.09    0.010768          21       518           brk
  0.08    0.009760          38       259           open
  0.03    0.003285          11       309           close
  0.03    0.003128           7       442           fstat
  0.03    0.003128          45        69        66 lstat
  0.02    0.002616          30        87           munmap
  0.02    0.002154          43        50           statfs
  0.01    0.001812          21        87           mmap
  0.01    0.001712           7       259           fcntl
  0.01    0.000826          17        50           openat
  0.00    0.000411           7        59           lseek
  0.00    0.000262           5        58           uname
  0.00    0.000078          78         1           restart_syscall
------ ----------- ----------- --------- --------- ----------------
100.00   12.248706                590746     74900 total

Note during this time I saw not any notable activity of MySQL server in atop. Just the maildir resource hogging up the CPU. Also Akonadi server itself is idling. Except for the end where MySQL and Akonadi server show a short burst of activity.

After this analysis I get more and more the impression that the time Akonadi maildir resource takes to sort 50-100 mails exceeds the actually necessary effort by a tremendous factor. Maildir resource is doing needless work here. For me there is no other explaination to it. Filtering mails to folders IMHO consists of the following work:

1) Running filters on the mails. These load the mail headers and check various criteria. Should be done in a second or two.

2) Actually moving the files to their respective folders which involves:
- changing folder path in database.
- mv the file
- okay in case of Akonadi this may go through file_db_data?

Anyway I cannot see why this ridicolously amount of system calls and CPU usage – as I demontrated in my analysis here – is even remotely necessary for the job.

After this analysis I intend to look at the source again. Maybe with reading API docs I understand it a bit better than last time and find something.

Akonadi is git 159bfbdf8667bb01b1f4c85ea144ca128ccb3195

Created attachment 86406 [details]
callgrind data of 5 minutes cpu busy maildir resource

About 5 minute callgrind data with akonadi maildir resource being 100% cpu busy after Akonadi restart. As its still busy I can add another dump later on :)

Done as:

martin@merkaba:~> env | grep AKONADI
AKONADI_VALGRIND=akonadi_maildir_resource
AKONADI_VALGRIND_SKIN=callgrind
AKONADI_VALGRIND_OPTIONS=--instr-atstart=no --callgrind-out-file=/home/martin/callgrind-vcard.%p

martin@merkaba:~> callgrind_control -i on 
PID 28282: /usr/local/bin/akonadi_agent_launcher akonadi_maildir_resource akonadi_maildir_resource_0
sending command instrumentation on to pid 28282
  OK.

martin@merkaba:~> date
Fr 2. Mai 14:06:14 CEST 2014

martin@merkaba:~> date
Fr 2. Mai 14:11:14 CEST 2014
martin@merkaba:~> callgrind_control -d
PID 28282: /usr/local/bin/akonadi_agent_launcher akonadi_maildir_resource akonadi_maildir_resource_0

Created attachment 86408 [details]
complete callgrind run over a longer time with three dumps

one after 5 minutes, another one after 15 minutes (i.e. ten minutes later) and a third one before stopping the thing

Testing with:

diff --git a/resources/maildir/libmaildir/maildir.cpp b/resources/maildir/libmaildir/maildir.cpp
index 9bd3802..36166ed 100644
--- a/resources/maildir/libmaildir/maildir.cpp
+++ b/resources/maildir/libmaildir/maildir.cpp
@@ -106,12 +106,14 @@ public:
     QStringList listNew() const
     {
         QDir d( path + QString::fromLatin1( "/new" ) );
+        d.setSorting(QDir::NoSort);
         return d.entryList( QDir::Files );
     }
 
     QStringList listCurrent() const
     {
         QDir d( path + QString::fromLatin1( "/cur" ) );
+        d.setSorting(QDir::NoSort);
         return d.entryList( QDir::Files );
     }

which disables QDir sorting which was so prominent in callgrind traces. I get the impression it helps the bit, but it does not seem to fix all of the performance issue. Need to take more time to check more thoroughly.

Suggestion was from David Faure and Sergio.

Created attachment 86418 [details]
second callgrind of about well at least half an hour, after nodirsorting patch applied

This one looks a quite bit different. The resource still hogs a core, partly for minutes still, and KMail. But KMail does respond more often in between. I think KMail reacts after each complete folder synchronisation for a short time.

So do you think about this callgrind?

During this callgrind run KMail did respond in between, so I really think folder sync got faster by nodir sorting patch as during last callgrind of also about half an hour or so it didn´t respond once. Yet… no hard numbers yet. I don´t know how to make repeatable tests so far.

Sergio, I tried:

107     QStringList listNew() const
108     {
109         static int count = 0; count++;
110         static int elapsed = 0;
111 
112         QDir d( path + QString::fromLatin1( "/new" ) );
113         d.setSorting(QDir::NoSort);
114 
115         QTime t;
116         t.start();
117         QStringList list = d.entryList( QDir::Files );
118         elapsed += t.elapsed();
119         kDebug() <<  "PERF: listNew NoSort " << count << elapsed << "\n";
120         return list;
121 
122         //return d.entryList( QDir::Files );
123     }
124 
125     QStringList listCurrent() const                                                                                                                                
126     {
127         static int count = 0; count++;
128         static int elapsed = 0;
129 
130         QDir d( path + QString::fromLatin1( "/cur" ) );
131         d.setSorting(QDir::NoSort);
132 
133         QTime t;
134         t.start();
135         QStringList list = d.entryList( QDir::Files );
136         elapsed += t.elapsed();
137         kDebug() <<  "PERF: listCurrent NoSort " << count << elapsed << "\n";
138         return list;
139 
140         //return d.entryList( QDir::Files );
141     }

as well as your

-        std::cout <<  "PERF: listNew NoSort " << count << elapsed << "\n";
+        kDebug() <<  "PERF: listNew NoSort " << count << elapsed << "\n";

I see no output on konsole or in ~/.xsession-errors even with full debug enabled in kdebugdialog before starting akonadi again.

Additionally I look at the callgrind files again and I still think they are quite different.

The nodirsorting callgrind-vcard.26128.1 has:
- QDir::entryList
- QAlgorithmsPrivate::qSortHelper
- QAlgorithmsPrivate::qSortHelper
- QDirSortItemComparator::operator
- QString::compare

as first 5 out of first 6 entries. And none of it has a very long bar.

Yet the old callgrind has

- QAlgorithmsPrivat::qSortHelper, lr 145684.95, lr per call  as the first of the entries and it has a very long outstanding bar. This very long outstanding bar is basically gone in the new callgrind run. Here as directly from callgrind file:

So somewhere QDir may still sort something, but not at the original place anymore. At least thats my bet on looking at the data.

Sergio, I tried:

107     QStringList listNew() const
108     {
109         static int count = 0; count++;
110         static int elapsed = 0;
111 
112         QDir d( path + QString::fromLatin1( "/new" ) );
113         d.setSorting(QDir::NoSort);
114 
115         QTime t;
116         t.start();
117         QStringList list = d.entryList( QDir::Files );
118         elapsed += t.elapsed();
119         kDebug() <<  "PERF: listNew NoSort " << count << elapsed << "\n";
120         return list;
121 
122         //return d.entryList( QDir::Files );
123     }
124 
125     QStringList listCurrent() const                                                                                                                                
126     {
127         static int count = 0; count++;
128         static int elapsed = 0;
129 
130         QDir d( path + QString::fromLatin1( "/cur" ) );
131         d.setSorting(QDir::NoSort);
132 
133         QTime t;
134         t.start();
135         QStringList list = d.entryList( QDir::Files );
136         elapsed += t.elapsed();
137         kDebug() <<  "PERF: listCurrent NoSort " << count << elapsed << "\n";
138         return list;
139 
140         //return d.entryList( QDir::Files );
141     }

as well as your

-        std::cout <<  "PERF: listNew NoSort " << count << elapsed << "\n";
+        kDebug() <<  "PERF: listNew NoSort " << count << elapsed << "\n";

I see no output on konsole or in ~/.xsession-errors even with full debug enabled in kdebugdialog before starting akonadi again.

Additionally I look at the callgrind files again and I still think they are quite different.

The nodirsorting callgrind-vcard.26128.1 has:
- QDir::entryList
- QAlgorithmsPrivate::qSortHelper
- QAlgorithmsPrivate::qSortHelper
- QDirSortItemComparator::operator
- QString::compare

as first 5 out of first 6 entries. And none of it has a very long bar.

Yet the old callgrind-vcard.28282.1  has

- QAlgorithmsPrivat::qSortHelper, lr 145684.95, lr per call  as the first of the entries and it has a very long outstanding bar. This very long outstanding bar is basically gone in the new callgrind run. Just look at each

Just look at these two files with kcachegrind to me they look quite different in the old the sort stuff is dominating everything else by a factor of 10 at least, in the new it looks way more balanced.

So somewhere QDir may still sort something, but not at the original place anymore. At least thats my bet on looking at the data. I wonder whether something else is using QDir as well and did not disable sorting and we see this now.

I don´t know whether this is called and what that does, but setting nosort here as well for testing:

diff --git a/resources/maildir/libmaildir/keycache.cpp b/resources/maildir/libmaildir/keycache.cpp
index f0af9c4..814ce6c 100644
--- a/resources/maildir/libmaildir/keycache.cpp
+++ b/resources/maildir/libmaildir/keycache.cpp
@@ -75,12 +75,14 @@ bool KeyCache::isNewKey( const QString& dir, const QString& key ) const
 QSet< QString > KeyCache::listNew( const QString& dir ) const
 {
     QDir d( dir + QString::fromLatin1( "/new" ) );
+    d.setSorting(QDir::NoSort);
     return d.entryList( QDir::Files ).toSet();
 }
 
 QSet< QString > KeyCache::listCurrent( const QString& dir ) const
 {
     QDir d( dir + QString::fromLatin1( "/cur"  ) );
+    d.setSorting(QDir::NoSort);
     return d.entryList( QDir::Files ).toSet();
 }

Created attachment 86428 [details]
callgrind with nodirsorting also in libmaildir/keycache.cpp

Well this is it. KMail is usable even with make -j4 on kdepim-runtime on a hyperthreaded Sandybridge mobile dual core with that. Akonadi maildir resource still appears with some CPU usage, but I didn´t see it at 100% for even just one 10 second measurement period in atop.

callgrind logs show a different bottleneck now: MaildirResource::listRecursive

To me these two nosorting patches have such a huge impact on Maildir resource performance that using KMail is a completely different experience now. KMail hardly ever blocks on folder changes anymore for example, although that seems just the case due to synchronisation being much faster now. So this still is good to *fix*.

Review Request:

Change libmaildir listNew and listCurrent functions to explicetely not sort QDir.entryList
https://git.reviewboard.kde.org/r/117975/

Git commit 4dd5cb7fcb37f9da481d8324421428a0028e9e2d by Martin Steigerwald.
Committed on 03/05/2014 at 14:52.
Pushed by steigerwald into branch 'master'.

Do not sort the directory entry list in listNew() and listCurrent():

According to callgrind dumps at

https://bugs.kde.org/show_bug.cgi?id=334218#c4

QDir sorts the list of directory entries unless specified otherwise.

After disabling the sorting the callgrind is quite different already:

https://bugs.kde.org/show_bug.cgi?id=334218#c7

This already helps shortening the time to synchronize folders visibly,
but KMail gets still blocked for half a minute or more. There is another
occurence of QDir entry sorting in keycache.cpp which the next commit
will address.

Thanks to Sergio and David for help and pointing out how to disable
the sorting.

REVIEW: 117975

DIGEST: Huge performance improvement for POP3 users with large maildir.

M  +2    -0    resources/maildir/libmaildir/maildir.cpp

http://commits.kde.org/kdepim-runtime/4dd5cb7fcb37f9da481d8324421428a0028e9e2d

Git commit 41be4d27dab49d600e3bd93cfc3804a347243257 by Martin Steigerwald.
Committed on 03/05/2014 at 14:53.
Pushed by steigerwald into branch 'master'.

Also do not sort directory entries in listCurrent and listNew in key cache:

QDir sorts the list of directory entries unless specified otherwise.
The last commit already disabled the sorting in maildir.cpp.

This commit completes the work and makes the remaining sorting calls
in the callgrind dumps at

https://bugs.kde.org/show_bug.cgi?id=334218#c7

go away completely as demonstrated in the callgrind dumps at:

https://bugs.kde.org/show_bug.cgi?id=334218#c12

Subjectively this has a huge impact on the performance of KMail with huge
maildir. KMail hardly blocks anymore on folder changes and feels much more
responsive now.

Akonadi maildir resource went from hogging a Sandy Bridge core for minutes
to not appearing using 100% of one core in even one averaged 10 second
interval in atop. I hardly see it in atop at all anymore, except when
accessing very large folders such as one with Linux Kernel mailing list
and more than 245000 unread mails.

Thanks to Sergio for pointing out the remaining sorting calls in the second
callgrind dump and for help.

Now the bottleneck appears to be MaildirResource::listRecursive but I do
not know whether further optimization is necessary at this point.

Both changes tested with kdepimlibs master as of today and KMail 4.12.4
from Debian unstable packages with an enormous maildir including a
folder for Linux Kernel mailing list with more than 245000 unread mails.

REVIEW: 117975

DIGEST: Huge performance improvement for POP3 users with large maildirs.

M  +2    -0    resources/maildir/libmaildir/keycache.cpp

http://commits.kde.org/kdepim-runtime/41be4d27dab49d600e3bd93cfc3804a347243257

Git commit 1af2704a63c6641867eaf2b67976f5309a994cca by Christian Mollekopf, on behalf of Martin Steigerwald.
Committed on 03/05/2014 at 14:52.
Pushed by cmollekopf into branch 'kolab/integration/4.13.0'.

Do not sort the directory entry list in listNew() and listCurrent():

According to callgrind dumps at

https://bugs.kde.org/show_bug.cgi?id=334218#c4

QDir sorts the list of directory entries unless specified otherwise.

After disabling the sorting the callgrind is quite different already:

https://bugs.kde.org/show_bug.cgi?id=334218#c7

This already helps shortening the time to synchronize folders visibly,
but KMail gets still blocked for half a minute or more. There is another
occurence of QDir entry sorting in keycache.cpp which the next commit
will address.

Thanks to Sergio and David for help and pointing out how to disable
the sorting.

REVIEW: 117975

DIGEST: Huge performance improvement for POP3 users with large maildir.

M  +2    -0    resources/maildir/libmaildir/maildir.cpp

http://commits.kde.org/kdepim-runtime/1af2704a63c6641867eaf2b67976f5309a994cca

Git commit 81f30e18c975f9cc44da7c904ed612df804200da by Christian Mollekopf, on behalf of Martin Steigerwald.
Committed on 03/05/2014 at 14:53.
Pushed by cmollekopf into branch 'kolab/integration/4.13.0'.

Also do not sort directory entries in listCurrent and listNew in key cache:

QDir sorts the list of directory entries unless specified otherwise.
The last commit already disabled the sorting in maildir.cpp.

This commit completes the work and makes the remaining sorting calls
in the callgrind dumps at

https://bugs.kde.org/show_bug.cgi?id=334218#c7

go away completely as demonstrated in the callgrind dumps at:

https://bugs.kde.org/show_bug.cgi?id=334218#c12

Subjectively this has a huge impact on the performance of KMail with huge
maildir. KMail hardly blocks anymore on folder changes and feels much more
responsive now.

Akonadi maildir resource went from hogging a Sandy Bridge core for minutes
to not appearing using 100% of one core in even one averaged 10 second
interval in atop. I hardly see it in atop at all anymore, except when
accessing very large folders such as one with Linux Kernel mailing list
and more than 245000 unread mails.

Thanks to Sergio for pointing out the remaining sorting calls in the second
callgrind dump and for help.

Now the bottleneck appears to be MaildirResource::listRecursive but I do
not know whether further optimization is necessary at this point.

Both changes tested with kdepimlibs master as of today and KMail 4.12.4
from Debian unstable packages with an enormous maildir including a
folder for Linux Kernel mailing list with more than 245000 unread mails.

REVIEW: 117975

DIGEST: Huge performance improvement for POP3 users with large maildirs.

M  +2    -0    resources/maildir/libmaildir/keycache.cpp

http://commits.kde.org/kdepim-runtime/81f30e18c975f9cc44da7c904ed612df804200da

Git commit afdc4bc8d85dbadfddd5d89169fede0f5a57317d by Martin Steigerwald.
Committed on 03/05/2014 at 14:52.
Pushed by steigerwald into branch 'KDE/4.13'.

Do not sort the directory entry list in listNew() and listCurrent():

According to callgrind dumps at

https://bugs.kde.org/show_bug.cgi?id=334218#c4

QDir sorts the list of directory entries unless specified otherwise.

After disabling the sorting the callgrind is quite different already:

https://bugs.kde.org/show_bug.cgi?id=334218#c7

This already helps shortening the time to synchronize folders visibly,
but KMail gets still blocked for half a minute or more. There is another
occurence of QDir entry sorting in keycache.cpp which the next commit
will address.

Thanks to Sergio and David for help and pointing out how to disable
the sorting.

REVIEW: 117975

DIGEST: Huge performance improvement for POP3 users with large maildir.

M  +2    -0    resources/maildir/libmaildir/maildir.cpp

http://commits.kde.org/kdepim-runtime/afdc4bc8d85dbadfddd5d89169fede0f5a57317d

Git commit ea0b761fbe3c43b42f83c5477018e10209f92ea3 by Martin Steigerwald.
Committed on 03/05/2014 at 14:53.
Pushed by steigerwald into branch 'KDE/4.13'.

Also do not sort directory entries in listCurrent and listNew in key cache:

QDir sorts the list of directory entries unless specified otherwise.
The last commit already disabled the sorting in maildir.cpp.

This commit completes the work and makes the remaining sorting calls
in the callgrind dumps at

https://bugs.kde.org/show_bug.cgi?id=334218#c7

go away completely as demonstrated in the callgrind dumps at:

https://bugs.kde.org/show_bug.cgi?id=334218#c12

Subjectively this has a huge impact on the performance of KMail with huge
maildir. KMail hardly blocks anymore on folder changes and feels much more
responsive now.

Akonadi maildir resource went from hogging a Sandy Bridge core for minutes
to not appearing using 100% of one core in even one averaged 10 second
interval in atop. I hardly see it in atop at all anymore, except when
accessing very large folders such as one with Linux Kernel mailing list
and more than 245000 unread mails.

Thanks to Sergio for pointing out the remaining sorting calls in the second
callgrind dump and for help.

Now the bottleneck appears to be MaildirResource::listRecursive but I do
not know whether further optimization is necessary at this point.

Both changes tested with kdepimlibs master as of today and KMail 4.12.4
from Debian unstable packages with an enormous maildir including a
folder for Linux Kernel mailing list with more than 245000 unread mails.

REVIEW: 117975

DIGEST: Huge performance improvement for POP3 users with large maildirs.

M  +2    -0    resources/maildir/libmaildir/keycache.cpp

http://commits.kde.org/kdepim-runtime/ea0b761fbe3c43b42f83c5477018e10209f92ea3

Git commit 431ae9b185aa9e74a6b822fbee44e4b93b3f1498 by Martin Steigerwald.
Committed on 29/05/2014 at 15:49.
Pushed by steigerwald into branch 'master'.

Manually sort the QStringList for the readEntryFlags test

Otherwise the individual test in this testcase may work on the wrong
mails as libmaildir doesn´t sort the entry list anymore.

REVIEW: 118400

M  +4    -0    resources/maildir/libmaildir/tests/testmaildir.cpp

http://commits.kde.org/kdepim-runtime/431ae9b185aa9e74a6b822fbee44e4b93b3f1498

Git commit c6659e74196b74ebe32254cf0b42d9ed4274d17c by Martin Steigerwald.
Committed on 31/05/2014 at 18:18.
Pushed by steigerwald into branch 'master'.

Manually sort the item list so that tagListHash.contains tests work

Otherwise the individual tests in this testcase may work on the wrong
mails as libmaildir doesn´t sort the entry list anymore.

This is related to the nosorting change from review request 117975.

This needs to go into 4.13 branch as well before release of 4.13.2.

REVIEW: 118441

M  +5    -0    resources/mixedmaildir/tests/itemfetchtest.cpp

http://commits.kde.org/kdepim-runtime/c6659e74196b74ebe32254cf0b42d9ed4274d17c

Git commit 2b8c7db15c11c6d76a691b831fe83dc0879872ab by Martin Steigerwald.
Committed on 29/05/2014 at 15:49.
Pushed by steigerwald into branch 'KDE/4.13'.

Manually sort the QStringList for the readEntryFlags test

Otherwise the individual test in this testcase may work on the wrong
mails as libmaildir doesn´t sort the entry list anymore.

REVIEW: 118400

M  +4    -0    resources/maildir/libmaildir/tests/testmaildir.cpp

http://commits.kde.org/kdepim-runtime/2b8c7db15c11c6d76a691b831fe83dc0879872ab

Git commit 6428954599ce87fd3844a43e3581aec47d4033ee by Martin Steigerwald.
Committed on 31/05/2014 at 18:18.
Pushed by steigerwald into branch 'KDE/4.13'.

Manually sort the item list so that tagListHash.contains tests work

Otherwise the individual tests in this testcase may work on the wrong
mails as libmaildir doesn´t sort the entry list anymore.

This is related to the nosorting change from review request 117975.

This needs to go into 4.13 branch as well before release of 4.13.2.

REVIEW: 118441

M  +5    -0    resources/mixedmaildir/tests/itemfetchtest.cpp

http://commits.kde.org/kdepim-runtime/6428954599ce87fd3844a43e3581aec47d4033ee

Git commit ceb7bdd423f5ed148b53f422795a167df48ab99e by Martin Steigerwald.
Committed on 08/06/2014 at 16:34.
Pushed by steigerwald into branch 'master'.

Sort by filename, which is RemoteId, instead of item id to make test pass.

Review request 118441 introduced sorting of items for ItemFetchJob
testcase as due to an performance optimization mixedmaildir resource
does not sort anymore. This fixed the testcase for me on BTRFS.

However on the Jenkins server the test case was still not fixed and
Kevin pointed out that qSort may sort by item id instead of RemoteId.
The test relies on sorting by filename, which is basically the RemoteId.

REVIEW: 118619

M  +8    -1    resources/mixedmaildir/tests/itemfetchtest.cpp

http://commits.kde.org/kdepim-runtime/ceb7bdd423f5ed148b53f422795a167df48ab99e

Git commit 79ca2264b5c210ec02f4240b161477d5d9314d25 by Martin Steigerwald.
Committed on 08/06/2014 at 16:34.
Pushed by steigerwald into branch 'KDE/4.13'.

Sort by filename, which is RemoteId, instead of item id to make test pass.

Review request 118441 introduced sorting of items for ItemFetchJob
testcase as due to an performance optimization mixedmaildir resource
does not sort anymore. This fixed the testcase for me on BTRFS.

However on the Jenkins server the test case was still not fixed and
Kevin pointed out that qSort may sort by item id instead of RemoteId.
The test relies on sorting by filename, which is basically the RemoteId.

REVIEW: 118619

M  +8    -1    resources/mixedmaildir/tests/itemfetchtest.cpp

http://commits.kde.org/kdepim-runtime/79ca2264b5c210ec02f4240b161477d5d9314d25

While this may be able to be tuned further the original hogging of a Sandybridge core for minutes is definately gone, currently using Akonadi 1.13 current master, kdepimlibs and kdepim-runtime from 24th of February and KMail 4.14.2. Akonadi 1.13 has the MySQL performance improvements.

I think it may still do needless work on maildir synchronisation, but I have other bugs open for that.

Thus closing in my attempt at which of my bug reports are still happening and matter in current versions.