On looking whether mixedmaildir resource needs a similar fix as in Bug #334206 - While maildir resources synchronizes a folder KMail blocks on switching to a different folder I thought I try to through my old mixedmaildir archive into Akonadi. During KMail 1 times I used mbox folders to archive mails from maildir folders at the time when KMail 1 became to slow with large folders (i.e. about 20000-40000 mails in one folder was the usability limit back then on my old laptop). Well I did and on trying mixedmailresource consumed memory until the Linux kernel OOM killed it. Reproducible: Always Steps to Reproduce: 1. Have a large mixedmail folder. 2. Add a mixed maildir resource in Akonadiconsole and point it to that folder 3. Watch machine becomes sluggish until OOM killer steps in. Actual Results: 6.1 GB Resident Set usage on a 8 GB machine. PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK VSIZE RSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD 1/7 6132 - 142595 41 328K 77724K 6.3G 136K 6.6G 5.4G 556.1M 95936K 903.6M martin martin 71% akonadi_mixedm PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK VSIZE RSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD 1/7 6222 - 140411 20 328K 77064K 6.5G 136K 6.8G 6.1G 547.8M 50216K 485.4M martin martin 80% akonadi_mixedm OOM killer: [103900.424539] Out of memory: Kill process 6222 (akonadi_mixedma) score 809 or sacrifice child [103900.424547] Killed process 6222 (akonadi_mixedma) total-vm:16966204kB, anon-rss:6681748kB, file-rss:1352kB Expected Results: Sane memory usage during scan of mixed maildir archive folder. Yes, it is large, but still, it doesn´t have to load it all into memory at the same time. martin@merkaba:~> cat /proc/version Linux version 3.14.0-tp520 (martin@merkaba) (gcc version 4.8.2 (Debian 4.8.2-17) ) #52 SMP PREEMPT Mon Mar 31 13:41:48 CEST 2014 martin@merkaba:~> free -m total used free shared buffers cached Mem: 7767 2771 4996 326 0 1331 -/+ buffers/cache: 1440 6327 Swap: 12287 2114 10173 Akonadi, kdepimlibs, kdepim-runtime are git master as of today. KMail is 4.12.4 as from Debian unstable package.
Created attachment 86447 [details] kern.log with OOM killer stopping mixedmaildir resource good that it did :)
Some data of mixed maildir directory that triggers this behavior: 643M .Amiga.directory 4,0K .Amiga.index 4,0K .Amiga.index.ids 278M .Computer.directory 4,0K .Computer.index 4,0K .Computer.index.ids 2,9G .Debian.directory 4,0K .Debian.index 4,0K .Debian.index.ids 4,0K .hplip-help-ml.index 4,0K .hplip-help-ml.index.ids 69M .KDE.directory 256K .KDE.index 4,0K .KDE.index.ids 6,1G .Linux.directory 4,0K .Linux.index 4,0K .Linux.index.ids 205M .Mitwelt.directory 4,0K .Mitwelt.index 4,0K .Mitwelt.index.ids 10G insgesamt In folders are several mbox files, sorted by year, example: martin@merkaba:~/.local/share/local-mail-archive/.Lichtvoll-Archiv.directory> du -sch .Linux.directory/* 12M .Linux.directory/ck-patch-ml-2006-2008 86M .Linux.directory/fsdevel-ml-2008-2009 144M .Linux.directory/fsdevel-ml-2010-2011 35M .Linux.directory/fsdevel-ml-2012-2013 20M .Linux.directory/hplip-help-ml-2007 288M .Linux.directory/kernel-ml-2008 249M .Linux.directory/kernel-ml-2009-1 322M .Linux.directory/kernel-ml-2009-2 532M .Linux.directory/kernel-ml-2009-3 808M .Linux.directory/kernel-ml-2010-1 674M .Linux.directory/kernel-ml-2010-2 154M .Linux.directory/kernel-ml-2011-1 484M .Linux.directory/kernel-ml-2011-2 620M .Linux.directory/kernel-ml-2012-1 691M .Linux.directory/kernel-ml-2012-2 255M .Linux.directory/kernel-ml-2013-1 15M .Linux.directory/lug-frankfurt-ml 2,1M .Linux.directory/savage40-ml-2004-2009 11M .Linux.directory/suspend2-users-ml-2006 247M .Linux.directory/thinkpad-ml-2005-2009 78M .Linux.directory/xfs-ml-2006-2008 5,6G insgesamt I can´t share this as is, at least the Amiga directory contains confidential stuff. I can share the Linux, KDE and Debian directories I think. Maybe except lug-frankfurt-ml and some folders.
I tried AKONADI_VALGRIND=akonadi_mixedmaildir_resource AKONADI_VALGRIND_SKIN=memcheck and after a short time got before quitting Akonadi server again after having trigger resync of mixedmaildir resource I got: ==7016== ==7016== HEAP SUMMARY: ==7016== in use at exit: 4,983,818 bytes in 92,727 blocks ==7016== total heap usage: 19,700,956 allocs, 19,608,229 frees, 210,374,310,427 bytes allocated ==7016== ==7016== LEAK SUMMARY: ==7016== definitely lost: 6,656 bytes in 26 blocks ==7016== indirectly lost: 2,151 bytes in 101 blocks ==7016== possibly lost: 640,874 bytes in 13,258 blocks ==7016== still reachable: 4,334,137 bytes in 79,342 blocks ==7016== suppressed: 0 bytes in 0 blocks ==7016== Rerun with --leak-check=full to see details of leaked memory ==7016== ==7016== For counts of detected and suppressed errors, rerun with: -v ==7016== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2) ProcessControl: Application valgrind stopped unexpectedly ( "Process crashed" ) Application 'valgrind' crashed. No restart! I think I only get it on exit... would that work? Well I can leave it running for a while and see.
Well got more information with AKONADI_VALGRIND_OPTIONS=--leak-check=full but still only about 6500 bytes left... so either I need to have this run for a rather long time… or I need a different approach. As valgrind seems to detect errors at exit time, it possibly doesn´t even catch the runtime usage. As technically spoken this may not be a leak, but "regular" memory usage. Please advice.
This bug has only been reported for versions older than KDEPIM 4.14 (at most akonadi-1.3). Can anyone tell if this bug still present? If noone confirms this bug for a recent version of akonadi (part of KDE Applications 15.08 or later), it gets closed in about three months.
Just as announced in my last comment, I close this bug. If you encounter it again in a recent version (at least 5.0 aka 15.08), please open a new one unless it already exists. Thank you for all your input.