Bug 334318

Summary:	on adding old mail archive mixedmaildir resource consumes tons of memory
Product:	[Frameworks and Libraries] Akonadi	Reporter:	Martin Steigerwald <Martin>
Component:	Mixed Maildir resource	Assignee:	kdepim bugs <kdepim-bugs>
Status:	RESOLVED UNMAINTAINED
Severity:	grave	CC:	krammer
Priority:	NOR
Version:	GIT (master)
Target Milestone:	---
Platform:	Debian unstable
OS:	Linux
Latest Commit:		Version Fixed In:
Sentry Crash Report:
Attachments:	kern.log with OOM killer stopping mixedmaildir resource

Description Martin Steigerwald 2014-05-04 10:01:05 UTC

On looking whether mixedmaildir resource needs a similar fix as in

Bug #334206 - While maildir resources synchronizes a folder KMail blocks on switching to a different folder

I thought I try to through my old mixedmaildir archive into Akonadi. During KMail 1 times I used mbox folders to archive mails from maildir folders at the time when KMail 1 became to slow with large folders (i.e. about 20000-40000 mails in one folder was the usability limit back then on my old laptop).

Well I did and on trying mixedmailresource consumed memory until the Linux kernel OOM killed it.

Reproducible: Always

Steps to Reproduce:
1. Have a large mixedmail folder.
2. Add a mixed maildir resource in Akonadiconsole and point it to that folder
3. Watch machine becomes sluggish until OOM killer steps in.
Actual Results:
6.1 GB Resident Set usage on a 8 GB machine.

PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK VSIZE RSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD 1/7
6132 - 142595 41 328K 77724K 6.3G 136K 6.6G 5.4G 556.1M 95936K 903.6M martin martin 71% akonadi_mixedm

PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK VSIZE RSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD 1/7
6222 - 140411 20 328K 77064K 6.5G 136K 6.8G 6.1G 547.8M 50216K 485.4M martin martin 80% akonadi_mixedm

OOM killer:

[103900.424539] Out of memory: Kill process 6222 (akonadi_mixedma) score 809 or sacrifice child
[103900.424547] Killed process 6222 (akonadi_mixedma) total-vm:16966204kB, anon-rss:6681748kB, file-rss:1352kB

Expected Results:
Sane memory usage during scan of mixed maildir archive folder. Yes, it is large, but still, it doesn´t have to load it all into memory at the same time.

martin@merkaba:~> cat /proc/version
Linux version 3.14.0-tp520 (martin@merkaba) (gcc version 4.8.2 (Debian 4.8.2-17) ) #52 SMP PREEMPT Mon Mar 31 13:41:48 CEST 2014
martin@merkaba:~> free -m
total used free shared buffers cached
Mem: 7767 2771 4996 326 0 1331
-/+ buffers/cache: 1440 6327
Swap: 12287 2114 10173

Akonadi, kdepimlibs, kdepim-runtime are git master as of today. KMail is 4.12.4 as from Debian unstable package.

Comment 1 Martin Steigerwald 2014-05-04 10:04:17 UTC

Created attachment 86447 [details]
kern.log with OOM killer stopping mixedmaildir resource

good that it did :)

Comment 2 Martin Steigerwald 2014-05-04 10:09:08 UTC

Some data of mixed maildir directory that triggers this behavior:

643M    .Amiga.directory
4,0K    .Amiga.index
4,0K    .Amiga.index.ids
278M    .Computer.directory
4,0K    .Computer.index
4,0K    .Computer.index.ids
2,9G    .Debian.directory
4,0K    .Debian.index
4,0K    .Debian.index.ids
4,0K    .hplip-help-ml.index
4,0K    .hplip-help-ml.index.ids
69M     .KDE.directory
256K    .KDE.index
4,0K    .KDE.index.ids
6,1G    .Linux.directory
4,0K    .Linux.index
4,0K    .Linux.index.ids
205M    .Mitwelt.directory
4,0K    .Mitwelt.index
4,0K    .Mitwelt.index.ids
10G     insgesamt

In folders are several mbox files, sorted by year, example:

martin@merkaba:~/.local/share/local-mail-archive/.Lichtvoll-Archiv.directory> du -sch .Linux.directory/*
12M     .Linux.directory/ck-patch-ml-2006-2008
86M     .Linux.directory/fsdevel-ml-2008-2009
144M    .Linux.directory/fsdevel-ml-2010-2011
35M     .Linux.directory/fsdevel-ml-2012-2013
20M     .Linux.directory/hplip-help-ml-2007
288M    .Linux.directory/kernel-ml-2008
249M    .Linux.directory/kernel-ml-2009-1
322M    .Linux.directory/kernel-ml-2009-2
532M    .Linux.directory/kernel-ml-2009-3
808M    .Linux.directory/kernel-ml-2010-1
674M    .Linux.directory/kernel-ml-2010-2
154M    .Linux.directory/kernel-ml-2011-1
484M    .Linux.directory/kernel-ml-2011-2
620M    .Linux.directory/kernel-ml-2012-1
691M    .Linux.directory/kernel-ml-2012-2
255M    .Linux.directory/kernel-ml-2013-1
15M     .Linux.directory/lug-frankfurt-ml
2,1M    .Linux.directory/savage40-ml-2004-2009
11M     .Linux.directory/suspend2-users-ml-2006
247M    .Linux.directory/thinkpad-ml-2005-2009
78M     .Linux.directory/xfs-ml-2006-2008
5,6G    insgesamt


I can´t share this as is, at least the Amiga directory contains confidential stuff. I can share the Linux, KDE and Debian directories I think. Maybe except lug-frankfurt-ml and some folders.

Comment 3 Martin Steigerwald 2014-05-04 10:26:29 UTC

I tried

AKONADI_VALGRIND=akonadi_mixedmaildir_resource
AKONADI_VALGRIND_SKIN=memcheck

and after a short time got before quitting Akonadi server again after having trigger resync of mixedmaildir resource I got:

==7016== 
==7016== HEAP SUMMARY:
==7016==     in use at exit: 4,983,818 bytes in 92,727 blocks
==7016==   total heap usage: 19,700,956 allocs, 19,608,229 frees, 210,374,310,427 bytes allocated
==7016== 
==7016== LEAK SUMMARY:
==7016==    definitely lost: 6,656 bytes in 26 blocks
==7016==    indirectly lost: 2,151 bytes in 101 blocks
==7016==      possibly lost: 640,874 bytes in 13,258 blocks
==7016==    still reachable: 4,334,137 bytes in 79,342 blocks
==7016==         suppressed: 0 bytes in 0 blocks
==7016== Rerun with --leak-check=full to see details of leaked memory
==7016== 
==7016== For counts of detected and suppressed errors, rerun with: -v
==7016== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
ProcessControl: Application valgrind stopped unexpectedly ( "Process crashed" )
Application 'valgrind' crashed. No restart!

I think I only get it on exit... would that work? Well I can leave it running for a while and see.

Comment 4 Martin Steigerwald 2014-05-04 10:59:28 UTC

Well got more information with

AKONADI_VALGRIND_OPTIONS=--leak-check=full

but still only about 6500 bytes left... so either I need to have this run for a rather long time… or I need a different approach. As valgrind seems to detect errors at exit time, it possibly doesn´t even catch the runtime usage. As technically spoken this may not be a leak, but "regular" memory usage.

Please advice.

Comment 5 Denis Kurz 2016-09-24 20:38:56 UTC

This bug has only been reported for versions older than KDEPIM 4.14 (at most akonadi-1.3). Can anyone tell if this bug still present?

If noone confirms this bug for a recent version of akonadi (part of KDE Applications 15.08 or later), it gets closed in about three months.

Comment 6 Denis Kurz 2017-01-07 21:44:42 UTC

Just as announced in my last comment, I close this bug. If you encounter it again in a recent version (at least 5.0 aka 15.08), please open a new one unless it already exists. Thank you for all your input.