Summary: | synchronizing large about 77000 mail Linux kernel mailing list folder blocks out KMail for more than 15 minutes | ||
---|---|---|---|
Product: | [Frameworks and Libraries] Akonadi | Reporter: | Martin Steigerwald <Martin> |
Component: | Maildir Resource | Assignee: | Sergio Martins <smartins> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | amantia, cfeck, kdepim-bugs, smartins |
Priority: | NOR | ||
Version: | 1.9.2 | ||
Target Milestone: | --- | ||
Platform: | Debian unstable | ||
OS: | Linux | ||
Latest Commit: | http://commits.kde.org/kdepim-runtime/d161e37622f8acb550d54cacd48f4886354fc1e5 | Version Fixed In: |
Description
Martin Steigerwald
2013-07-04 09:31:28 UTC
martin@merkaba:~> ps aux | grep 7234 | grep -v grep martin 7234 1.5 1.9 466260 152452 ? Sl Jul03 32:58 /usr/bin/akonadi_agent_launcher akonadi_maildir_resource akonadi_maildir_resource_0 I think I try archiving mails to mbox´es with mixed mail dir resource just as I did with KMail 1. I wonder that Akonadi will do if I through the existing archive of more than a million of mails at it. I would like to test this. Is there an easy way for me to get such a big folder ? Well subscribe the Linux kernel mailing list. Anyway, I am currently tar --xz -cf´ing the folder. I will send you a mail where I placed it. It shouldn´t contain any private stuff, but I still feel more comfortable sharing it with you directly. I would upload it somewhere onto my server. If thats not enough I could try the same with my complete Linux and/or Debian folders which contain huge amounts of mails. :) Thanks, Martin Sergio, thank you for your interesting to look into this. I sent you a mail with link to upload. Note: Akonadi stuff and maildir is on BTRFS on my system, but I pretty much think that does not matter here, as the workload is clearly not I/O bound: merkaba:~> atopsar -d -b 0:00 -e 1:00 merkaba 3.10.0-tp520 #18 SMP PREEMPT Tue Jul 2 09:41:49 CEST 2013 x86_64 2013/07/04 -------------------------- analysis date: 2013/07/04 -------------------------- 00:00:02 disk busy read/s KB/read writ/s KB/writ avque avserv _dsk_ 00:10:02 sdb 0% 0.0 0.0 0.0 0.0 0.0 0.00 ms sda 2% 66.8 34.3 17.6 16.1 12.0 0.26 ms 00:20:02 sda 0% 0.1 4.0 9.5 8.2 18.9 0.07 ms 00:30:02 sda 1% 27.7 22.2 16.8 18.4 8.7 0.28 ms 00:36:21 sda 1% 32.0 12.8 24.6 20.3 12.4 0.18 ms merkaba:~> atopsar -d -b 10:30 -e 11:30 merkaba 3.10.0-tp520 #18 SMP PREEMPT Tue Jul 2 09:41:49 CEST 2013 x86_64 2013/07/04 -------------------------- analysis date: 2013/07/04 -------------------------- 10:51:12 disk busy read/s KB/read writ/s KB/writ avque avserv _dsk_ 11:01:12 sdb 0% 0.0 0.0 0.0 0.0 0.0 0.00 ms sda 6% 225.2 35.9 28.4 27.1 23.3 0.25 ms 11:11:12 sda 2% 70.5 27.3 27.9 21.2 23.2 0.19 ms 11:21:12 sda 7% 217.4 22.9 24.8 31.7 11.0 0.29 ms sda is the primary Intel SSD 320 where data resided. sdb is a new Intel mSATA SSD that I do not yet use. Just killed it, it started to use 2GB of memory. How's memory consumption for you ? Hmmm, I didn´t look closely. I will have a look with atop data from today. 2013/07/04 00:30:02 PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK VSIZE RSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD 1/15 7234 - 1729 299 76K 67972K 222.2M 136K 455.3M 142.9M 0K 8164K 15212K martin martin 2% akonadi_agent_ 15622 - 14418 11 16K 128.0M 1.5G 220K 2.1G 127.8M 87152K -76K 4400K martin martin 2% kmail Resident Set Size 142 MB for the agent and 127 MB for kmail. 2013/07/04 10:51:12 and from there in 10 minute steps: PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK VSIZE RSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD 1/15 7234 - 260080 3172 76K 67972K 222.2M 136K 455.3M 102.8M 455.3M 102.8M 56260K martin martin 1% akonadi_agent_ 7234 - 93235 363 76K 67972K 222.2M 136K 455.3M 146.2M 0K 44496K 11860K martin martin 2% akonadi_agent_ 7234 - 74297 14 76K 67972K 222.3M 136K 455.5M 147.9M 128K 1760K 10732K martin martin 2% akonadi_agent_ 7234 - 14661 19 76K 67972K 222.2M 136K 455.3M 148.9M -128K 952K 9652K martin martin 2% akonadi_agent_ Again about 148 MB RSIZE. I put the atop binary log aside for further analysis if needed. I think first RGROW is large due to waking up from hibernation. Maybe hibernating triggered swapping akonadi mail dir agent out. Sergio, one idea regarding memory usage, I had mail indexing disabled as that happens, so maybe try adding the folder with mail indexing disabled gives a lower memory usage? Thanks, Martin Sergio, if you could test http://pastebin.com/5CamSj5b would be great. (In reply to comment #9) > Sergio, if you could test http://pastebin.com/5CamSj5b would be great. This paste has been removed Patch posted as http://git.reviewboard.kde.org/r/113918/ . András and Sergio, thanks for looking into this. In case you want to try with an even larger data set I now have 150000+ mails in that folder that I could pack together and put onto my server. Actually with KDEPIM SC 4.11.3 and Akonadi 1.10.2 it even works quite well here. Scalabity issues are more with the list view in KMail, especially with the thread sorting at the moment. Right now while accessing the folder KMail seems to be responsive otherwise. Lets see. I think you actually may have done some improvements there already. Git commit 6100bbf86a5ddf5ae009fb6724d4d2ac53593c7f by Andras Mantia. Committed on 18/11/2013 at 16:18. Pushed by amantia into branch 'master'. Use QDirIterator for listing the maildir folders. This: - avoids some extra stats - avoids calling into Maildir::findRealKey that caches the keys and uses a lot of memory for no real reason (in this case) REVIEW: 113918 M +16 -0 resources/maildir/libmaildir/maildir.cpp M +6 -0 resources/maildir/libmaildir/maildir.h M +32 -23 resources/maildir/retrieveitemsjob.cpp M +3 -3 resources/maildir/retrieveitemsjob.h http://commits.kde.org/kdepim-runtime/6100bbf86a5ddf5ae009fb6724d4d2ac53593c7f Will this be merged/backported into 4.11/4.12 release branches? If not, it would be released starting with KDE 4.13. I will backport it, just didn't want to do right away before some more feedback and testing. Git commit d161e37622f8acb550d54cacd48f4886354fc1e5 by Andras Mantia. Committed on 18/11/2013 at 16:18. Pushed by amantia into branch 'KDE/4.12'. Use QDirIterator for listing the maildir folders. This: - avoids some extra stats - avoids calling into Maildir::findRealKey that caches the keys and uses a lot of memory for no real reason (in this case) REVIEW: 113918 (cherry picked from commit 6100bbf86a5ddf5ae009fb6724d4d2ac53593c7f) M +16 -0 resources/maildir/libmaildir/maildir.cpp M +6 -0 resources/maildir/libmaildir/maildir.h M +32 -23 resources/maildir/retrieveitemsjob.cpp M +3 -3 resources/maildir/retrieveitemsjob.h http://commits.kde.org/kdepim-runtime/d161e37622f8acb550d54cacd48f4886354fc1e5 |