Bug 166653

Maybe KMail thinks that your index files are out of date and recreates them.
If the index is more than 5 seconds older than the newest mail in maildir or the mbox modification date, it will recreate the index.

Please check if that happens by running KMail from a console and check if the debug output says something about this.

Are your folders on a NFS server? Do you have uncommon clock problems (maybe NTP)?

Thanks for your comments. All folders are on a local disk. I understand what you are saying about the index and mbox modification dates, but the problem seems to be that kmail modifies the maildir and mbox (or sets the date) even when they don't change at all (no new mails or other activity)?

Actually it looks as if my bug is the same as 166634? I also found an older bug 66409 that says that all mboxes and maildirs are touched every time.

The output from console is attached, but I don't find it particularly enlightening. Please let me know if there is anything else I can try.

Created attachment 26167 [details]
kmail console output

One additional information: This problem started with the upgrade to kde 4.1. I've used kmail for years and never had an issue with undue touching before (including KDE 4.0).

I experience the same problem. After startup and also at some later times (the pattern is not obvious to me), kmail becomes unusable for several minutes during which continuous hard disk activity is indicated. Investigation showed that CPU and memory usage is around 10 to 20 percent during that time; apparently the hard disk access is the major bottleneck. File modification times of indexing files in the mail directories confirm that their update spawns over more than 5 minutes each time. The problem occurs since my update to KMail 1.10.3 (KDE 4.1 Kubuntu) and makes it really hard to continue using this tool for real work.

My maildir directory is around 1.8G large and contains a total of 53957 files. Indexing (as expected) takes especially long on large folders. Those are typically folders for mailing lists which are filtered to be archived there. This is usually combined with some automatic delete of old messages. Large, frequently updated folders with automatic delete should be extremely common when subscribing high-traffic mailing lists.

Suggestions:

* Defer indexing of folders! A simple index maintenance policy could be enough:
** Keep a queue for index recreation jobs, and process this queue folder by folder when Kmail is inactive for some time.
** Lazily trigger index recreation whenever a folder with dirty index is opened. Most of the large folders I have are not even opened during a session, so there is no use updating their indices all the time.

* Avoid complete locks of the UI when recreating indexes. I can live with KMail's continuously high resource consumption, but I want to still be able to browse open email folders and to type the message I am editing.

* And, obviously: make (incremental) indexing more efficient -- if you can manage it (clearly this is not easy).

I have Fedora-rawhide (FC11) with KDE 4.2.0, QT 4.5.0:

fedora-release-10.91-1.noarch
kdepim-4.2.0-2.fc11.i386
kdebase-4.2.0-5.fc11.i386
kdelibs-4.2.0-13.fc11.i586
qt-4.5.0-0.3.rc1.fc11.i386

I experience the same problem here too. Especially when deleting or moving mail at the local maildir or imap server, kmail becomes unusable for several minutes with continuous hard disk activity. 
This bug should not be classified as NORMAL severity, but at least MAJOR. For me it should be classified as BLOCKER since I can't even use Kontact anymore as it stays indexing the files everytime.
Also, the priority should be classified as HIGH.

I put all my folders at the same level, so each of them have their own directory. But, deleting a message on one folder makes Kontact indexes all the folders (on each separated directory)! This I confirm when using "strace -p" on Kontact process after moving a message from "Trash" to another local folder(the output is huge, since it has 12 lines for each message on each folder - and I have thousands of messages on my local folders):
-----------------------------------------------------------------------------
munmap(0xb2f53000, 4096)                = 0                                     
time(NULL)                              = 1235136584                            
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.3DoTZ:2,RS", {st_mode=S_IFREG|0664, st_size=8127, ...}) = 0            
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.3DoTZ:2,RS", {st_mode=S_IFREG|0664, st_size=8127, ...}) = 0            
lstat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.3DoTZ:2,RS", {st_mode=S_IFREG|0664, st_size=8127, ...}) = 0           
access("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.3DoTZ:2,RS", W_OK) = 0                                                 
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.3DoTZ:2,RS", {st_mode=S_IFREG|0664, st_size=8127, ...}) = 0            
open("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.3DoTZ:2,RS", O_RDWR|O_LARGEFILE) = 44                                    
fstat64(44, {st_mode=S_IFREG|0664, st_size=8127, ...}) = 0                      
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb2f53000                                                                          
read(44, "Return-Path: <paulosoares@sigmaon"..., 4096) = 4096                   
read(44, "arset=3Diso-8859-1\">\n<meta name=3"..., 4096) = 4031                 
close(44)                               = 0                                     
munmap(0xb2f53000, 4096)                = 0                                     
time(NULL)                              = 1235136584                            
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.zHaZs:2,S", {st_mode=S_IFREG|0664, st_size=3915, ...}) = 0             
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.zHaZs:2,S", {st_mode=S_IFREG|0664, st_size=3915, ...}) = 0             
lstat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.zHaZs:2,S", {st_mode=S_IFREG|0664, st_size=3915, ...}) = 0            
access("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.zHaZs:2,S", W_OK) = 0                                                  
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.zHaZs:2,S", {st_mode=S_IFREG|0664, st_size=3915, ...}) = 0             
open("/home/cristiano.cunha/.kde/share/apps/kmail/mail/Pendencias/cur/1234891206.11152.zHaZs:2,S", O_RDWR|O_LARGEFILE) = 44                                     
fstat64(44, {st_mode=S_IFREG|0664, st_size=3915, ...}) = 0                      
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb2f53000                                                                          
read(44, "Return-Path: <contato@sipam.gov.b"..., 4096) = 3915                   
close(44)                               = 0                                     
munmap(0xb2f53000, 4096)                = 0                                     
time(NULL)                              = 1235136590                            
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.BuxUK:2,S", {st_mode=S_IFREG|0664, st_size=5249, ...}) = 0
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.BuxUK:2,S", {st_mode=S_IFREG|0664, st_size=5249, ...}) = 0
lstat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.BuxUK:2,S", {st_mode=S_IFREG|0664, st_size=5249, ...}) = 0                                                                               
access("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.BuxUK:2,S", W_OK) = 0                                     
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.BuxUK:2,S", {st_mode=S_IFREG|0664, st_size=5249, ...}) = 0
open("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.BuxUK:2,S", O_RDWR|O_LARGEFILE) = 44                        
fstat64(44, {st_mode=S_IFREG|0664, st_size=5249, ...}) = 0                      
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb2f53000                                                                          
read(44, "Return-Path: <pcf.inf-l-bounces@d"..., 4096) = 4096                   
read(44, "> SETEC/SR/DPF/AM\n> =\n\n> =\n\n> ---"..., 4096) = 1153              
close(44)                               = 0                                     
munmap(0xb2f53000, 4096)                = 0                                     
time(NULL)                              = 1235136590
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.A4d27:2,S", {st_mode=S_IFREG|0664, st_size=3556, ...}) = 0
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.A4d27:2,S", {st_mode=S_IFREG|0664, st_size=3556, ...}) = 0
lstat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.A4d27:2,S", {st_mode=S_IFREG|0664, st_size=3556, ...}) =0
access("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.A4d27:2,S", W_OK) = 0
stat64("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.A4d27:2,S", {st_mode=S_IFREG|0664, st_size=3556, ...}) = 0
open("/home/cristiano.cunha/.kde/share/apps/kmail/mail/.inbox.directory/Pcf.inf/cur/1170674169.5956.A4d27:2,S", O_RDWR|O_LARGEFILE) = 44
fstat64(44, {st_mode=S_IFREG|0664, st_size=3556, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb2f53000
read(44, "Return-Path: <pcf.inf-l-bounces@d"..., 4096) = 3556
close(44)                               = 0
-----------------------------------------------------------------------------
Please set this bug as confirmed, high priority, major severity and assign it to someone that can fix it as soon as possible. I really like kontact/kmail, but with this bug I can't work with it anymore.

[CONFIRMED][FIXED][WORKAROUND]
Since I only use KDE at my job, I really need Kontact to work. So I downloaded the source RPM and patched it so it stops the indexing process. 

Now I can work again! No more 15 minutes delays every 16 minutes!

I know the workaround is not a FIX (and it an ungly solution) but since I don't really need the full text search, I just disabled it and use a filesystem "flag" to turn it on whenever I need it.

--- BUILD/kdepim-4.2.0/kmail/kmfoldersearch.cpp 2008-11-19 08:19:06.000000000 -0200
+++ kmfoldersearch.cpp  2009-02-20 19:54:04.000000000 -0300
@@ -252,7 +252,11 @@

 void KMSearch::slotProcessNextBatch()
 {
-  if ( !running() ) {
+  char *flag = "/tmp/kontact.reindex";
+  struct stat buffer;
+  int reindex = (stat(flag, &buffer) == 0);
+
+  if ( !running() || !reindex ) {
     return;
   }

@@ -273,6 +277,9 @@
     } else {
       --mRemainingFolders;
     }
+    if (mRemainingFolders==0) {
+       unlink(flag);
+    }
     mProcessNextBatchTimer->start( 0 );
     return;
   }

Thanks, Cristiano, at least we know now where in the code this unnecessary resource intensive behaviour is happening. And that it can in principle be disabled.

This should make it easier for developers to fix this. For those of us who can't patch the source, it would be great to get a solution soon. Otherwise kmail is not usable for people with more than a few messages. You can't send a quick urgent email if you have to wait for 15 minutes before the programme responds, and often when it's done with its frantic disk and cpu activity it has drained the battery on the laptop ...

I agree that this bug should get higher priority and severity.

I have this problem as well with kde 4.2.2.
I am a very conservative email user, not using search folders or IMAP. Just
a number of POP servers, but many old emails.
I always loved kmail for being able to handel large amounts of emails without
performance degradation. This love is now somewhat diminished by having to
wait for 5-10 minutes after starting kmail for it to become usable.

This problem should be confirmed ASAP and solved.

Are all of you who see this behavior on Red Hat or Fedora?

It doesn't happen with here, KDE 4.2.4 Debian Unstable, with several folders, some of them with tens of thousands of messages.
Mostly maildir, some smaller folders (<1000 messages) mbox

I am on kubuntu 9.04, which is derived from Debian, but generally not as stable :-)

Using KDE 4.2.96 on Fedora Rawhide: kdelibs-4.2.96-2.fc12.i586

I'm experiencing the same problem reported by John on kontact/kmail start up. I don't know if this has something to do with multi-core processors, but kontact don't hang completely, instead it gets really really slow with the hard drive and CPU working for 5 minutes!

This problem was worse before, it would hang kontact/kmail completely on evey mail deletion or when moving an e-mail from inbox to the local folders, which makes kmail useless for 15 minutes everytime!

Maybe we need some flag to tell us "hey, there were no changes since last kmail session: so we don't need to rebuild the index!"

Regards,

Cristiano

*** This bug has been confirmed by popular vote. ***

Yes, I have that as well.
After startup, kmail is not completely unresponsive but might take 30 seconds
or so to react to a keypress or a click in its interface.
This lasts for about 5 minutes, in the meantime it is giving my ageing
hard disk hell :-(
I sometimes do not enter the kwallet password, but even so the disk activity
goes on. So it is not related to getting emails from my pop servers.

The behaviour is definitely different from kmail kde 3.5.x
For me and my wife, this bug is the worst one for kde 4.x, we are
bitten by it every day...
Must be the same for many other users.
Is there a change we might fix this for 4.3 (or 4.3.1?)
Would be great!

I had this problem as well, but now with KDE 4.2.4 and Qt 4.5 on Gentoo, I didn't have it for weeks. Seems like it's gone.

Problem still exists for me in kde 4.3
Packages from kubuntu.
But don't let that fact stop you from analyzing this report :-)

I deleted the trash folder (hadn't been cleaned up fro 2001) and manually recreated the index file for a few large folders. This seemed to solve the problem. Not entirely clear how this explains no automatic indexing anymore. My best guess is that I made an error somewhere about a year ago when I manually migrated all emails from 3.5 to 4.2...

I've got this same problem for years. I'm now using Kmail 1.12.1 with Kubuntu 9.04 and still the same. I can't have any search folder or kmail takes minutes after startup until it's usable. For me, virtual folders are useless although it should be a great tool to increase productivity.
Kmail rocks but this behavior is awful. It were great if it was solved.
I've voted for this bug and for two other related:
https://bugs.kde.org/show_bug.cgi?id=93158
https://bugs.kde.org/show_bug.cgi?id=84162

After not having this for a long time (see comment #15), I'm now back again with this bug. I'm still on KDE 4.2.4 and Qt 4.5 on Gentoo and the bug somehow reappeared. I don't have a search folder nor do I do things different than before. One can see Kontact is actually reindexing by executing "lsof -p <kontact's process id>|grep index". This displays the index files currently opened by Kontact.

kontact: 4.3.2
kmail: 1.12.2
KDE Version: 4.3.2 (KDE 4.3.2)
Qt Version: 4.5.2
Operating System: Linux 2.6.31-14-generic x86_64
Distribution: Ubuntu 9.10

I had the same problem and hoped with changing to KMail 1.12.2 the problem was
fixed, but only for one week. :( Now i have the same problems since bevor.

I have over 160.000 lokal emails in maildir with over 16GB disk space.
So kmail needs over 30(!) minutes bevor i can use it!!!

*** Bug 172519 has been marked as a duplicate of this bug. ***

*** Bug 192216 has been marked as a duplicate of this bug. ***

*** Bug 194951 has been marked as a duplicate of this bug. ***

*** Bug 231019 has been marked as a duplicate of this bug. ***

*** Bug 233782 has been marked as a duplicate of this bug. ***

This bug is open for months, now, there are dozens of duplicates reported on current versions of kmail. Are there any official patches/fixes? 

Is there a "button" to rebuild indices globally (for all folders - I have many) - *WITHOUT* losing tags/markers? 

It is quite annoying that rebuilding the index will destroy all tagging and possibly read messages that are to be kept (by policy) might get lost because unread messages > 1 year old are deleted by local policy...

I also noticed, that while there is background activity, it is hard to find out, what kmail is actually doing. And some actions (like highligthing message headers while hoovering over them) are more responsive than others (like displaying message bodies which can take several minutes).

BTW: version 4:4.3.4-2 http://ftp.us.debian.org sid/main Packages

Now i think it not really a bug of kmail because, since my last post in november, i updating the linux kernel from 
Linux 2.6.31-14-generic x86_64 to 
Linux 2.6.31-20-generic #58-Ubuntu SMP x86_64 GNU/Linux 
and now i have no problems with kmail anymore! :)
Everthing works fine and fast(!) and startup of kmail needs only seconds without creating any index!
So it is perhaps a bug of the kernel or combination of kernel and kmail...

Best regards,
Axel Schmidt

Since the last updates last week, the bug is back! :(
I have now kernel
Linux axel-laptop 2.6.31-21-generic #59-Ubuntu SMP x86_64 GNU/Linux
but don't think anymore it's a kernel problem, because booting the old kernel i have the same bug. So anything else must be the problem. On last friday 2010-05-14 i updated some kde packages:
libkopete4, kopete, kppp, krdc, krfb, kde-zeroconf
So perhaps one of these packages has the bug in combination with kmail.

Best regards,
Axel Schmidt

Thank you for taking the time to file a bug report.

KMail2 was released in 2011, and the entire code base went through significant changes. We are currently in the process of porting to Qt5 and KF5. It is unlikely that these bugs are still valid in KMail2.

We welcome you to try out KMail 2 with the KDE 4.14 release and give your feedback.