Bug 71549

I experimented a bit and found out:

- There is no output before, kmail loads for about 10 seconds, then immediately starts to dump the  "QGArray::at:" lines with no preceeding errors or warnings
- Removing the ~/Mail folder does not help, I still get the error.
- Removing the ~/.kde/share/config/kmailrc file helps, so the error seems to be caused by something in there. Tell me which part of kmailrc could be relevant and I'll post it here.

I found it the entry causing it!

This is it:

[Account 3]
Folder=
Name=Pilz-Imap
Type=imap
auth=*
auto-expunge=true
check-exclude=false
check-interval=0
hidden-folders=false
host=pilz
login=roli
pass=[password deleted]
port=143
precommand=
prefix=/
store-passwd=true
subscribed-folders=false
trash=trash
use-ssl=false
use-tls=false

--- End ---

I have to add that there is no correctly working imap-server on that host (yet), so maybe kmail is a little bit confused.

I've tried to add "folder=inbox", but that didn't change it, if I should try anything else, please advise. (Hopefully I'll get the email ;-)

I can't reproduce this problem with the current development version and a non-working IMAP server. Did this server work in the past? If yes, try if deleting ~/.kde/share/apps/kmail/imap/.Pilz-Imap.directory helps.

You can also try if closing the folder tree for this IMAP account helps. Also please try if enabling 'Exclude from "Check Mail"' for this IMAP account helps.

It's really strange, but now I can't reproduce the bug myself anymore :-/
I could reliably reproduce it 2 days ago (I tried several times with and without "Account 3") but now it works fine with and without that Account. I must add that I deleted a couple of thousand emails in the meantime, but didn't change the configuration except for adding/removing Account 3.

The IMAP-Server never worked, but there was a POP3-server on that machine a couple of months ago.

It is really strange because I also tried to move the Mail-folder 2 days ago and the bug was still triggered with no messages at all, so me deleting mail shouldn't have any effect on this bug.
Also, it's impossible that kmail "healed" itself because I tried it several times and the bug came back even after kmail was started successfully in the meantime.

Not really reproducable in this situation...

This bug is NOT fixed as of kdepim 3.2.2 - I was just bitten for the second time - (the first time was months ago).  I figured that this is a pretty obscure bug, so I'd better track it down.  I think I've found the cause.  In KMMsgDict::readFolderIds(),  I found that one of my folders had count=1296769059 (after the byte swapping).  This folder had only one message in it, so there's obviously a  problem.  Looking at the .ids file, I found that it had the header written to it twice:
    >cat /home/dan/Mail/.OONL.directory/.request.index.ids
    # KMail-Index-IDs V1002
    *xV4# KMail-Index-IDs V1002
    *xV4

So the problem is NOT at startup - it's when that file was last written to.  This file was messed up somehow.  I don't remember what happened last time I closed kmail - it may or may not have been a clean exit.  I was working on a control center module for kicker at the time, and it was buggy, so anything could have happened.  I wasn't doing anything unusual in KMail though.

I do remember that the last time this happened was when I was running a beta KDE and it definitely did NOT close kmail nicely.  

So, I found only two places where this header is written, in the files:
    kmfoldersearch.cpp:789:    
        fprintf(tmpIndexStream, IDS_HEADER, IDS_VERSION);
        in function: KMFolderSearch::writeIndex( bool )
    kmmsgdict.cpp:451:  (line # is from my modified version of the file)
        fprintf(fp, IDS_HEADER, IDS_VERSION);
        in function: KMMsgDict::openFolderIds(KMFolder *folder, bool truncate

I'll leave the actual debugging to someone who knows what the heck this code does (I'm just a kmail user, not a kmail developer :).  My tired brain can't puzzle it out right now.

I'm attaching a modified kmmsgdict.cpp that throws some debugging info up (if you have debugging enabled), a copy of said debugging, and a stack trace.

Created attachment 5988 [details]
Modified kdepim-3.2.2/kmail/kmmsgdict.cpp - added some debugging statements that help track the problem down

Created attachment 5989 [details]
debugging output that shows what's happening when count is misread

Created attachment 5990 [details]
stack trace

One thing I forgot.  If you let kmail churn through all 1296769059 of the  messages, most of which don't exist, everything will be fine, which is probably why the original reporter couldn't reproduce it.  I doubt that it has anything to do with IMAP at all, the machine had just finished running through the messages when the reporter changed the config.  
If you kill it after it starts spewing the "QGArray::at: Absolute index XXXXXXX out of range" stuff, it will just pick up where it left off (not sure why).  I saved a copy of my Mail dir and the kmail files in ~/.kde, so I can reproduce it, if needed.

Actually, I (the reporter) was also hit by this bug again a     couple of weeks ago, so I reopen it.

"So the problem is NOT at startup - it's when that file was last written to."

In my opinion, a   program should never crash, no matter how messed up the input files are, this could even be a denial-of-service vulnerability, possibly triggered by something innocent as an email.

P.S.: I don't know wether this helps, but I get the feeling that most strange behaviour is caused on shutting down the computer: KMail tries to do it's shutdown procedure but is interrupted by the shutdown sequence which could be the cause of such "messed up" files.

I have created a smaller Mail directroy and pasted in
some of my mail. The startup is then ok, but if I
use the large mail directroy, kmail crashes at the startup.
Large = 80 MB Small = 46

I had no problems using the older version of kmail
This problem started 4 days after upgrading from
SuSE 8.2 to SuSE 9.1

I sure would like to be able to see all my past
mail. 
Any suggestions ?
Thanks

I had the same problem without any IMAP account. My mailbox was quite large (~ 1 Go)... 
After cleaning all index file i could restart kmail but lost a lot of parameters (folder preferences like icons, default identity etc.).

Was with KDE 3.2.2.

Hope that help.
-- 
Sebastien

Not imap specific.

This has been resolved a few days ago in the development version of KMail at last.

I am getting this error too on running "Apply Filters".

I have about 200 messages in my inbox. I have a number of filters configured. If I select all messages in Inbox and click on "Apply Filters", Kmail happily eats up all CPU. I ran lsof on the kmail process. I found out that kmail has all the .index files of all the maildirs that I have in kmail (about 50+). It seems as if Kmail is searching in all the folders even if I have selected specific messages in my inbox to run filters on. I am running kmail as packaged on FC2.

On Thursday 09 September 2004 04:20, Sandip Bhattacharya wrote:
...
> I am running kmail as
> packaged on FC2.

I missing version info. You may be running a pre KDE 3.3 version that 
may not include the fix.

I did mention 1.6.2 while filing the bug(in the meta dat, not the comment though). FC2 has kmail-1.6.2 with KDE-3.2.2. Isn't there a backport fix for this bug in these older versions? Or one that can be submitted to, say Fedora guys, so that they can fix kmail on it?

Right now, this bug makes kmail pretty unusable for a few minutes if I press the apply filter button by mistake.
 

First I thought it's a Year 2005 Mistake ;-)
31.Dec.2004 i could start kontact/kmail, and today 1.Jan.2005 I get this error.
  * SuSE 9.2
  * Qt: 3.3.3
  * KDE: 3.3.0
  * KMail: 1.7.1
1,1GB in Mail-Folders, some pop3 and some imap Folders.
With SuSE 9.1 I had the same config and did never run in troubles.

I have to access my mails in the Mail-Folder as soon as possible. 

Now I found the corrupt ids file:

~/.kde/share/apps/kmail/imap/.1935081305.index.ids:
# KMail-Index-IDs V1002
*xV4^R# KMail-Index-IDs V1002
*xV4^R

kmail is working again, but I think, after a year, kmail should be smart enough to solve such a problem by itself.

Created attachment 9335 [details]
backtrace when kmail processing bad index.ids file

I'd like to add that I see this same bug in FC2, latest FC2 kmail (1.6.2).  Same exact index.ids caused the problem.  Backtrace is a bit different that the one previously attached; I've added mine here.

For those who may follow, I fixed the problem by running lsof on kmail (actually kontact in my case) and grepping for files in my ~/Mail/.  It was obvious that one particular index.ids file was the problem.  I exited kmail, deleted that specific index.ids.  When I restarted kmail, it came back up fine.

Looks like this bug still exists. I can reproduce it on KMail: 1.9.1 from KDE 3.5 (Debian SID packages). It causes KMail to eat all CPU and memory (1 GB in 2 seconds ;). 

Everything works as usual when I remove this file:
~/.kde/share/apps/imap/.1074588759.directory/Trash.index.ids 

File attached.

Created attachment 14321 [details]
Trash index file which crashes KMail.

File from ~/.kde/share/apps/imap/ folder.

This bug is not resolved.  I just had the exact same issue with kmail 1.9.1 running on FC3 with KDE 3.5.1.  I performed the lsof|grep described above in comment #22 and deleted the "corrupt" .ids file. I'm back to running, but this must be fixed.

Please reopen. I've encountered this bug twice now. The first time as I added some filter rules for the first time (the bug appeared when closing and restarting KMail afterwards). I removed my whole .kde/share/apps/kmail folder to fix it because I didn't know what to do.

Today it hit me a second time, but this time I hadn't done anything unusual. In my opinion (as mentioned earlier), this bug is most likely to happen when KMail gets closed by a machine shutdown.

You can find broken .ids files with the following Bash line:
find ~/.kde/share/apps/kmail -name '*.ids' -exec grep -Hc '^\*xV4' '{}' \; | grep -v ':1$'

To remove them automatically (at your own risk), change that to:
rm $(find ~/.kde/share/apps/kmail -name '*.ids' -exec grep -Hc '^\*xV4' '{}' \; | grep -v ':1$' | cut -d : -f 1)

Imho a really simple workaround could be written when reading the .ids files to prevent this bug from happening.

It's been three months since the last whining, time for another one.

I've encountered this bug again today, so *please* reopen.

One more whiner :-) It just happened to me too today (Kmail 1.9.1, Ubuntu Breezy), after shutting down the laptop with an open Kmail instance. After starting again the computer, it was completely unresponsive, busily swapping while Kmail was eating up all RAM. Also killing Kmail and restarting it did not solve the problem. Only deleting the hidden index files in the Mail folder helped. 

+1

I've filed bug 134134 requesting that this bug be reopened.

Using Tim Weber's notes, I got my kmail/kontact back up and running.  With 1.1GB of email, man am I glad it's running again :) .  That's not a fix for the bug, but at least we have a workaround to recover from this bug.

PLEASE, don't give my co-workers who rail against KDE, Kontact & Kmail any more ammunition :) .

Reopening as requested.

Just had this same problem in Fedora Core 5, with kdepim-3.5.4-0.1.fc5. Method suggested in comment #30 worked around the issue. Konact shut down via KDE shutdown on previous successful use. When Kontact came up on problamatic trial, it complained that it may already be opened, from then on I had the problem where kontact did not want to open. Ran it from terminal, noticed the error messages, and gooled my way here.

Lots of confirmations, lets correct status of this issue.

This bug happened to me two weeks before. I could not start up kmail. The problem was solved when I removed the index files. Few days latter it showed up that I had a hard drive problem, which caused the corruption of the index (and not only the index) files.
   In my opinion KMail should be able to start up even if the index files are corrupt, it should report the problem and ignore the corrupt data, when it is possible.

SVN commit 633012 by mkoller:

BUG: 71549

Make sure that a corrupted .ids file does not allocate huge amount
of memory by checking consistency of file length vs. number of
items expected.

The patch does not cure the original problem why the ids
file has corrupt content. I could not find a way how this can happen
up till now


 M  +15 -2     kmmsgdict.cpp  


--- branches/KDE/3.5/kdepim/kmail/kmmsgdict.cpp #633011:633012
@@ -16,6 +16,7 @@
 #include <stdio.h>
 #include <unistd.h>
 
+#include <string.h>
 #include <errno.h>
 
 #include <config.h>
@@ -73,8 +74,7 @@
   KMMsgDictREntry(int size = 0)
   {
     array.resize(size);
-    for (int i = 0; i < size; i++)
-      array.at(i) = 0;
+    memset(array.data(), 0, array.size() * sizeof(KMMsgDictEntry *));  // faster than a loop
     fp = 0;
     swapByteOrder = false;
     baseOffset = 0;
@@ -412,6 +412,19 @@
   if (swapByteOrder)
      count = kmail_swap_32(count);
 
+  // quick consistency check to avoid allocating huge amount of memory
+  // due to reading corrupt file (#71549)
+  long pos = ftell(fp);       // store current position
+  fseek(fp, 0, SEEK_END);
+  long fileSize = ftell(fp);  // how large is the file ?
+  fseek(fp, pos, SEEK_SET);   // back to previous position
+
+  // the file must at least contain what we try to read below
+  if ( (fileSize - pos) < (count * sizeof(Q_UINT32)) ) {
+    fclose(fp);
+    return -1;
+  }
+
   KMMsgDictREntry *rentry = new KMMsgDictREntry(count);
 
   for (unsigned int index = 0; index < count; index++) {

I had this problem happen about a week ago.  I thought it was a different bug. But, it's exactly as described here. KMail 1.9.4. The index file(s) may get corrupted because when shutting down computer from KDE it does not let some of the processes finish normally (they may be just "killed"). I notice this with KNode: when shutting down I can see a window with flashing "Deleting expired articles from <newsgroup>" (or something like it), but not all newsgroups are reported. Seems like the processes just gets interrupted. Same thing may happen with KMail.

KMail 1.9.6 / KDE 3.5.6, Kubuntu 7.04 exhibits the same behaviour

*** Bug 63218 has been marked as a duplicate of this bug. ***

removing all *.ids files in the mails folder (recursively) helps

use following script:

find mail -name "*.ids" > ids.list
echo "END" >> ids.list
cat ids.list | while [ "$line" != "END" ]; do read line; rm "mail/$line"; done
rm ids.list

mail is the folder containing the mails from kmail

hth et

The problem occured on the actual Gentoo KDE 3.5.5 again. The script provided by Thomas did not help. Please reopen!!! This bug is not fixed.

>The problem occured on the actual Gentoo KDE 3.5.5 again. The script provided by Thomas did not help. Please reopen!!! This bug is not fixed. 
Not in the 3.5.5 version of KMail. Try KDE 3.5.7, it should be fixed there.