Bug 50462

Summary: mail is permanently lost if disk is full
Product: [Applications] kmail Reporter: Alastair Scott <alastair.scott>
Component: pop3Assignee: kdepim bugs <kdepim-bugs>
Status: RESOLVED UNMAINTAINED    
Severity: grave CC: amitshah, ana, arne.schmitz, bero, bluedzins, bss, cshobe, dev, diaxen, etkopa, jakub.januszkiewicz, kamaraju, kde, kebekus, krzysiek, madman, marc.hansen, Marcin.Kasperski, Marko.vanDooren, orzechowskip, panagiotis.issaris, rasasi78, richard, winter, xbarr
Priority: NOR    
Version: unspecified   
Target Milestone: ---   
Platform: unspecified   
OS: Linux   
Latest Commit: Version Fixed In:

Description Alastair Scott 2002-11-09 13:33:47 UTC
Version:           1.5 (using KDE 3.0.9)
Compiler:          gcc version 3.2 (Mandrake Linux 9.1 3.2-3mdk)
OS:          Linux (i686) release 2.4.19-16mdk

The scenario is:

i. I have about 30 emails on a POP3 server;

ii. I download them and delete them from the server while doing so;

iii. Unknown to me / (including /tmp) has very recently become full - immediately after step iv df returns 0% free;

iv. The emails appear in Kmail with 'no subject' and 'no author' shown and no contents.

The result is that the email contents appear to be permanently lost; closing kmail, freeing some space on / and reopening it still gives 'no subject', 'no author' and no contents for each email.

Granted it is an unlikely situation, but kmail should fail safe if it arises!
Comment 1 Ingo Klöcker 2002-11-09 15:07:54 UTC
Subject: Re:  New: emails null on download if / (including /tmp) is full

On Saturday 09 November 2002 13:33, Alastair Scott wrote:
> The scenario is:
>
> i. I have about 30 emails on a POP3 server;
>
> ii. I download them and delete them from the server while doing so;
>
> iii. Unknown to me / (including /tmp) has very recently become full -
> immediately after step iv df returns 0% free;

Are you using maildir or mbox folders? Did KMail crash resp. exit 
abruptly when you ran out of disk space?

> iv. The emails appear in Kmail with 'no subject' and 'no author'
> shown and no contents.
>
> The result is that the email contents appear to be permanently lost;
> closing kmail, freeing some space on / and reopening it still gives
> 'no subject', 'no author' and no contents for each email.

The index of this folder is corrupted. Exit KMail, delete the index of 
the affected folder and then restart KMail.
For example the index files of the inbox are called 
~/Mail/.inbox.index*. You should delete all index files of the affected 
folder.


Comment 2 Alastair Scott 2003-01-03 11:34:48 UTC
Apologies for the delay in responding (illness). Questions answered: 
 
1. mbox folders: kmail neither crashed nor exited abruptly, and seemed content to 
continue with no disk space; 
 
2. deleting indices fixed that problem! 
Comment 3 Till Adam 2004-05-16 20:23:37 UTC
*** Bug 63166 has been marked as a duplicate of this bug. ***
Comment 4 Casey Allen Shobe 2004-05-17 07:34:22 UTC
"Can you confirm that indeed no mail was lost? If so, we should probably change the description of this bug to something like: "Downloading pop mail to a full disk silently fails without showing an appropriate error message". Otherwise it should be raised in severity."

No I most certainly cannot.  The mails are gone forever.
Comment 5 Fred Emmott 2004-05-28 13:04:23 UTC
There should be a 'disk full' warning or similar.
Comment 6 Robert Klein 2004-06-12 09:20:04 UTC
Today kmail 1.6.1 (on FreeBSD 5.2.1-RC2) crashed during downloading mails with a message like "kmail crashed because of disk full".  It lost all mails already downloaded :(

Using mbox folders.
Comment 7 Martin Spasov 2004-07-20 00:14:01 UTC
imho Kmail should not download mail, if there is not enough disk free space. A simple "<checkbox> Do not download mail if free space is below <textbox>XXX</textbox>" should be sufficient.
Comment 8 Don Sanders 2004-07-29 09:57:52 UTC
For the record KMail already has code to detect if disk space is 
exhausted and then cleanly exit without losing mail.

Robert Kleins report is especially perplexing as either he's using 
IMAP in which case mail shouldn't be deleted from the server when 
downloaded, or he's using POP in which case if KMail exits when disk 
space is exhausted while still downloading mail then the QUIT command 
can't have been issued to the POP server so the POP server shouldn't 
have deleted any mail (unless it's extremely buggy).

Don.

Comment 9 Michael Jahn 2004-08-31 11:15:45 UTC
*** Bug 57342 has been marked as a duplicate of this bug. ***
Comment 10 Jamie Matthews 2004-09-26 04:29:15 UTC
I don't know about how full my disk is, but I can verify that this is no fluke.  I have clicked on emails in various folders only to see them turn into No Author, No Subject, no content or the following content: "( .  )"  I also have deleted the indices, and found that the emails are still gone.  The emails in question are not fresh off the server, but do tend to be some of the most recently downloaded.  Very disturbing to see emails disappear right before your eyes.
Comment 11 S. Burmeister 2004-09-27 17:53:33 UTC
I can confim this issue too. People should vote to get this marked as new. I think that it would be reasonable to expect kmail to complain about not having enough space on the disc and not doing anything until this is resolved in order to avoid losing mail.
Comment 12 Amit Shah 2004-11-06 16:33:31 UTC
I'm using maildir with pop3; my / was full and kmail retrieved some messages; but they were truncated to 170bytes. My mail was lost. I thought it was some random spam, but upon receiving several such messages, I thought something was wrong. I tried sending a message to myself; that reported disk full.

This is a severe bug.
Comment 13 Amit Shah 2004-11-06 16:34:43 UTC
*** This bug has been confirmed by popular vote. ***
Comment 14 Don Sanders 2004-12-21 09:04:12 UTC
> ------- Additional Comments From amitshah gmx net  2004-11-06 16:33 -------
> I'm using maildir with pop3; my / was full and kmail retrieved some
> messages; but they were truncated to 170bytes. My mail was lost.

Can you reproduce the problem? We are unable to.

Don.

Comment 15 Amit Shah 2004-12-21 09:10:36 UTC
On Tuesday 21 Dec 2004 13:34, Don Sanders wrote:
> > ------- Additional Comments From amitshah gmx net  2004-11-06 16:33
> > ------- I'm using maildir with pop3; my / was full and kmail retrieved
> > some messages; but they were truncated to 170bytes. My mail was lost.
>
> Can you reproduce the problem? We are unable to.

Yes.

>
> Don.

Comment 16 Don Sanders 2004-12-22 08:19:51 UTC
On Tuesday 21 December 2004 18:10, Amit Shah wrote:
..
> Yes.

How?

Don.

Comment 17 Amit Shah 2004-12-22 08:38:05 UTC
On Wednesday 22 Dec 2004 12:49, Don Sanders wrote:
> > Yes.
>
> How?

As reported in comment #12 by me and the exact steps to reproduce it are in 
the original reporter's bug report.

Comment 18 Don Sanders 2005-01-06 00:56:50 UTC
> As reported in comment #12 by me and the exact steps to reproduce it are in
> the original reporter's bug report.

No. How can someone else reproduce the bug?

Don.

Comment 19 S. Burmeister 2005-01-06 10:38:51 UTC
I could reproduce it with dimap when my home-directory was full. All I needed to do was d/l and see the author and subject be converted to no author and no subject when clicking on the message in the header-list.
Comment 20 S. Burmeister 2005-01-06 10:39:50 UTC
Is this maybe related to people having their home on a seperate partition then their /tmp?
Comment 21 Leo Spalteholz 2005-02-11 22:04:41 UTC
I can confirm this bug.  KMail will silently crash when I run out of diskspace.  I have my mail stored in my home folder, which is on a seperate partition as /tmp.
I can't recall which one ran out of room, but I will test this when I get home.

I'm not sure about lost mail, but the crash is certainly unnerving.
Comment 22 Carsten Burghardt 2005-04-19 17:23:57 UTC
*** Bug 89143 has been marked as a duplicate of this bug. ***
Comment 23 Carsten Burghardt 2005-04-19 17:24:59 UTC
*** Bug 104116 has been marked as a duplicate of this bug. ***
Comment 24 Daniel Rendall 2005-12-10 09:37:10 UTC
This just happened to me - kmail version 1.8.1 in KDE 3.4.1, when checking POP accounts. The / partition was full but /home wasn't; 'ghost' mails then showed up in my inbox with no real content. I was able to send myself a test mail and this came back similarly blank. Freeing up space on / cured the problem, but I've still lost quite a few mails since they've been deleted from the POP server.
Comment 25 S. Burmeister 2005-12-10 09:48:56 UTC
A simple call of df and parsing the output would solve the problem. If the free space is below 10 MB, the user should get a notification to free space in order for the d/l of emails to work again.
Comment 26 Casey Allen Shobe 2005-12-10 12:04:07 UTC
That's an inefficient and nonportable solution.  df can take significant time to return results in the case of GFS, NFS, and probably many other filesystems, plus you have to work out the details of which partition the kmail store resides on instead of just guessing /home.  Symlinks are possible too so simply reading the config file is no good.

Better is to just write the file out, but catch any errors instead of blindly continuing.  If there is an error, do not issue the delete command to the POP3 server.  Present warning dialog and do not check any more POP3 accounts while the error window is still open.
Comment 27 S. Burmeister 2005-12-10 12:33:38 UTC
This bug is from 2002 and hence any solution is better than data loss. Unexperienced users will not even know why they lose mail, or what to do.

The problem is not only the kmail store, but the / or I guess tepm-dir, as comment #24 states.
Comment 28 Casey Allen Shobe 2005-12-10 12:57:46 UTC
So?  If file write errors are not ignored, then the problem doesn't happen, regardless of which file can't be written when.  This should be checked upon ANY file write.  Randomly guessing a partition that might be at fault and issuing df's which can be prohibitively expensive is not a good solution.

If the message is not successfully transferred into the store for ANY reason, then it should not be deleted from the POP3 server, simple as that.  If you were too lazy to fix the error catching, you could do a size comparison instead (IIRC POP3 supports this).  The errors really ought to be caught at the source of the problem though, rather than bandaided around.
Comment 29 S. Burmeister 2005-12-10 13:05:30 UTC
:) What are you trying to prove to me? All I say is: 3 years to fix a data loss bug is too long, so there should be a solution quickly! If there is a dirty but quick solution around, pick that one and use it until a technically satisfying solution can be implemented.

Delivering data loss since 2002 is not a solution but apparently the only possible way for now.
Comment 30 Casey Allen Shobe 2005-12-10 14:09:56 UTC
Well, I agree that it should be fixed, but things that are fixed wrongly have a way of being forgotten or left as-is forever and relied upon by other dirty fixes, and then you end up with Microsoft Outlook! :P  Cheers.
Comment 31 Thiago Macieira 2005-12-11 15:59:24 UTC
The statfs(2)/statvfs(2) system calls return the free space on a given filesystem.
Comment 32 Ismail Donmez 2005-12-13 21:45:32 UTC
Lost lots of mails today due to this. Online imap was downloading fine but filters use /tmp and / was out of space. No warning nothing. Just lots of empty mails which are just downloaded & filtered & results in empty files.
Comment 33 Thiago Macieira 2005-12-20 11:47:53 UTC
*** Bug 118690 has been marked as a duplicate of this bug. ***
Comment 34 Thiago Macieira 2006-02-02 19:28:18 UTC
*** Bug 120965 has been marked as a duplicate of this bug. ***
Comment 35 Thiago Macieira 2006-02-02 19:28:25 UTC
*** Bug 104705 has been marked as a duplicate of this bug. ***
Comment 36 Thiago Macieira 2006-02-02 19:28:59 UTC
*** Bug 93473 has been marked as a duplicate of this bug. ***
Comment 37 Tom Christensen 2006-03-13 01:07:40 UTC
After a preliminary readthrough of the code, I believe this bug will require a complete rewrite of the popaccount class.  Currently, popaccount offloads the actual communication with the POP server to a KIO slave object, which spawns a separate process and downloads the actual messages and handles all of the POP3 protocol stuff.  Once this process downloads a chunk of messages, it returns the raw messages to kmail to "handle".  Kmail then takes the messages and stores them, however since this process appears to be async, once kmail finds out that there is no hard drive space, it is too late to tell the pop server not to delete the messages, because the KIO process already disconnected the pop connection, and deleted the messages.

Someone mentioned that kmail has checks to throw an error if the hard drive is full, and I see those in the code to store the data in the mailbox, however running an strace the filter manager saves the document to a tmp folder first, and then the storage system (in kmfoldermbox.cpp) just does a move.  I don't see anything in the filter code, or the tmp storage code that does checking for errors, therefore, what happens is as follows:

1) Person downloads POP messages, the KIO process drops those messages into memory for kmail to handle and disconnects the pop connection (deleting the messages on the server).

2) Kmail runs the messages through the filter code, which "stores" the messages on the temporary location, because this temporary file system is full, the store fails, but there is no checking at this point in the code.

3) Kmail tries to "store" the message in the mailbox, which simply performs a rename on the file which doesn't contain anything because the temp disk was full.

4) Kmail's mailbox has 0 length files that were "moved" from the full tmp storage area.


I may be looking at the code all wrong, but from what I can see, fixing this bug requires 2 things.

1) rewrite popaccount to use a syncronous methodology to retrieve and store messages.  Or attempt to write a file to the mailbox/temp storage area before spawning the async process, and assure that there is space before going into async mode to download the messages.

2) put more checks in the filter code so that temporary storage failures cause the message download to abort, and disconnect cleanly from the pop server without deleting messages.
Comment 38 Leo Spalteholz 2006-03-13 07:53:23 UTC
I think the best preliminary solution would be just to check free space before starting message download.  You don't even have to be that exact about it.  If there is less than 50MB available, refuse to download anything and inform the user of that fact.  It's definitely not the most elegant way, but it beats a rewrite of popaccount and fixes this bug, which is incredibly serious.  Nothing is worse than bugs which lose users' data.
Comment 39 Marcin Kasperski 2006-03-13 10:00:23 UTC
'50MB' solution is plainly wrong. For two reasons:

a) 50MB is strictly arbitrary, it can happen that someone gets larger email (say, video or bunch of photos...), at the same time it places unnecessary limits on the people using shared machines with low disk quotas

b) lack of disk space is not the only possible problem, one can imagine that - for instance - the managing process is killed while saving email (for instance because of lack of RAM or just because of shutdown, or...)

The only correct order in case of POP email download is:
a) download messages
b) save them
c) only after they are correctly saved, delete them from the server
Comment 40 Tom Christensen 2006-03-13 14:44:19 UTC
After some more analysis this morning, I am about 100% sure that this bug has to do with the asyncronous method of download for pop accounts.

When/If an error in writing a file is detected (as I said, the filter code doesn't appear to have any error checking WRT the temporary storage of a message as it is being filtered) the code returns an error, however this error is returned to the PopAccount::slotProcessPendingMsgs() function which has no ability whatsoever to pass the error up the line to the KIO process that is actually communicating with the server.  Thus, when an error is detected in writing to the mailbox, the messages that are already downloaded are lost because the KIO process is out of the return path of the error message.

I'm not the greatest c++ coder, or I would be working on a fix... I can understand basic execution paths though, and this one seems pretty clear to me.  I hope this points someone in the right direction and they can write a fix pretty quickly... The problem is the async message retrieval process used by kmail.  As Marcin said, the ONLY appropriate methodology for POP email is:
a) download messages
b) save messages
c) IIF save successful delete messages on server

Kmail violates this process by backgrounding the message download process and leaving it out of the return path of errors.

Comment 41 Ismail Onur Filiz 2006-03-13 18:32:43 UTC
Thanks for all your help Tom. By the way, you don't need to be the greatest C++ coder. Just try experimenting, download the code, play around with it. Fixing many bugs actually turn out to be quite simple. And in this case -I didn't look at the code but- it sounds like you are almost there. Your best friends are the KDE API documentation ( http://developer.kde.org/documentation/library/3.5-api.php ) and QT documentation ( http://doc.trolltech.com/3.3/index.html ) I don't think anybody would reject if you asked for help when you got stuck -it might just take some time. And you don't need to work on it until you have "the perfect patch", somebody who knows the code better could come up with suggestions later on. And good luck if you decide to code a fix:)
Comment 42 Adrien Cordonnier 2006-03-27 01:11:25 UTC
I got this bug with a disk not full but which is going to crash soon.

My drive started to make strange noise, I discovered SMART tools which told me the disk is at the end of its life. It is still possible to read and write, yet it sometimes require more than one try.

I have no idea of what is possible or not but I think when writing fails, Kmail should keep the messages in RAM and propose to the user to:
1) display them
2) save them on another disk
3) forward them to another mailbox
4) forward them to the mailbox they come from and desactivate deletion and automatic retrieval from this mailbox
Comment 43 Marc Schütz 2006-04-10 14:34:49 UTC
I can only agree that checking for sufficient disk space is a bad idea. We will start using NFS4/Kerberos here at work soon, and a scenario that comes to my mind is that someone forgets to log out in the evening, his/her kerberos ticket expires during the night and KMail will loose all mail of that night because it is unable to store them in the user's home directory.
Comment 44 Ismail Onur Filiz 2006-05-05 09:06:18 UTC
*** Bug 126737 has been marked as a duplicate of this bug. ***
Comment 45 Arne Schmitz 2006-08-02 10:24:38 UTC
Is anything being done to avoid this bug? Comment #40 seems to hit the nail on the head. Wouldn't it suffice to give the disk writing object a signal which is connected to the server-communicating object, if something fails? Or a signal that means that everything is ok. One could even pass a message-ID or something which identifies the mail that has correctly been written to disk. Then the mailserver-object could delete it safely from the server.
Comment 46 Stefan Borggraefe 2006-09-11 20:42:20 UTC
*** Bug 133928 has been marked as a duplicate of this bug. ***
Comment 47 Tommi Tervo 2006-09-15 15:58:23 UTC
*** Bug 134122 has been marked as a duplicate of this bug. ***
Comment 48 Martin Koller 2006-11-01 15:48:36 UTC
I also had a look at the popaccount class, and I can not see the code path described in comment #37.
What I see is that the mail download happens in separate steps:
mails are fetched from the server (by the kio_pop slave) and only if they could be stored into the target folder (after the filtering), they are added to the internal list named idsOfMsgsToDelete for later deletion (which is a separate call). See PopAccount::slotProcessPendingMsgs(): there is:

    addedOk = processNewMsg(*cur); //added ok? Error displayed if not.

    if (!addedOk) {
      mMsgsPendingDownload.clear();
      msgIdsAwaitingProcessing.clear();
      msgUidsAwaitingProcessing.clear();
      break;  // >>>> so no delete here
    }
    else {
      idsOfMsgsToDelete.append( *curId );   // >>>> only if OK, they will be deleted
...
 

To proof that, I put a users home dir on an extra partition, sent him a few mails, filled the partition to 100% and tryed to retrieve the mails.

kmail detected that the disc is full and terminated with a Notify dialog.

Checking the POP mail server (doing this by telnet on port 110) I found all mails still there.

So I can not confirm that there is such a problem with POP servers and the users $HOME is full.
Also, I found no hint in the code that the received mail is somewhere temporarily stored elsewhere (I mean not below $HOME).
Comment 49 S. Burmeister 2006-11-01 15:56:47 UTC
So why not try with a full /, i.e. /tmp partition?
Comment 50 Martin Koller 2006-11-12 18:38:32 UTC
OK, just to be sure, I did now the test with also filling up /
=> same result (which is what I expected); I can not see that this bug is valid any longer.
I would close it - except there is anybody still somehow seeing this.
Anybody ?
Comment 51 S. Burmeister 2006-11-14 10:14:10 UTC
Yes, me! / Was full, new mail arrived, it was displayed in the list, I clicked on it and got "no subject"...
Comment 52 Jonathan Marten 2006-11-23 21:11:57 UTC
*** Bug 100354 has been marked as a duplicate of this bug. ***
Comment 53 Jonathan Marten 2006-11-24 22:21:03 UTC
*** Bug 106125 has been marked as a duplicate of this bug. ***
Comment 54 Martin Koller 2006-11-26 10:55:18 UTC
to #51: What you see is a corrupted _index_ file, and not a lost mail.
See comment #2 from the original reporter:
 2. deleting indices fixed that problem!

In fact the subject of this bug entry is wrong.
It should say "kmail index is corrupted when disk is full"
 
Comment 55 S. Burmeister 2006-11-26 11:47:55 UTC
What about comment #12 and others then?
Comment 56 Martin Steigerwald 2006-11-26 12:39:03 UTC
@Martin Koller: Well then probably another bug report "corrupted or out of date index files should be rebuilt automatically" should be opened. I know that for example the AmigaOS mail client YAM (http://www.yam.ch) actually rebuilds index files when they are out of date (I think it checks whether the modification date of the folder directory is newer than that of the index file, but I am not sure). It probably makes sense to split this bug report in several smaller ones...

I can open such a bug report but I first would like to get some feedback whether it makes sense.
Comment 57 Martin Koller 2006-12-09 18:03:41 UTC
I think this is then probably the same as Bug 122028
Comment 58 Michael Born 2006-12-11 09:15:02 UTC
Sorry, if this is the wrong place to post this. But I think it is connected to this bug.

Yesterday it happend again.
My /home folder ran out of space and kmail crashed. After I freed some megabytes on /home kmail refused to start because of a corrupted .inbox.index
I deleted that .inbox.index file and could start kmail again.
I think I did not lose any mail BUT a lot of "unknown" mails appeared as well as old deleted mails.
My inbox and sent-mail files are huge (730MB and 430MB). The "compact folder" option ("Ordner komprimieren" in german) seems not to work - last time I had this "out of space" crash I deleted all mail from before 2006. Now everything back to year 2000 is back :-(
I use the same Mail folder for years now. Everytime I updated to a new Suse Linux version I kept my mails. So I can not tell you with which kmail version I created the files.
Today I use a Suse 10.1 with KDE 3.5.1 Level "a" and Kmail 1.9.1
Comment 59 Jonathan Marten 2006-12-14 16:41:58 UTC
*** Bug 138765 has been marked as a duplicate of this bug. ***
Comment 60 Alexander Borghgraef 2007-02-14 11:27:38 UTC
Had the same problem on my Fedora 6 machine at work. Lost 2 mails that way, they changed to "No subject" when I clicked them, and vanished when I restarted Kontact. It was a freshly installed machine, so I doubt the /tmp directory was full. However, my home dir is an nfs filesystem on a fileserver, so it's quite possible that the network connection with the server boinked momentarily (it does that sometimes, unfortunately), and kmail had no place to write the mails. Hasn't happened since, so I can't give you more info.
Comment 61 Jakub Januszkiewicz 2007-03-19 08:50:53 UTC
This bug just hit me, too.

My / (with /tmp) and /home are separate partitions. /home ran out of space and KMail quit with a message like "KMail crashed because disk is full" (well something similar). I immediately started KMail again (before freeing some space on /home) and it started happening - clicking on *some* emails or applying filters erased the messages (no author, no title, no content).

I guess KMail should either refuse to start when any partition it writes to is full or (better) start in read-only mode (no mail fetching, filtering, moving/copying etc), possibly switching back to read-write mode when it detects that some space has been freed.

If this is just a corrupt index problem then KMail should either inform user what to do or do it automatically.

I'm using KDE 3.5.6, KMail 1.9.6.
Comment 62 Juan I. Guardia 2007-04-16 04:44:43 UTC
I can also confirm that this bug is still around (KDE 3.5.6, KMail 1.9.6).  My / (which includes /tmp) and /home are in different partitions.  However, as opposed to what is described by comment #61, it is / which ran out of space. /home still has plenty of space left.  As I checked mails this morning KMail went through the process as it usually does.  No warnings, no crashes, no message boxes.  Both my local as well as POP3 accounts were checked for incoming mail with no problems.  But when I checked their respective folders, however, only the supposedly NEW mails were bogus (Subject: No Subject; Sender: Unknown; Date: unknown; size: 177 B).  All my old mail is there, I can even open their attachments.  So this also confirms my mail folders are intact.

I haven't had a chance to closely study the KMail code, but I can conclude from the comments attached to this bug and my experience as described above that at some point in the mail checking chain of events the NEW messages must be stored SOMEWHERE in / (my guess would be /tmp from what I have read).  I can try to step through KMail with a debuger to see where in the code this happens and go from there, but I am sure the KMail developers are much better at this than I am.  With regards to comments #48 and #50, this IS fully reproducible, as it has happened to me twice under the same circumstances.  To reproduce:

1. Make sure /home is in a separate partition from / (and /tmp, in my case).
2. Fill up / so it gets to 100% capacity. /home should have plenty of space left.
3. Check your mail.  NEW mail is bogus (and lost forever, since it was deleted from mail server).  Old mail should be there.
4. Free up enough space in / .
5. Check mail again.  NEW mail (since step 3) is now as it should be.
Comment 63 Martin Koller 2007-04-16 19:06:49 UTC
SVN commit 654636 by mkoller:

BUG: 50462

I dare to close this bug report with this commit.
It basically automatically recreates the index whenever a "should never 
happen" situation occurs in getting data from the index and retries to get 
the requested information.
I hope it works as advertised ;-)



 M  +12 -0     kmfolderindex.cpp  
 M  +2 -0      kmfolderindex.h  
 M  +14 -2     kmmsgbase.cpp  


--- branches/KDE/3.5/kdepim/kmail/kmfolderindex.cpp #654635:654636
@@ -479,4 +479,16 @@
   return msgInfo;
 }
 
+void KMFolderIndex::recreateIndex()
+{
+  kapp->setOverrideCursor(KCursor::arrowCursor());
+  KMessageBox::error(0,
+       i18n("The mail index for '%1' is corrupted and will be regenerated now, "
+            "but some information, including status flags, will be lost.").arg(name()));
+  kapp->restoreOverrideCursor();
+  createIndexFromContents();
+  readIndex();
+}
+
+
 #include "kmfolderindex.moc"
--- branches/KDE/3.5/kdepim/kmail/kmfolderindex.h #654635:654636
@@ -80,6 +80,8 @@
   virtual QString indexLocation() const;
   virtual int writeIndex( bool createEmptyIndex = false );
 
+  void recreateIndex();
+
 public slots:
   /** Incrementally update the index if possible else call writeIndex */
   virtual int updateIndex();
--- branches/KDE/3.5/kdepim/kmail/kmmsgbase.cpp #654635:654636
@@ -1109,6 +1109,7 @@
 //-----------------------------------------------------------------------------
 QString KMMsgBase::getStringPart(MsgPartType t) const
 {
+retry:
   QString ret;
 
   g_chunk_offset = 0;
@@ -1145,7 +1146,12 @@
     type = (MsgPartType) tmp;
     if(g_chunk_offset + l > mIndexLength) {
 	kdDebug(5006) << "This should never happen.. " << __FILE__ << ":" << __LINE__ << endl;
-	break;
+        if(using_mmap) {
+            g_chunk_length = 0;
+            g_chunk = 0;
+        }
+        storage()->recreateIndex();
+        goto retry;
     }
     if(type == t) {
         // This works because the QString constructor does a memcpy.
@@ -1178,6 +1184,7 @@
 //-----------------------------------------------------------------------------
 off_t KMMsgBase::getLongPart(MsgPartType t) const
 {
+retry:
   off_t ret = 0;
 
   g_chunk_offset = 0;
@@ -1217,7 +1224,12 @@
 
     if (g_chunk_offset + l > mIndexLength) {
       kdDebug(5006) << "This should never happen.. " << __FILE__ << ":" << __LINE__ << endl;
-      break;
+      if(using_mmap) {
+        g_chunk_length = 0;
+        g_chunk = 0;
+      }
+      storage()->recreateIndex();
+      goto retry;
     }
     if(type == t) {
       assert(sizeOfLong == l);
Comment 64 S. Burmeister 2007-04-16 21:38:18 UTC
How does re-building the index help when the email is deleted on the server?
Comment 65 Amit Shah 2007-04-17 06:37:00 UTC
Also, this is fixing the symptom, not the problem. If there's no space while fetching mail, mail fetching should be turned off till space becomes available (in /tmp or in /home)
Comment 66 S. Burmeister 2007-04-19 12:09:02 UTC
Could this bug pelase be re-opened until the data-loss is fixed and not just the indexing?
Comment 67 Thomas McGuire 2007-04-19 15:15:56 UTC
>Could this bug pelase be re-opened until the data-loss is fixed and not just the indexing? 
Ok, I'll reopen it.
Comment 68 Martin Koller 2007-04-19 19:19:01 UTC
More or less all comments in this report are talking about "I click on a mail and I see nothing/unknown/etc."
I do understand that for a user this is perceived as "mail is lost".
But as I described already in comment #48, #50 and #54, all those perceived "data loss" issues are due to corrupted index files.
The only exception I found here is comment #12 - still, nobody else could reproduce this and also I could not reproduce it.

If anybody here can provide test results with the fix from current CVS (!) which produces a real data loss, then please tell me how exactly this can be reproduced.

Otherwise, please don't simply claim "there is still a problem" without checking what really happens.
Comment 69 S. Burmeister 2007-04-19 20:26:27 UTC
You are the only one claiming that there is no data-loss, yet there are several that claim the opposite, just read the comments, e.g. #62. You even claimed that there never was data-loss for any of us but this is just an indexing issue. Do you have any proof that all of us are wrong and you (as the only one in this bug-report) know what happend on our computers?

I lost email, period. Fixing indexing does not help, since I tried to delete those files and the index was re-build yet not email appeared, so those that were received are lost. (No, I do not use any auto-delete SPAM.
Comment 70 Adrien Cordonnier 2007-04-19 21:11:24 UTC
Moreover, when a new index file is rebuilt, data from the former index file are lost and depending of the amount of mail, this can be a lot of data.

For different reasons, I choose to delete mails on POP3 servers only after 7 days (one of the reason is because of this bug, another is to be able to reply even when I can only connect from a public computer though I downloaded mails a few days ago). When index file is rebuilt, all info such as if a mail as been reply or not are lost. And mails are downloaded again which lead to duplicate mails. Not so easy to sort when there are several hundreds.
Comment 71 Juan I. Guardia 2007-04-20 16:53:41 UTC
In response to comment #68, I realize that perhaps not many people have a setup such as mine.  But that does not mean that this bug is not reproducible.  As I said, it happened to me TWICE, under the SAME circumstances.  I have an external disk mounted in a directory under /mnt (which is in the same partition as / and /tmp).  I also have a backup script that backs up my /home directory to this disk everyday.  If for some reason I forget to mount the external disk the backup runs but all the data gets written to said directory under /mnt.  However since the disk is not mounted the data winds up on the / partition, thus filling it up.  Because / is much smaller than the external disk, it runs out of space, and this bug happens.

The issue here is not whether someone can or can't reproduce this bug because he/she doesn't have someone else's setup.  I think the root of the problem is that if the partition KMail needs as temporary storage for processing of incoming mail (be it filtering or something else) runs out of space, messages get lost and KMail is completely oblivious to the problem.  This has been evidenced by two COMPLETELY different situations, comment #12 and mine (comment #62).  I concede that I haven't tried Martin Koller's patch, and I promise to do so as soon as I get the chance.  In fact, if someone else can do it before I can, please post your results here.  Although currupt index files might be a result of this bug for some, I don't believe that has happened in my case because right now KMail is working perfectly since I have space left in / .  But as soon as I run out of it I am convinced this bug will resurface.
Comment 72 Thomas McGuire 2007-07-17 12:36:48 UTC
See also bug 83585, which is the same thing for IMAP.
Comment 73 S. Burmeister 2007-07-28 10:21:13 UTC
What could be more important then to fix a bug that causes data-loss, is known for 5 years and has that many votes? Templates? Anything?

For the record, I use bogofilter. The email is shown as unknown, there is not message left on the server, so it was deleted although /tmp had no space left. The content of the lost email is:

X-Bogosity: Unsure, tests=bogofilter, spamicity=0.520000, version=1.1.1
X-UID: 
Status: R
X-Status: NT
X-KMail-EncryptionState:  
X-KMail-SignatureState:  
X-KMail-MDN-Sent:  
Comment 74 Peter Huber 2007-10-02 10:19:03 UTC
I can confirm this: for IMAP/DMAP it is the same thing! I have lost about 20 mails today because of a full /home. (KDE 3.5.7 fc6, KMail 1.9.7)

I live with this bug for years now (different systems, different versions of KDE). It is very annoying. It would be great if someone could fix this. Thanks in advance.

Comment 75 Thorsten Staerk 2008-01-06 23:15:48 UTC
This has been fixed by commit 614058, see
http://websvn.kde.org/branches/KDE/3.5/kdepim/kmail/kmfoldermaildir.cpp?r1=614058&r2=614057&pathrev=614058

If this happens today, kmail finishes and shows a message: "KMail encountered a fatal error and will terminate now. The error was: Message could not be added to the folder, possibly disk space is low"
Comment 76 Maciej Pilichowski 2008-01-07 08:08:38 UTC
Why KMail terminates in such situation? It does not help in anything.
Comment 77 Martin Steigerwald 2008-01-07 22:47:53 UTC
While terminating the application is way better than a data loss IMHO it is not a proper fix for this bug. Maybe within KMail from KDE 3 nothing more is possible / sensible, but for KDE 4 there should be a more approbiate solution.

In my eyes, KMail doesn't need to terminate at all. It should just stop the mail download and tell the user to make some free space to download the mails into beforing trying downloading them again ;-).

Thus I would reopen the bug until a more suitable fix is made.
Comment 78 Thorsten Staerk 2008-01-07 23:43:14 UTC
Setting this bug to VERIFIED because I tested it.

I agree kmail's termination is useless, however if you do not like it, file a new bug report. Do not use the severity (grave) of this bug for an "easy" bug like "do not quit on error". Do not reopen.
Comment 79 Maciej Pilichowski 2008-01-08 09:22:20 UTC
Here:
https://bugs.kde.org/show_bug.cgi?id=155268
Comment 80 Andrew Crouthamel 2018-09-04 18:20:14 UTC
Hello! Sorry to be the bearer of bad news, but this version of Kmail has been unmaintained for many years so I am closing this bug. Please try using the latest version of Kmail to see if your issue persists. If it does, please submit a new bug in "kmail2". Thank you!