Bug 104956 - non-connected imap: spontaneous message disintegration with unstable network connection
Summary: non-connected imap: spontaneous message disintegration with unstable network ...
Status: RESOLVED FIXED
Alias: None
Product: kmail
Classification: Applications
Component: disconnected IMAP (show other bugs)
Version: 1.8
Platform: unspecified Linux
: VHI critical
Target Milestone: ---
Assignee: Till Adam
URL:
Keywords:
: 100859 111139 (view as bug list)
Depends on:
Blocks:
 
Reported: 2005-05-02 11:05 UTC by Carsten Pfeiffer
Modified: 2009-12-06 17:00 UTC (History)
16 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
backtrace (1.28 KB, text/plain)
2005-08-28 15:17 UTC, Ferdinand Gassauer
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 09:22 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 09:22 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 09:23 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 09:24 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 09:24 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 09:25 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 09:29 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 09:34 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 09:52 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 10:26 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 10:36 UTC, Till Adam
Details
patch against 3.5 with file handling safety and debug helpers (5.99 KB, text/plain)
2006-08-07 11:07 UTC, Till Adam
Details
Updated debugging patch (7.74 KB, patch)
2006-08-14 12:15 UTC, Till Adam
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Carsten Pfeiffer 2005-05-02 11:05:57 UTC
Version:           1.8 (using KDE 3.4.0, Debian Package 4:3.4.0-0pre4 (3.1))
Compiler:          gcc version 3.3.5 (Debian 1:3.3.5-8)
OS:                Linux (i686) release 2.6.11.6

Since my upgrade to KDE 3.4, I've had some sudden mail loss, several times. This happened on a notebook with a dimap account on a courier imap server.

I didn't actually see the mails disappearing, but at some point, I noticed that one mail folder was completely emptied. Another two times, a mail folder had lost a few  thousand mails.

I can only speculate about the reasons: this is a notebook, sometimes connected via wlan, the connection to imap server is over an ssh-tunnel and sometimes suspended to ram. Regular background mail checking is enabled.

So it might be that
1) kmail was synchronizing mailboxes while the network went down / the system was suspended
2) kmail started or continued synchronizing after system woke up again (network being available or not)

I remember seeing a messagebox a few times:

"Error while deleting messages on the server: Timeout" The address shown in that dialog is usually "(unknown)", the hostname is listed under "Additional information".

That dialog has a continue- and a cancel-button. I remember having pressed 'Continue' at least once and 'Cancel' at least twice, so I can't tell which option might be the cause -- provided that that dialog (or rather the actions before or after it) are related to the unanticipated mail deletion.
Comment 1 Jan Steffan 2005-05-03 19:06:21 UTC
I had a similar experience today, too ;-( 

Our Internet connection failed several times yesterday for a few minutes. 
Kmail complained about timeouts and connection losses while synchronizing
with the imap server.
Today I found several hundred mails gone both in the local cache and on the
server.

To me it seems, that kmail synchronized an incomplete cache back to the 
server. 

Kmail and KDE 3.4 packages are from Debian experimental.
Comment 2 Marcel Offermans 2005-05-04 09:54:31 UTC
I too have had a dimap folder completely deleted in a wifi environment which is known to have intermittent connection failures. At one point my folder contained a couple of thousand mails, the next moment, they were all deleted permanently from the imap server and I had to restore a backup. KDE 3.4 release on Gentoo.
Comment 3 konold 2005-05-26 01:01:42 UTC
I can verify the same problems with KDE 3.4
Comment 4 Till Adam 2005-06-11 18:57:40 UTC
*** Bug 100859 has been marked as a duplicate of this bug. ***
Comment 5 Gerco Dries 2005-07-13 13:45:39 UTC
I can confirm this as well, a few thousand messages suddenly dissappeared from my dimap folder. Also using a notebook on a WiFi connection. I was in the process of setting up my dimap over several machines so I might have done something wrong in the process, but it looks like it was this bug.

I switched to regular imap and all is fine now.
Comment 6 Carsten Pfeiffer 2005-07-13 15:22:18 UTC
On Wednesday 13 July 2005 13:45, Gerco Dries wrote:

> I switched to regular imap and all is fine now.


FWIW, I didn't lose any mail anymore since I always press the Cancel-button in 
that dialog.

Cheers
Carsten
Comment 7 Michal Palczewski 2005-07-25 22:06:08 UTC
I've gotten complete mail loss from my mailbox recently.  

I was connected to my imap server using kmail.  Then used another client to connect to it(webmail).  Kmail gave an error because it got disconnected.  When it reconnected(when I clicked on inbox again)  it deleted all my mail. 
Comment 8 Ramon van Alteren 2005-08-05 10:44:45 UTC
I can confirm this report as well with both 3.4 and 3.4.1

Dimap looses email if the connection is lost during synchronisation and you press the OK button in the timeout dialog.

It doesn't occur 100% though, it seems to depend on the sync state of the folder.

Ramon
Comment 9 Thomas Alexander Frederiksen 2005-08-08 11:53:43 UTC
This bug has happened to me as well, several times. I'm running KDE 3.4 on SuSE 9.3 at work, and can confirm that it will happen whenever the connection to the mailserver fails in mid-transfer, as well as most of the times kmail segfaults.

Basically, whenever the cache is smashed for some reason...
Comment 10 Thomas Beinicke 2005-08-09 01:33:23 UTC
It happened to me as well several times but I was using imap and not dimap.

It seems to be reproduceable if one switches between folders with a lot of mails quickly.

I was using courier-imap at that time but now switched to binc-imap though I don't think it has anything to do with the server itself.
Comment 11 Jan Jitse Venselaar 2005-08-11 23:52:18 UTC
Today I experienced the same problem. Mail server went down, when it went up KMail decided to delete thousands of mails, both on the client side, and on the server side. I was the only one experiencing mail loss because of the server crash.
After setting a backup back, kmail happily begins to delete all the mails again :(... Any work-around on how I can put my mails back?
Comment 12 Gerco Dries 2005-08-12 07:46:05 UTC
On Thursday 11 August 2005 23:52, Jan Jitse Venselaar wrote:
> ------- Additional Comments From janjitse a-eskwadraat nl  2005-08-11 23:52
> ------- Today I experienced the same problem. Mail server went down, when
> it went up KMail decided to delete thousands of mails, both on the client
> side, and on the server side. I was the only one experiencing mail loss
> because of the server crash. After setting a backup back, kmail happily
> begins to delete all the mails again :(... Any work-around on how I can put
> my mails back?


Als je een backup van de server hebt:
In KMail je de account deleten, je backup terugzetten en een nieuwe account 
aanmaken, KMail zal dan de DiMap cache opnieuw gaan opbouwen. Misschien werkt 
"Clear dIMAP cache ook wel".

Als je alleen een backup van de client kant hebt:
De backup zal waarschijnlijk in Maildir formaat zijn, deze kun je dan ergens 
extracten binnen je ~/Mail directory (~/Mail/backup ofzo). Zorg dat er in 
~/Mail/backup een 'new' 'cur' en 'tmp' directory aanwezig zijn en dat 'cur' 
alle mails bevat (dit werkt ook voor subfolders, gewoon meer directories in 
~/Mail/backup en daarin weer een 'cur' 'tmp' en 'new' directory.

Als het goed is staat de boel al in dat formaat, dus hoef je niets meer te 
doen dan extracten in ~/Mail/backup. Nu kun je KMail openen en zul je in de 
"Local folders' een 'backup' folder zien staan, daarin staan al je mails. 
Kopieer nu alles naar je IMAP folders en je hebt alles weer terug.

Ik heb hetzelfde gedaan toen KMail bij mij de boel sloopte, ben gelijk 
overgestapt naar gewone IMAP, iets minder snel, maar wel veiliger.

Groeten,
Gerco Dries.
Comment 13 Jan Jitse Venselaar 2005-08-12 11:25:53 UTC
> ------- Additional Comments From gerco gdries com  2005-08-12 07:46 -------
>
> On Thursday 11 August 2005 23:52, Jan Jitse Venselaar wrote:
> > ------- Additional Comments From janjitse a-eskwadraat nl  2005-08-11
> > 23:52 ------- Today I experienced the same problem. Mail server went
> > down, when it went up KMail decided to delete thousands of mails, both on
> > the client side, and on the server side. I was the only one experiencing
> > mail loss because of the server crash. After setting a backup back, kmail
> > happily begins to delete all the mails again :(... Any work-around on how
> > I can put my mails back?
>
> Als je een backup van de server hebt:
> In KMail je de account deleten, je backup terugzetten en een nieuwe account
> aanmaken, KMail zal dan de DiMap cache opnieuw gaan opbouwen. Misschien
> werkt "Clear dIMAP cache ook wel".
>
> Als je alleen een backup van de client kant hebt:
> De backup zal waarschijnlijk in Maildir formaat zijn, deze kun je dan
> ergens extracten binnen je ~/Mail directory (~/Mail/backup ofzo). Zorg dat
> er in ~/Mail/backup een 'new' 'cur' en 'tmp' directory aanwezig zijn en dat
> 'cur' alle mails bevat (dit werkt ook voor subfolders, gewoon meer
> directories in ~/Mail/backup en daarin weer een 'cur' 'tmp' en 'new'
> directory.
>
> Als het goed is staat de boel al in dat formaat, dus hoef je niets meer te
> doen dan extracten in ~/Mail/backup. Nu kun je KMail openen en zul je in de
> "Local folders' een 'backup' folder zien staan, daarin staan al je mails.
> Kopieer nu alles naar je IMAP folders en je hebt alles weer terug.
>
> Ik heb hetzelfde gedaan toen KMail bij mij de boel sloopte, ben gelijk
> overgestapt naar gewone IMAP, iets minder snel, maar wel veiliger.
>
> Groeten,
> Gerco Dries.



Bedankt!
Dat heeft geholpen. Ik stap ook maar over op gewoon imap, want zomaar +/- 2000 
mailtjes kwijtraken vind ik wat eng...

Groeten,

Jan Jitse
Comment 14 Ferdinand Gassauer 2005-08-28 14:55:58 UTC
*** Bug 111139 has been marked as a duplicate of this bug. ***
Comment 15 Ferdinand Gassauer 2005-08-28 14:59:19 UTC
Hi!

as described further in http://bugs.kde.org/show_bug.cgi?id=111139
today ALL (!!)  subfolders were lost at once.

compiled KDESVN as of today Aug 28th 2005.

IMHO more than critical!!!
Comment 16 Ferdinand Gassauer 2005-08-28 15:17:12 UTC
Created attachment 12407 [details]
backtrace

Not sure that this has something to do with the problem
I was sending a mail AFTER all the subfolders have been deleted (including the
imap sent-mail destination)
Comment 17 Thomas Beinicke 2005-08-28 15:28:59 UTC
It happened to me again too and it's difficult to track. All the people who have that problem do you use kmail only or kontact?
Comment 18 Ferdinand Gassauer 2005-08-28 22:00:32 UTC
Hi!
I use kontact!

IMHO a warning should be issued to discourage people to use this kmail/imap version in a production system, at least without a good backup system.

cu
Comment 19 Thomas Beinicke 2005-08-28 23:16:05 UTC
Do you use the preview function in Kontact?
The one that shows how many mails you have in a new folder on the summary page.
Maybe it's totally wrong but I disabled it and since then it didn't happen anymore but maybe I was just lucky.
I am just trying to pin down the problem.
Comment 20 Ferdinand Gassauer 2005-08-28 23:49:44 UTC
yes - I turned it off now.

I quote my sysadmin:
"der slox-cyrus ist eine krücke, der wurde gar schändlich misshandelt um 
mit dem comfire zu funktionieren. so wurden zb die delimiter auf '/' 
geändert. es kann durchaus sein, dass kmail darüber stolpert."

may be this helps.
Comment 21 Ferdinand Gassauer 2005-08-29 09:36:01 UTC
Hi!
I quote my sysadmin again:
"erstaunlicherweise waren im backup von user/gass/EDV die magischen files 
cyrus.* nicht vorhanden, die man normalerweise als user nicht entfernen 
kann ausser man löscht den imap-folder in dem die liegen, was aber nur 
dann geht wenns keine subfolder gibt. einen bug im slox-cyrus kann ich 
daher nicht ausschliessen.
vorsicht bei der verwendung von acls via imap und timsieved via port 
2000. die sind hier sicher für den comfire verbogen."

hope that helps
Comment 22 Thomas Beinicke 2005-08-29 09:44:16 UTC
I don't think it's a bug in the imap server.
I tried it with courier-imap and binc and had the same problems.

I also don't think it's a speed related problem since I have my server in my LAN. Also I only use imap and not dimap.
Comment 23 Norbert Schuch 2005-08-29 22:37:56 UTC
I experience a related problem (although it's maybe not the same bug). It
happened, I think, twice, that kmail wasn't quit properly (one time, I killed it,
the other time, my system crashed completely). In both cases, after restarting
kmail, there were duplicate messages, and there appeared a few completely empty
messages (kmail displays them as "No Subject" etc.). Seemingly, kmail gets messed up with local and remote folders. Also, when deleting the
duplicate messages, some of the remaining ones are transformed to empty messages
and thus completely lost. The empty messages really exist on the IMAP account, I
checked with pine. After a few starts of kmail this gets "stable", but it's still severe. Normal IMAP is not a good alternative as it does not support automatic filtering.
KMail 1.7.2, KDE 3.3.2, Debian Sarge
Comment 24 Andreas Gungl 2005-08-30 09:32:47 UTC
Am Montag, 29. August 2005 22:37 schrieb Norbert Schuch:
> KMail 1.7.2, KDE 3.3.2, Debian Sarge


If you're using disconnected IMAP, you better upgrade to the latest KMail 
despite the problems it may have. There have been a lot of fixes since. And 
having KDE 3.5 out in a few weeks, there is almost no chance to get 
something fixed in KDE 3.3.x (KMail 1.7.x).
Comment 25 Ferdinand Gassauer 2005-08-30 10:08:38 UTC
just want to say that I lost the mails in 3_5 branch too.

so your sugestion does not help in this respect
Comment 26 Andreas Gungl 2005-08-30 20:31:15 UTC
Am Dienstag, 30. August 2005 10:08 schrieb Ferdinand Gassauer:
> just want to say that I lost the mails in 3_5 branch too.
> so your sugestion does not help in this respect


You're right. But there have been so many fixes in KMail 1.8 that it's worth 
upgrading nevertheless.

BTW, I've been using dIMAP for about two years now. I've never had a message 
loss (except when I deleted myself a folder by accident  - well, we had a 
backup).
The main problem is to reproduce such a situation, the same goes for the 
"deleting many messages crashes KMail" problem...
Comment 27 Ferdinand Gassauer 2005-08-30 20:46:29 UTC
I agree with you, that it's worth updateing.

IIRC, before the mail loss (3 times in a week from 2 different boxes with KDE 3.4.2 from SUSE and copiled 3.5 sources) I was working almost always with kmail and NOT with kontact.

so I am back working with kmail, which I have used without mail loss also for  some years now.
Comment 28 Thomas Beinicke 2005-09-01 13:16:57 UTC
Ok, it happened again just now :-/
All the folders except INBOX, sent and trash are gone.

Unfortunately I don't have a backtrace but I can describe what I have done.
I am running an SVN checkout version from yesterday.

I was writing an e-mail to a German friend. ( I changed the spell checker from English to German (view->dictionary).
After that I wanted to check the whole mail again with tools->spelling.
There English was still selected as dictionary language so I changed it to German and kmail told me I have to restart it. I clicked ok and kmail crashed.

So I thought I lost my mail but tried to start kmail again. So kmail popped up again including my mail (which is nice) just to crash again 2 seconds afterwards.

So now I opened kmail for the third time and it didn't crash but then I realized that ALL my folders and it's content except the ones mentioned above were gone again.

I am using imap and not dimap so it's not only limited to dimap.

If I can provide any more information I will do so. I hope next time I can include a usefull backtrace.

I think we should really fix this somehow before kde 3.5 since such a serious bug would render kmail useless which would be very sad.
Comment 29 Thomas Beinicke 2005-09-01 13:30:06 UTC
One update on the crash.
It seems as only folders in the root folders are deleted.
For example, I had a folder ML&F for mailing lists and forums.
That folder got deleted but the subfolders are still there. So I created the folder ML&F again and so I could access the subfolders again and they still contained all the e-mails.

Comment 30 Ferdinand Gassauer 2005-09-01 14:08:50 UTC
Thomas: question, you say you are using kmail and not kontact.

I can't remember that I have lost folders (or anything else) using kmail itself.

Comment 31 Ferdinand Gassauer 2005-09-01 14:09:44 UTC
BTW I use imap here too.
Comment 32 Thomas Beinicke 2005-09-01 15:56:21 UTC
Yes, I was just using kmail and not kontact since I suspected the problem lying within kontact but it seems like I was wrong, I was running kmail standalone.
I just finished compiling a new svn checkout with full debug symbols and will try to make it crash again.
Comment 33 Carsten Burghardt 2005-09-02 15:46:48 UTC
> I am using imap and not dimap so it's not only limited to dimap.


That is new to me. Are these folders actually deleted on the server?


Carsten
Comment 34 Ferdinand Gassauer 2005-09-02 16:02:05 UTC
for kmail config says type=imap
once the accounts were not accessible, twice they have been deleted completely
Comment 35 Thomas Beinicke 2005-09-02 17:32:46 UTC
Yes, they are deleted from the server.
Comment 36 konold 2005-09-02 18:46:56 UTC
My research together with Till Adam showed that we think that the problem is independent of kmail or kontact. Technically always the same kmail code is used. 

I myself observed the problem twice in the past but it did not happen to me since some months in the stable 3.4.x branch.

The root of the problem seems limited to dIMAP and not related to the IMAP resource.

Basically what happens is that the current code uses implicit instead explicit deletion of messages which leads to data loss in case something goes wrong with the indexing. So if the local dIMAP cache gets corrupted for whatever reason the sync process cannot find any local messages anymore and assumes that it shall delete all messages on the server :-(

The solution to this problem is to make the deletion of messages explicit.

Basically this boils down to keep a FIFO of commands which shall be transfered to the server. As a side effect this will also speed up dIMAP significantly and allows for smart optimizations of the communication latency and bandwith.

Unfortunately the current kioslave approach is not well suited for this so a new implementation is required.

In the meantime I highly recommend to upgrade to the most recent stable version as I cannot reproduce the problem anymore.


 
Comment 37 Thomas Beinicke 2005-09-02 21:23:43 UTC
Well, as I stated above I am using kmail from svn which should have even more bugs squashed and I am using imap and not dimap.
Should I file a new bug report then although the problems seems to be the same?
Comment 38 konold 2005-09-02 22:36:46 UTC
I think it is interesting to know that the problem also happens with imap not only dIMAP but I cannot imagine which common bug causes it.

Which exact version are you running from svn?
Comment 39 Thomas Beinicke 2005-09-03 04:16:28 UTC
I am updating svn once or twice a week so it's really difficult to say which version I am running. But I have been having that problem for some time now.
I am trying to find a way to reproduce the loss.
I could do it once like described in post nr. 28.
Comment 40 Ferdinand Gassauer 2005-09-12 20:02:12 UTC
FYI: I just managed to crash kmail (svn of today), it destroyed the local imap files, but luckily nothing on the server, so kmail started downloading and recovering everything automatically. 
Unfortunately I have lost the backtrace.  
Comment 41 Bastian Venthur 2005-10-10 10:08:22 UTC
I've posted two[1,2] different(!) grave bugs to the debian BTS against kmail in which kmail and dimap lose mails. I don't know which of the two applies best to the original poster, since both bugs are quite similar but not the same.


HTH

Bastian

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=321102
[2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=332473
Comment 42 Manfred 2005-10-20 18:53:27 UTC
This might be the same bug? One user uses Kontact/KMail and experienced the following on multiple mails:
Most of the time everything works well. Today we could verify that probably 
KMail has somehow destroyed the body including attachements of mails that 
have been moved (drag&drop) from IMAP-Server to local folder (maildir). No 
error messages, Counter in folder view is raised by one, mail deleted on 
server.
This happened to multiple mails. Unlikely that something with disk has to do 
with it because all mails have the email headers fine, just nothing left of 
the email body and any attachements. The IMAP Server is running fine since 
years (Courier IMAP with maildirs from qmail). [Debian 2.6.11-kanotix-11, Kontact 1.1.1, KMail 1.8.1] Is it worth trying to upgrade from Debian KDE 3.4.1 to 3.4.2? Doesn't sound like it?
Comment 43 Rob Kaper 2005-11-21 09:58:11 UTC
I've had similar problems with DIMAP. Data loss seems to be possible when Kontact/KMail crashes during a folder content upload.

Last week I lost all e-mails (local and server-side) received since July 25, so (in my case) the loss is not related to the last synchronisation or use of KMail - I suspect it has to do with the state of resynchronisation.
Comment 44 Kusi 2005-12-09 14:57:56 UTC
same here. Server side mail loss in inbox. Somehow the header cache got out of sync/was deleted and all messages were downloaded again. In the middle of this process (Okt 2004 was just downloaded), kmail crashed. All mails dated between start and Okt 2004 were in the inbox now, and the newer ones were deleted.

Same situation as previous #43. The dangerous part seems to be the (useless) re-downloading of the messages. If this crashes (maybe I just killed kmail, can't remember) something is going wrong
Comment 45 Kusi 2005-12-12 14:12:15 UTC
I just read the technical reason for this bug (Konold, poster #36) which exactly fits with my mail loss. As a temporary bugfix, is it possible to check the consistency of the dimap cache in prior to the sync process? How would a quick check have to look like? If the check fails, the entire cach has to be invalidated.

As for me, cachsize@now >= cachesize@previous_mailcheck, since I don't delete any mails. Would it be enough to compare the two cache sizes and spit a warning when the actual cache size is smaller than the previous cache size? This would be a very easy consistency check.


Comment 46 Kimmo 2006-01-31 19:10:04 UTC
Hi,

sadly engough, but I can confirm data loss with imap on a SuSE 9.3 + KDE 3.5.0 (KMail 1.9.1) machine (with all updates). I have today lost the _whole_ content of one my imap account, when KMail crashed during folder content upload :(
Comment 47 Bastian Venthur 2006-02-15 12:09:30 UTC
Hmm just lost again a few hundred mails, thanks dIMAP...

Since this bug is nearly open for a year somebody should disable dIMAP support until this problem is fixed or at least put a big fat warning to users who want to use it. 
Comment 48 Bastian Venthur 2006-02-15 14:14:25 UTC
Can a KMail developer confirm whether it is save or not to use IMAP instead of dIMAP? Does IMAP have the same problem?

I've lost too much mails today and now I have to decide whether to go on using KMail with IMAP or use something else until KMail fixed it's problems.
Comment 49 Daniel Hahler 2006-02-15 19:16:38 UTC
I'm not a dev, but use IMAP (with filtering).

I've never lost mail (knock-knock-knock) and was just annoyed that KMail has been very slow on displaying mails with a spamcop.net account, but luckily even this works fast now again. (using Ubuntu Dapper packages)
Comment 50 Adam Porter 2006-02-16 03:45:03 UTC
FYI, I'm using Debian's KMail 1.8.3 with KDE 3.5.1, and my dIMAP accounts have not lost any mail yet.  I downgraded from 1.9.1 because of other problems, but all's well here.  But I make regular backups on my desktop computer and on the server, so if KMail ever did lose some, I could recover them.
Comment 51 Bastian Venthur 2006-02-17 00:23:32 UTC
Just for the record, I use KMail 1.9.1 (KDE 3.5.1) and lost mails on my dIMAP account yesterday. 

There seems do exist a corelation between losing mails an rebuilding the index file (via rightclick on a folder and the solve-imap-problems-item [sorry translated from german version]).
Comment 52 Tom Christensen 2006-03-17 00:25:15 UTC
I am working on a bug which is a little related to this with pop accounts deleting mails.  It seems to me that all of these weird "mail loss" problems come from using the KIO slave object in basically an async mode to communicate with the mail server.  The POP lost mails bug is caused by this as well, because any error that occurs on the client machine cannot be passed to the KIO process, and so that process happily goes on its way and deletes things on the server.  This is a serious architechtural problem and needs to be fixed quickly.
Comment 53 Arne Schmitz 2006-04-03 16:15:35 UTC
Hm, I am using dIMAP, but never lost any mail -- I think! What is the status of this bug? How severe is it? When does this happen?
Comment 54 Adam Porter 2006-04-13 11:01:02 UTC
FYI, since upgrading to KDE 3.5.2 (Debian), my KMail IMAP problems and crashes have gone away.  If you have had this problem, please test with 3.5.2.  If we don't get any more reports in, say, a couple of weeks, I think we should consider closing this bug.
Comment 55 Julian Mehnle 2006-04-13 12:01:45 UTC
No, this bug is too serious to be closed easily.  You should wait for at least some _explicit_ feedback that the issue has really been resolved.  Also, some users may not be using self-compiled upstream source directly but prepared distributions, which usually take a while before they distribute new releases.  For example, I'm using Debian/Testing where it could take a month or two before KDE 3.5.2 becomes available.
Comment 56 Bastian Venthur 2006-04-13 19:39:30 UTC
ACK, this bug is hard to reproduce as a KMail-developer already confirmed and I would not close this bug, just because it has not been seen for a while. I for one hat the last occurence of this bug two months ago with 3.5.1 -- the day I switched to thunderbird.

Could a developer say something about this bug? Would be nice to hear something like "we know what the problem was, it has been fixed now".

BTW: Kmail won't hit testing until this bug is fixed.
Comment 57 Carsten Pfeiffer 2006-04-16 14:02:44 UTC
On Thursday 13 April 2006 19:39, Bastian Venthur wrote:

> ------- ACK, this bug is hard to reproduce as a KMail-developer already
> confirmed and I would not close this bug, just because it has not been seen
> for a while. I for one hat the last occurence of this bug two months ago
> with 3.5.1 -- the day I switched to thunderbird.
>
> Could a developer say something about this bug? Would be nice to hear
> something like "we know what the problem was, it has been fixed now".
>
> BTW: Kmail won't hit testing until this bug is fixed.


I for one haven't experienced this for quite a while -- not even the 
aforementioned dialog box.

The only weird dialog box I currently get is one telling me, that kmail can't 
create some folders on the server (that have been there for ages), asking if 
I want to continue anyway. I've always hit 'yes' and haven't had a problem 
with that.

Cheers,
Carsten
Comment 58 Ismail Donmez 2006-05-06 00:38:27 UTC
This looks like related to bug #63353 , I also added some symptoms I see there.
Comment 59 konold 2006-05-08 07:04:30 UTC
I can not reproduce the bug since any 3.5.x version. (It occured to me about every 2-3 weeks before.
Comment 60 Carsten Pfeiffer 2006-05-08 13:35:17 UTC
Same here, didn't have any mail loss anymore.
Comment 61 Ferdinand Gassauer 2006-05-08 21:23:05 UTC
same here, but what bothers me somewhat is that no one claims that he has fixed the bug or found a reason.
Comment 62 Bastian Venthur 2006-05-08 21:33:41 UTC
"but what bothers me somewhat is that no one claims that he has fixed the bug or found a reason."

Good point, I really doubt this bug has vanished "somehow", plus: 3.5.1 still had this bug! Please note this bug was always hard to reproduce (just look how long this bug has been open!) and since it seriously affects valuable user-data please don't close it to frivolous.

If it's really fixed, it should be possible to provide the the relevant svn-log or something.
Comment 63 Martin Steigerwald 2006-05-08 22:41:56 UTC
"If it's really fixed, it should be possible to provide the the relevant svn-log or something."

Not necessarily. Someone may have fixed it without noticing it while working on something else related to DIMAP. But how to tell?

Usually I would say close it and reopen it when it appears again. But since its a data loss bug I suggest leaving it open for quite a while and see whether there are further findings.
Comment 64 Jan Steffan 2006-05-08 23:02:43 UTC
have a look at comment #36 and #52. What I read from this is that the developers seem to know the reason for this bug, but fixing it would require far reaching changes in the design. 
So I absolutely don't understand this discussion about closing this bug just in case that it magically might have fixed itself... 
Comment 65 Ramon van Alteren 2006-05-08 23:19:00 UTC
I had a fairly reproducable test-case once....

Too dependent on my imap mail, so switched away from kmail since this bug bit me in the arse several months ago twice.

Testcase:

Pick up a fairly large collection of mail and setup kmail to sync it with the imap server on dimap. Fairly large was, in my case, >9K mails

Receive some new mails and generate a sync time-out (yank net cable from laptop)

Click OK on the time-out dialog

=> Mail loss.

Not 100% reproducable, seems to depend on markers in the underlying dimap folder state whether or not mail-loss occurs.

However fairly reproducable, 1 out of 3 would be a hit :-(

I don't have time to debug or test this, I do have a mail-archive that can be used to debug and I'm willing to distribute, it's a public mailing-list archive
I can even bounce emails to someone willing to test, just don't have the time myself....

I'd hate to see this bug closed without proper resolve, I'd like to return to Kmail for various reasons, not in the least because of the missing attachments feature ;-) I will not though if this bug isn't closed with a *known cause*, won't risk losing mail again and do not have the resources to implement backup to guard against it.

Ramon
Comment 66 Arne Schmitz 2006-05-16 14:11:36 UTC
Bam! I had this bug right now. Using KDE 3.5.2 on Debian. Lost a whole folder of mails (thank god it was only a ML). Nevertheless this is totally uncool...
Comment 67 Adam Porter 2006-05-17 05:34:12 UTC
Arne, thanks for reporting it.  I'm also using 3.5.2 on Debian.  But could you please provide a more detailed account of how it happened and how you noticed it?

On another note, what would be the best way to recover from this bug when/if it bites?  I'm thinking of a very frequent rdiff-backup of one's local Maildir; maybe every 5 minutes or so, nice'd down to avoid too much system slowdown.  Backupninja makes it quite easy to use rdiff-backup, and recovery is very easy too (I once had to recover my Firefox profile from a few days earlier; one command restored the directory as it was to a temp directory, and then I just moved it into place).
Comment 68 Tom Albers 2006-06-18 18:31:10 UTC
Ramon, reproducing this bug as you described was not possible. 

So this is a call out to everyone: please provide a reliable way to reproduce this mail loss. If anyone has a way to reproduce, we can hunt down this bug. 
Comment 69 Timo Springmann 2006-06-18 23:43:58 UTC
This bug hit me another time. Last week I lost my complete inbox (luckily we do regular backups). Maybe a brief description of the circumstances can help to track down this bug. Here's what happend:

In our company we store home directories of users on a single (nfs-)server and the client computers mount them using nfs. Last week this server stopped working. Because the complete home directory on the clients is mounted via nfs the clients also hang (waiting for timeout). One of the harddisks of the nfs server caused several system crashed, so the server got rebooted several times... sometimes the running kde session on the clients survived the reboot of the nfs server but some downtimes were too long... 

After replacing the broken harddisk and rebooting server and clients, many (but not all) of my kde settings were lost (like background picture, font settings, style setting, some kmail setting etc.). Also my complete inbox got deleted.

The mailserver runs on a different machine and wasn't affected by the broken.. I'm running debian/sid with latest KDE (3.5.3) and using kontact with disconnected imap to access my imap account on a cyrus mail server.

Hope this helps.
Comment 70 markus 2006-06-19 03:13:54 UTC
Dear community,

perhaps I can contribute with a symptom: One of my kolab / kontact users 
recently complained about the following: Emails come in into the inbox and 
when he clicks on such an email, it suddenly changes into a blank mail, 
containing nothing but "unknown subject", "unknown sender" and "unknown 
receiver". It seems to be due to the fact that his client computer crashed 
some time before and that the index was somehow corrupted. 

I could state that on the server side, the email contained just a short 
string "X-UID:0" or something like that. Obviously the client index 
inconsistency was reflected back onto the server again, and there the email 
was deleted, too. 

Perhaps the issue is due to some index corruption on the client side, which 
can happen at the moment of a client crash. Obviously the client side index 
is not crash-safe. 

I'm not a programmer and I don't know anything about the kmail code, but 
perhaps a start point of these problems would be the question where a client 
crash can corrupt the index - as such index corruptions will be reflected to 
the server and there the emails get deleted. - so that nothing will be left. 

Best wishes,

Markus
Comment 71 Pierre Habouzit 2006-06-28 18:31:55 UTC
I have a way to reproduce kmail bug, that "simulates" a NFS failure and basically shows the weakness of kmail.

1) setup a new folder with only one mail in it, and sync your dimap account fully.
2) quit kmail.
3) go to your ~/.kde/share/apps/kmail/ and find the .fooo.uidcache file chmod it to '0000' instead of 0644.
4) add some mail on the imap server in that folder.
5) start kmail, force a sync of that folder
6) chmod the .uidcache to 0644 again, and force kmail sync again
----> this makes presumably the kio_imap4 crash (at least my kmail tells that the transmission to the imap server broke).
7) force sync *again* and kmail says "no new mail"
----> go back to your imap server: mails have been deleted, only the one single mail that was put at stage 1) remains


I know this is a bit perv to do that, but it simulates quite well the fact that if for some reason kmail can't write its uidcache files correctly, it still assumes it wrote it correctly, and do no consistency checks wrt that. so if for some time that file goes wrong, next time kmail starts, if the file is here, it kio_imap4 crashes a first time (whereas it should in fact detect a cache poisoning), and next time you loose your mail.
Comment 72 Pierre Habouzit 2006-06-28 18:37:44 UTC
I suppose similar problems can occur with index files too. the problem is that kmail always trust .index/.uidcache files, even when the previous kio_imap4 run crashed, or last kmail crashed.

a simple workaround would be that when a transaction is started with that or this folder, a file stating it must be created (like a lock file). that MUST be done with care. if that file already exists, then NOTHING must be done, and the user must be warned that something went wrong, that he should backup the mail in that folder somewhere else, kmail then *HAS* to sync that folder from scratch (invalidating its whole .index/.uidcache files completely, not trusting them *AT ALL*) and let the user take the files that he backed up back if needed.

kmail could even automatize that, and let the user use the ^-* shortcut to avoid useless dupes, or ...
Comment 73 Arne Schmitz 2006-06-28 18:46:04 UTC
Am Mittwoch, 28. Juni 2006 18:37 schrieb Pierre Habouzit:
> a simple workaround would be that when a transaction is started with that
> or this folder, a file stating it must be created (like a lock file). that
> MUST be done with care. if that file already exists, then NOTHING must be
> done, and the user must be warned that something went wrong, that he should
> backup the mail in that folder somewhere else, kmail then *HAS* to sync
> that folder from scratch (invalidating its whole .index/.uidcache files
> completely, not trusting them *AT ALL*) and let the user take the files
> that he backed up back if needed.


Interesting idea. How do other MUAs handle this? For example how does 
Thunderbird handle (D)IMAP accounts?

Arne
Comment 74 Pierre Habouzit 2006-06-28 19:22:37 UTC
I've rethought a bit, and here is IMHO the explanation for the many instances of the dimap mail loss bug (that I really think is not in one but many places).

dimap is stateful, and kmail then has to trust a set of file to track what is already synced and what needs to. Those files are critical, because *any* corruption has to be noticed, and in case of corruption you have to fall back to the previous state.

When I tried to make kmail loose mails (in #71) I expected it to crash, or to say "internal error" because he can't write/read the uidcache file. which it didn't. so what has to be done is (IMHO):

(1) list all the files responsible for the statefulness of dimap in kmail (uidcache is part of them) ;
(2) track all uses of those files in the kmail source code, and CATCH THE DAMN ERRORS, and in case of errors, fail loudly.

If it happens to do things like:

a. begin transaction (read uidcache)
b. do a lot of things with our local cache of mails
c. end transaction (write uidcache) <--- fails here *bang*

then kmail should try hard to fall back to the state of (a)

Moreover if kmail craches during (b), it must have a way to tell he was in the middle of a transaction. A way could be to write in the uidcache (or like I already suggested to create a new file, but one more is maybe one too much ;p) that kmail is starting a transaction. When you restart a transaction, and that it already says that a transaction is started, then you are recovering from a previous crash, and you MUST HANDLE IT WITH GREAT CARE.



What I proved with my way to reproduce the bug is that kmail trusts files that it does not deals with with enough care. so it trusts crap, and no surprise it breaks from time to time.


and since kmail design of implicit deletion is fragile, (like kmail developers already aknowleged it) that quite big flaw of handling of index files, becomes a terrible catastrophe.
Comment 75 Rob Kaper 2006-06-28 22:16:26 UTC
Anyone who next experiences this bug please check whether all data is still present locally (and if so, back up) before starting KMail/Kontact again. Perhaps check the IMAP account with a different client, not supporting DIMAP.

I've checked the IMAP RFC a few times and I think the loss of data does not occur before or during the crash but when KMail starts again with an incomplete index and then starts deleting/expunging all messages it doesn't know about. In the IMAP protocol messages are marked for deletion and then expunged. Clients should not obviously not remove actual data until explicitly receiving the expunge responses per message and servers should not expunge messages not explicitly communicated to be. Neither would make sense to happen for to-be-kept messages during synchronisation even in the case of a crash and/or network failure. However Kmail expunging the messages (and only then delete them locally upon success) starting with an incomplete index would cause the deletion.

If not useful for locating the bug, checking the actual state of the data locally and on the server might prevent data loss for users experiencing it, or enable them to backup and restore.

I cannot explain this problem at all unless at some point KMail trusts a list of messages to be kept (presumably the index) instead of a list of messages to be deleted and then imposes that trust on the server.  I recommend someone check if this could indeed occur and then make sure DIMAP only relies on an explicit deletion list instead. This might turn out to be a large change, but from a design perspective anything else means asking for problems.

(I swear, if I find the time I'll just have to get back to KDE development.)
Comment 76 Pierre Habouzit 2006-06-28 22:29:18 UTC
Le mer 28 juin 2006 22:16, Rob Kaper a écrit :

> (I swear, if I find the time I'll just have to get back to KDE
> development.)



well, following my previous remarks, I've grepped kmail sources 
for 'uidcache' in the kmail directory of kdepim sources. here is what I 
found:

  there is a nice function:
    KMFolderCachedImap::readUidCache that returns -1 if something failed
    at read time (that's good).

  there is a less nice function:
    KMFolderCachedImap::writeUidCache that returns errno (not knowing if
    it's set or not if we believe the comment) on error.

    meaning that here already, there is undefined behaviour on the
    uidcache files.

let's see where those are used:

  * readUidCache is used only in KMFolderCachedImap constructor:

    KMFolderCachedImap::KMFolderCachedImap( KMFolder* folder,
                                            const char* aName )
        : [ lots of initializations ]
    {
        setUidValidity("");
        readUidCache();
        mProgress = 0;
    }

    *BOOOOOOOOOOOOOOOHHHHH* that's horrible, if the uidcache reading
    fail we skip that silently BOOOH BOOOH BOOH THAT'S HORRIBLE ! and is
    obviously a first cause for a crash, because in taht case lots and
    lots of members are not initialized correctly.


  * writeUidCache is used many times. and the (innacurate) return value
    is actually *NEVER EVER* used. so kmail just assumes it has been
    written.

    BOOOOOOOOOOOOOOOOOOOOOOOH again


I'm almost sure that fixing that will fix 99% of the dimap mail losses. 
I'll see if I can create a patch of this, but I'm not very sure, I 
don't know kmail's code at all.
Comment 77 Pierre Habouzit 2006-06-28 23:12:21 UTC
Le mer 28 juin 2006 22:29, Pierre Habouzit a écrit :

> I'm almost sure that fixing that will fix 99% of the dimap mail
> losses. I'll see if I can create a patch of this, but I'm not very
> sure, I don't know kmail's code at all.


to add the "Coup de Grâce":

int KMFolderCachedImap::writeUidCache()
{
  if( uidValidity().isEmpty() || uidValidity() == "INVALID" ) {
    // No info from the server yet, remove the file.
    if( QFile::exists( uidCacheLocation() ) )
      unlink( QFile::encodeName( uidCacheLocation() ) );
    return 0;
  }

  QFile uidcache( uidCacheLocation() );
  if( uidcache.open( IO_WriteOnly ) ) {
    QTextStream str( &uidcache );
    str << "# KMail-UidCache V" << UIDCACHE_VERSION << endl;
    str << uidValidity() << endl;
    str << lastUid() << endl;
    uidcache.flush();
    fsync( uidcache.handle() ); /* this is probably overkill */
    uidcache.close();
    return 0;
  } else {
    return errno; /* does QFile set errno? */
  }
}

is completely broken: it uses QTextStream that has absolutely *NO* error 
handling, so you may open a file, not beeing able to write to it due to 
NFS errors, and close it without problems, and return with a nice 0 
status (that is ignored anyway). yay \o/


uidcache in kmail is FUBAR from head to toe, and IS the culprit without 
any doubt possible of all mail losses in dimap. The fact that it's hard 
to reproduce is that it causes problems iff kmail thought it had 
written the uidcache, and it didn't. And kmail has two ways to write an 
uid cache:

 * on timers (I've seen that in the source) ==> hence the fact that a
   lot of reports say that very big directories trigger the bug more
   easily).
 * or when the sync is done completely.

In fact, this has nothing to do with the fact that kmail uses implicit 
mail deletion (or at least, it's not the worse thing in all the bad 
designs that lead to that nasty bug).



at least for writeUidCache, I'd suggest something like:

int KMFolderCachedImap::writeUidCache()
{
  if( uidValidity().isEmpty() || uidValidity() == "INVALID" ) {
    // No info from the server yet, remove the file.
    if (QFile::exists(uidCacheLocation())) {
      /* YES UNLINK CAN FAIL TOO AND IT'S A PROBLEM TOO !!! */
      return unlink( QFile::encodeName( uidCacheLocation() ) );
    }
  }

  /* yes C APIs are good when we need to deal with errors... */
  FILE *f = fopen(uidCacheLocation(), "w");
  if (!f)
    return -1;

  if (fprintf(f, "# Kmail-UidCache V%d\n", UIDCACHE_VERSION) < 0
  ||  fprintf(f, "%ld\n", uidValidity()) < 0
  ||  fprintf(f, "%ld\n", lastUid()) < 0)
  ||  fflush(f))
  {
      fclose(f);
      return -1;
  }

  /* YES FCLOSE CAN FAIL IF THERE IS QUOTAs INVOLVED !!! */
  return fclose(f);
}


and *EVERY* single call to writeUidCache MUST read the returned value, 
and act correctly if it's non zero.

I'll now let the kmail coders deal with what "act correctly" means, the 
code is not localized enough so that I can provide a useful patch.
Comment 78 Martin Steigerwald 2006-06-29 10:06:13 UTC
Hello,

Hmmmm, this is getting more and more complex. I think we are dealing is lots of different bugs here. At least those:

1) Some operations do not return a success or failure indication
2) On some operations the success or failure indication is not checked
3) KMail seems to trust its index file more than it should. If KMail also deletes locally stored mails that are not in the index file the implications are far beyond IMAP related mail losses!

All of this indicate that KMail desperately needs application-internal data journalling and atomicity. What are about mails that have been downloaded and not added to the index? What are about mails that have been marked as downloaded, added to the index, but not saved (due to an error)?

So KMail needs:

1) Journalled index file writing: KMail needs to be absolutely sure, that the index file represents the actual folder contents. So it has to write any changes to a folder to a journal before executing them. So it can check whether all changes have been executed and update the index file accordingly in case it has been stopped abruptly. 

For this to work reliably it needs to be able to sync write the journal, i.e. tell filesystem and block layer to write it immediately and only return when it really has been written to disk (and not only sitting in a write cache). (Not unlike write barrier functionately in kernel 2.6.16+).

This way kmail can tell whether a mail has been written to disk or not, even if it has been aborted abruptly. On next start it will go through the journal if it is not empty. It will add all already commited transactions to the index file. Also it will complete all not yet completed transactions if possible. In case of a downloaded e-mail that was about to be written to disk, but has not been written (completely) it should tell the user to synchronize the IMAP account again in order to re-download that mail - or to redownload mails from a POP3 acocunt.

Or it needs a way to rebuild the index file of a folder if its last access time is newer than the date of the index file. I know that the Amiga mail program YAM (http://www.yam.ch - GPL) rebuilds the index file of a folder in that case. This already is quite reliabled, but is not complete: What if a mail has been added to the index file but not yet been flushed to disk? So preferably journalled index file writing is used! Or each mail has to be sync-written, which probably would slow things down more than journalled index file writing.

2) Complete error handling during the IMAP sync process including reporting to the user. KMail should know whether an IMAP action was successful or not. It should add a new mail to its local mail storage only if downloading it via IMAP has been successful!

Actually I believe this needs some design work before implementing code. It should be documented how mail and index file saving as well as IMAP or POP3 should be done and how errors should be handled.

It would probably be best to rewrite mail syncing, mail download and mail and index file code from scratch. This is a lot of work which likely cannot be done before KDE 4.

For KDE 3.5.x fixing at least the most important cases of non complete error handling might help dramatically as well.

I am willing to help with design work and documentation. I currently don't have enough C++ skills to code the stuff.

Regards, Martin
Comment 79 Andreas Gungl 2006-06-29 10:30:18 UTC
Am Donnerstag, 29. Juni 2006 10:06 schrieb Martin Steigerwald:
> It would probably be best to rewrite mail syncing, mail download and mail
> and index file code from scratch. This is a lot of work which likely
> cannot be done before KDE 4.
> [...]
> I am willing to help with design work and documentation. I currently
> don't have enough C++ skills to code the stuff.


And this is the main problem. KMail doesn't need countless proposals for 
ways to solve the problems, it needs people which solve the already known 
problems.
Comment 80 Martin Steigerwald 2006-06-29 11:19:27 UTC
Well, Andreas, I offered to help where I can - I think my idea of journalling in KMail can help to fix the issue at its cause. Pierre Habouzit recently offered some code analysis and fixes. Rob Kaper recently brought the index file issue on the table. Others contributed insights and reports. Thats a good start.

My hope is that in the end when everyone contributes his or her skills it all comes together: Analysis, design and documentation of a fix as well as its implementation.

Yes, no one has yet fixed the issue and someone is needed to implement, say code, a fix, Andreas. We all know that, don't we? I ask you kindly to concentrate on constructively contributing to solving the bug as your skills allow, instead of stating what seems rather obvious to me. Its broken, only acceptance of this can help to free energy needed to fix it.

I am also offering to test any fixes as time and resources permit.
Comment 81 Gerco Dries 2006-06-29 11:33:27 UTC
I might be way of of my leage here, but all this talk of journalling the uidcache file and sync-writing to disk seems a bit far fetched for a mail client to implement. Could the local cache perhaps be re-implemented using some transactional mechanism (like an embedded, lightweight database, sqlite perhaps) instead of coding actual journalling into KMail?

I believe using an embedded database might actually be less work than implementing actual journalling logic in KMail.
Comment 82 Adorjáni Gábor 2006-07-02 11:22:32 UTC
I've just run into the same bug here, on Kubuntu 5.10, with KDE version upgraded some months ago to 3.5.2. KMail today morning lost ~200 mails from my personal disconnected IMAP folder, the server on the other end is a Cyrus IMAPD 2.2.12 running on Debian Sarge. However, I discovered that the messages _DO_ exist in the filesystem, currently I'm backing up my ~/.kde/share/apps/kmail folder and will try to rebuild the message index.

Switching back to Thunderbird for my IMAP needs, though...
Comment 83 Martin Steigerwald 2006-07-02 13:49:25 UTC
Can you try what happens when you quit KMail, delete the index file of the folder (is it inbox in your case?) and restart KMail? I think KMail should try to rebuild the index file then. Maybe one of the different functions to solve problems with the IMAP cache might help also.

Index files are named are named like this.

martin@deepdance:~/Mail -> ls -la .inbox*
-rw------- 1 martin martin 149605 2006-07-01 21:49 .inbox.index
-rw-r--r-- 1 martin martin   1485 2006-07-01 22:54 .inbox.index.ids
-rw-r--r-- 1 martin martin  18149 2006-06-30 22:27 .inbox.index.sorted

But this example is for a POP3 account. In your case the mails and index files should be under ~/.kde/share/apps/kmail/imap or ~/.kde/share/apps/kmail/dimap (do not know which one without looking on my work account, I think it is dimap tough). There should be one directory for every DIMAP account.

If you get your mails back through this, this clearly points at the issue that KMail trusts the index file more than it should.

Make a backup first tough.
Comment 84 Adorjáni Gábor 2006-07-02 23:18:32 UTC
Martin, thanks for the tip. I tried it before reading your post, but I wouldn't call it absolute success. However, I managed to bring back my mails.

I deleted the index files, fired up Kmail, but refused to give neither Kwallet's master password, nor the IMAP account's password. It rebuilt the index, all of my old mail was there, as well as the 8 new messages from a previous session. I could make a copy about the whole folder - just for testing purposes, of course I made a complete backup before. As soon as I reconnected to the IMAP server, Kmail synchronised the local folder to the remote mailbox and deleted all of the old mails again. :( I ended up with the 8 new messages I got this morning.

I hope this description help a bit. :)
Comment 85 Chris Peikert 2006-07-07 03:09:51 UTC
I have been bit by this bug, twice now.  As in #84, I noticed the symptoms, rebuilt my index files, and all appeared to be well -- all my mail in the affected folder was visible.

Hours later, I returned to another machine (also running kmail with dimap), and nearly all of the emails from the affected folder were gone.  (Even more were missing than before.)  This time I was not able to recover -- the emails were gone from all my local caches, and from my webmail IMAP account.

This is extremely annoying, as I *must* rely on dIMAP in order to share my groupware info (contacts, to-dos, etc.) between laptop and desktop systems!!  (Someone please correct me if this has changed.)
Comment 86 Kai Krakow 2006-08-01 19:02:11 UTC
I'm loosing mails too, sometimes. It's usually not any current mail that vanishes but mails which are some days older. Later when I search for a mail which I know must be there, I cannot find it. Instead I sometimes see completely empty mails with correct date but no subject, no sender, no receipient, not any header and no body. Just like a zero byte file. But it shows a date in the mail list view. I suppose these are orphans of the mails which have been gone.

Most of the time I'm experiencing this is while KMail is busy doing stuff (like syncing folders or moving/expiring mails) and then crashes. Sometimes I'm presented with a dialog about differing local and server versions and which one I want to use. But sometimes mails just vanish.

I'm using dimap and kontact, sometimes kmail. But it makes no difference if using kontact or kmail standalone. When I used Evolution inbetween with exchange-connector, kmail almost always redownloads _all_ my mails. Is this intended? Maybe this has to do with the problem.

KMail Version 1.9.3
Kontact Version 1.2.3
OpenSUSE 10.1
MS Exchange 2003 IMAP Server (yeah ugly I know, and SLOW with IMAP)

Apart from this bug, KMail is my favorite and in my eyes best mail client ever made. It beats Evolution by far in terms of functionality, usuability and stability (at least when connected to Exchange). Speed could be a little better but it at least doesn't die like Evolution does if it tends to become slow by processing large amounts of data.
Comment 87 Till Adam 2006-08-02 10:29:47 UTC
Pierre, thanks for a constructive contribution. Your analysis is largely correct, and I'll see what I can do to incorporate your suggestions. Go easy on the attitude, though, dude. ;)
Comment 88 m.wege 2006-08-02 10:58:24 UTC
I have found one nearly reproduceble pattern, which seems to cause the mail loss, but I believe there are more. I have noticed that when new mails are sorted in with the filtering of Inbox and then the mail check continues to check the other folders, some of the mails which have been filtered into the folders disappear again. Not all, but some. So may be this is a first hint to find the problem.
Comment 89 Till Adam 2006-08-07 09:20:49 UTC
I've attached a patch which adds some file handling safety for the uidcache file and some more aggressive debugging. I'd appreciate it if anyone who sees this bug could apply the patch and report debug output and sudden exits. Make sure you compile KMail with full debugging and start it from a konsole, to see the debug output. Thanks.
Comment 90 Till Adam 2006-08-07 09:22:34 UTC
Created attachment 17253 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 91 Till Adam 2006-08-07 09:22:57 UTC
Created attachment 17254 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 92 Till Adam 2006-08-07 09:23:22 UTC
Created attachment 17255 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 93 Till Adam 2006-08-07 09:24:17 UTC
Created attachment 17256 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 94 Till Adam 2006-08-07 09:24:59 UTC
Created attachment 17257 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 95 Till Adam 2006-08-07 09:25:52 UTC
Created attachment 17258 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 96 Till Adam 2006-08-07 09:29:54 UTC
Created attachment 17259 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 97 Till Adam 2006-08-07 09:34:00 UTC
Created attachment 17260 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 98 Pierre Habouzit 2006-08-07 09:45:56 UTC
Will do when I come back home on tonight. And I'll try the couple of torture tests I've elaborated against the previous versions of kmail.

thanks for the work btw !
Comment 99 Till Adam 2006-08-07 09:52:16 UTC
Created attachment 17261 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 100 Till Adam 2006-08-07 10:26:46 UTC
Created attachment 17262 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 101 Till Adam 2006-08-07 10:36:29 UTC
Created attachment 17263 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 102 Rob Kaper 2006-08-07 11:06:19 UTC
I think there's also a bug in bugs.kde.org, where Till sees an "upload failed" message despite a successful upload..
Comment 103 Till Adam 2006-08-07 11:07:21 UTC
Created attachment 17264 [details]
patch against 3.5 with file handling safety and debug helpers
Comment 104 Till Adam 2006-08-07 11:10:13 UTC
Timeouts, actually, and the old page delivered from some cache or other. My apologies for spamming.
Comment 105 Arne Schmitz 2006-08-07 11:15:27 UTC
Doesn't matter Till. You not only help us stop losing mail, but also generate tons of new mails for us! :)
Thanks for the effort of creating the patch!
Comment 106 Pierre Habouzit 2006-08-12 18:17:10 UTC
>   Will do when I come back home on tonight. And I'll try
> the couple of torture tests I've elaborated against the previous
> versions of kmail.
>
> thanks for the work btw !


sadly, due to a lot of debian work, and me beeing in vacation for the 
coming week, I don't think I will have enough time to check it 
immediately. I will definitely will send my remarks before end of the 
month.
Comment 107 Till Adam 2006-08-14 12:12:22 UTC
I've just committed revision 572899 to 3.5 branch which deals better with various situations related to the slave dying or disconnecting during flags upload. These changes might impact other scenarios where slave errors lead to sync state resets, such as power management resume or wlan connection flakyness. I would appreciate any and all testing especially from those who can reproduce the problem somewhat reliably. I'm also uploading a new ultra-verbose-debugging patch which applies cleanly against 3.5 branch and contains yet more debugging info, in case this does not fix it yet.
Comment 108 Till Adam 2006-08-14 12:15:25 UTC
Created attachment 17367 [details]
Updated debugging patch
Comment 109 Kimmo 2006-08-22 07:03:03 UTC
Hi,

this might by a "heretical" suggestion, but maybe coders should take a look on how Evolution or/and Thunderbird handles this syncronisation issue in their source-code?? Immediately after my first mail loss I switched to Evo. Since then, I have encountered no problems at all with my IMAP-folders. Maybe this kind of an approach could help us to solve this problem...
Comment 110 Arne Schmitz 2006-08-30 19:14:32 UTC
Does Evolution do Disconnected IMAP?

Arne
Comment 111 Kimmo 2006-09-01 17:44:16 UTC
AFAIK, Evo does not support dIMAP, but my mail losses have occurred with normal IMAP accounts, not with dIMAP. Thus, could this bug be merely related to how Kmail handles IMAP in general, not just dIMAP...

Kimmo
Comment 112 Tom 2006-09-03 00:40:00 UTC
I have also had mail losses with a normal IMAP account.  Most recently a FC5 machine with the rpm kdepim.i386 6:3.5.2-0.1.fc5 talking across a 802.11b link to an FC3 machine with imap-2004-0.fdr.8.a.3.  Kmail was set to a 1 minute check and the wifi had backups run across it so it was sometimes quite busy.  The second time I lost mail I moved to Thunderbird.  Loss was (fortunately!) rare happening twice over a period of perhaps 6 months to a year.

Tom
Comment 113 Martin Steigerwald 2006-09-03 10:31:11 UTC
Did anyone who can reproduce this bug test the patch that Till Adam kindly provided? I do not see the point in adding further "I have this too" reports as long as no one tests whether the patch fixes those problems.

I did not yet experience this bug. I use POP3 at home and didn't find (!) any mail losses with dIMAP using an NFS mounted home directory at work.

There are instructions for compiling KDE available:
http://developer.kde.org/build/old/compile_kde3_5.html

One would have to build arts and kdelibs, and then can compile kdepim as I understand the instructions. Of course one should apply the patch from Till Adam before. ;)

According to this entry it should easily be possible to compile and install KDE or parts of it into a directory in the user's home directory to avoid messing up and existing KDE installation:

http://docs.kde.org/development/en/kdebase/faq/install.html#id2548646

I have lots of open TODOs, but if no one else volunteers I may look into providing a compiled KMail version with this patch applied after mid of september. But then this wouldn't probably work everywhere anyway as there is a mixture of different GCC compiler versions and shared libraries floating around in different linux installations.

So best is when someone who can actually reproduce this problem tries to compile KMail 3.5.4 with the patch applied and test it. I am pretty sure there are people who can help with compiling questions!
Comment 114 Fathi Boudra 2006-09-04 18:35:05 UTC
following Martin Steigerwald comment, Debian package was updated with necessary patches since 26/08/2006.
Comment 115 Adam Porter 2006-09-04 22:56:30 UTC
I love Debian.  =D
Comment 116 Chris Peikert 2006-09-13 07:48:23 UTC
I have been running the Debian version with the patch for the past few weeks.  Here is my experience...

I have three machines set up to access the same IMAP account, all using dIMAP.  Often I will move mail to trash, delete mail, etc. on machine A, then refresh my view on machines B and C some time later.  When refreshing B, I routinely get several dialogs stating "messages so-and-so were deleted in folder such-and-such, do you want to delete them locally?" followed by a list of message UIDs.  I typically get one dialog per folder that has changed.  I also have a separate spam folder, in which old messages are automatically purged every once in awhile; I get dialogs for this folder too whenever some messages have been purged.

The dialogs are slightly annoying and seem to me to be not strictly necessary (aren't we worried about the *client* telling the *server* to delete mail incorrectly, based on a corrupt index?), but they are a small price to pay on the path to fixing this bug.  So far I have not had any mail loss, even with a few situations in which a flaky wireless connection has died in the middle of a refresh.  (Previously, such situations were a high risk for mail loss in my experience.)

This evidence is anecdotal, as I've never found a reliable means of inducing mail loss.  However, I think it is encouraging that my mail has survived through a couple of scenarios that may have caused loss in prior versions.
Comment 117 Carsten Pfeiffer 2006-09-13 09:23:12 UTC
Also running the Debian version, I often get the same dialog "... do you want to delete them locally". One way to reproduce this is dragging a mail from one imap folder to another. kmail will show the moved mail just fine, but as soon as the destination folder is checked again (e.g. via F5), you get that irritating dialog.
Comment 118 Till Adam 2006-09-13 09:25:55 UTC
Thanks for the feedback, Chris. The purpose of those dialogs is twofold. Firstly they allow you to detect the mail loss situation, since a deletion on the server that comes as a surprise to you (is not one of the scenarios you describe seeing the dialog in atm) would likely be such a case. You could then, while the dialog is up, copy the debug output (~/.xsession-errors, or wherever your system puts it) up to that point into a mail to me, providing me with valuable information about what preceded this. Detecting unwanted deletions coming from the client during sync is much harder, otherwise I would have been able to fix the bug long ago. Secondly it allows you to copy away those mails into a backup folder, having declined the offer to delete them locally, in case the deletion is indeed due to a mail loss scenario. That way you at least avoid the mail loss as such.
Comment 119 Till Adam 2006-09-13 09:27:54 UTC
Guys, this patch is a debugging tool, as are those dialogs. They are not meant to be in a production build. The debian folks obviously decided to make all their users debug helpers for this bug, which is nice, but maybe not intended.
Comment 120 Philippe Cloutier 2006-09-13 09:34:05 UTC
s/their users/unstable users
Comment 121 Pierre Habouzit 2006-09-13 10:54:48 UTC
It is totally intended, because kmail cannot migrate to testing due to the dimap bug, and only pure unstable users have those dialogs. if they don't want to, they have to downgrade to testing where the debug is not turned on, full stop.

If dimap bug is not solved one way or another, kmail won't have dimap support on (using very crude patches) in etch.
Comment 122 Chris Peikert 2006-09-13 18:18:43 UTC
Till and all,

thanks for the clarifications about the dialogs.  I didn't mean at all to complain about them, if it came out sounding that way -- I just didn't know what purpose they served.

I'd like to test this patch in a more controlled setting (trying to induce mail loss), but it may take some time for me to set that up properly.
Comment 123 Adam Porter 2006-09-19 08:41:22 UTC
Regarding the dialogs, they seem to be coming up with a lot of false positives.  I'm getting the warnings after simply moving some messages from my inbox to a subfolder; it warns me that messages on the *server* in the *target* folder were deleted.  After saying Yes, and then resyncing with the server, the messages reappear in my local store.  Strangely, it seems to happen with messages that existed in the target folder before I moved the newer messages to it.

Personally, and thankfully, I've never lost any mail with KMail (yet).  I'm glad to see this effort to track down the problem, and I'll be glad to help if I can.
Comment 124 m.wege 2006-09-20 09:23:16 UTC
I know also run the patch. I get a lot of messages about deleted mails on the server and if I want to delete them locally in folders where I definitely have not deleted any mails. But I have read mails there and so the status has changed. I do not know if this is due to false dialogs or false interpretation through kmail, but if kmail misinterpreted mails whose status has changed as deleted on the server this could be one reason for mail loss.
Yesterday (before upgrading) I had also another rather severe mail loss in my inbox, which occured while I was running thunderbird parallel with non-cached imap on another machine and then (my laptop had just returned from repairs) started kmail with cached Imap. After syncing several Email not disappeared but turned into "unknown subject/sender" and lost their content in my kmail. Lucally this seemed to be only locally even though I just had synced and I stopped Kmail while it was writing again to sever, so I only lost a few mails. In general I newer loose mails by just disappearing. It is always that Mails loose content/subject/sender but remain in the inbox.
Comment 125 Christopher Martin 2006-09-27 04:08:39 UTC
Hmmm, so no one can more-or-less confirm that the patch fixes the d-imap data loss? It looks like KDE 3.5.5 will still contain this problem, which is a shame, though perhaps someone will come through with a definitive yay/nay at the last minute.

(Debian Etch will probably ship with KDE 3.5.4 or 3.5.5 - certainly nothing later. Looks like we'll have to disable d-imap, which is a shame.)
Comment 126 Larry Garfield 2006-09-27 04:35:37 UTC
To my knowledge, I've never had data loss as a result of this bug.  The first time I noticed it was when Debian Sid (my distro of choice) added the debugging code, which of course has now basically hosed my mail setup. :-(  (I leave KMail running and use client-side filters over DIMAP to sort my 400 messages a day.  The warning dialog prevents it from downloading any more mail, so it doesn't filter anything.)

The pattern for me seems to be any time there's a difference between my local copy and the copy on the server.  99.9% of the time, that's because something was just filtered into it.  Eg, starting from fully synched, KMail checks for new messages and finds 15, 10 of which it filters into the Inbox/lists/list1 folder.  The next time it checks for new mail, it sees that Inbox/lists/list1 has mail that the server doesn't, and whines at me that something was deleted on the server (false), in case I want to delete it locally.

I don't know what other information I can provide to help track this down.  If there is any, let me know and I'll see what I can dig up, because this has rendered my mail setup nigh on unusable.  (Not the bug, the debugging code.  I want that gone!)
Comment 127 David Lichterman 2006-09-27 04:47:11 UTC
I only had the problem when I ran kmail with spamassassin on gentoo. I
stopped using it (kmail) since I couldn't afford to lose my mail. I hope
this helps.
-Dave

On 27 Sep 2006 02:35:52 -0000, Larry Garfield <larry@garfieldtech.com>
wrote:
[bugs.kde.org quoted mail]
I only had the problem when I ran kmail with spamassassin on gentoo. I stopped using it (kmail) since I couldn't afford to lose my mail. I hope this helps.<br>-Dave<br><br><div><span class="gmail_quote">On 27 Sep 2006 02:35:52 -0000, 
<b class="gmail_sendername">Larry Garfield</b> &lt;<a href="mailto:larry@garfieldtech.com">larry@garfieldtech.com</a>&gt; wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
------- You are receiving this mail because: -------<br>You are a voter for the bug, or are watching someone who is.<br><br><a href="http://bugs.kde.org/show_bug.cgi?id=104956">http://bugs.kde.org/show_bug.cgi?id=104956</a>
<br><br><br><br><br>------- Additional Comments From larry garfieldtech com&nbsp;&nbsp;2006-09-27 04:35 -------<br>To my knowledge, I've never had data loss as a result of this bug.&nbsp;&nbsp;The first time I noticed it was when Debian Sid (my distro of choice) added the debugging code, which of course has now basically hosed my mail setup. :-(&nbsp;&nbsp;(I leave KMail running and use client-side filters over DIMAP to sort my 400 messages a day.&nbsp;&nbsp;The warning dialog prevents it from downloading any more mail, so it doesn't filter anything.)
<br><br>The pattern for me seems to be any time there's a difference between my local copy and the copy on the server.&nbsp;&nbsp;99.9% of the time, that's because something was just filtered into it.&nbsp;&nbsp;Eg, starting from fully synched, KMail checks for new messages and finds 15, 10 of which it filters into the Inbox/lists/list1 folder.&nbsp;&nbsp;The next time it checks for new mail, it sees that Inbox/lists/list1 has mail that the server doesn't, and whines at me that something was deleted on the server (false), in case I want to delete it locally.
<br><br>I don't know what other information I can provide to help track this down.&nbsp;&nbsp;If there is any, let me know and I'll see what I can dig up, because this has rendered my mail setup nigh on unusable.&nbsp;&nbsp;(Not the bug, the debugging code.&nbsp;&nbsp;I want that gone!)
<br></blockquote></div><br>
Comment 128 Adam Porter 2006-09-27 08:04:01 UTC
Larry,

Thanks, I think you have indeed figured out why that dialog box keeps popping up for no apparent reason.  I also use client-side filters in KMail, and it usually complains about folders that the filters use.  Although it also "complains" about the Sent folder every time I send an e-mail...

You should be able to remove the Debian patch that adds the debug dialog and rebuild KMail quite easily.  The bad part would be waiting for all of kdepim to compile, since you can't compile *just* KMail, AFAIK.  But you can nice down the build process and let it run in the background or overnight.
Comment 129 Larry Garfield 2006-09-27 08:47:23 UTC
I have the same issue with the Sent folder, but I'm not sure if it happens on every message or not.  I haven't been able to confirm that yet.  And I actively try to avoid compiling things myself if I can avoid it.  That's why I use Debian in the first place. :-)  I'm not even sure how I'd selectively recompile a Deb-Src package (although that's OT for this thread).

Hopefully something in this rambling will help someone fix this issue. :-)
Comment 130 Tom Albers 2006-09-27 09:50:33 UTC
Since Till fixed KMail I have not seen a single report of mail loss. I think this case can be closed now. Any objections?
Comment 131 m.wege 2006-09-27 10:32:22 UTC
Yes. I have. But I am not 100% sure. That since this patch asking me if I want to delete a mail on the server which was deleted locally, I get asked if I want to delete that on the server too. The problem these mails have never been deleted locally. They just have changed status. I am not sure if this mechanism is implemented wrong or if kmail is really trying to delete them. So I always answer no. Would be nice to hear something about this and have the mechanism corrected. 
Comment 132 Tom Albers 2006-09-27 10:44:03 UTC
Set up a temp folder, drag some trash mails to there, sync, change there state, sync, and answer yes to the question and see what happens and report it here. Then you know it..... ;-) 
Comment 133 m.wege 2006-09-28 00:20:42 UTC
Hi Tom, nothing happend. But something else. I think I know how at least part of the mail los happens. I am using filters. Kmail fetches mail from INBOX, moves a mail to a folder. The problem could be that I have really a lot of folders with a lot of mails so it takes a long time for kmail to finish a mail check. I do not know in what way kmail notifies a server that it has moved a mail into another folder, but I think could have something todo with a kind of wrong timing of that. Now with the patch and kmail asking "mail has been deleted on the server, do you want to delete it locally" means that at this point where Kmail wants to syncronize the move has not happened on the server yet. So has to be something with bad timing, something like, I have to shut down my computer because I have to leave or I loose internet connection so that synchronising is not finished. I guess that kmail still has a mechanisms to prevent this, but may it does not work allways. for example with large mailboxes with a lot of folders.
Comment 134 Pierre Habouzit 2006-10-02 16:16:57 UTC
I've been unable to make kmail do a faulty sync. The design remains brittle, but at least, now, when there is a doubt, kmail barfs to the user, and do what's sensible: not trusting the .uidcache anymore.

I think most of the issue is if not solved, at least avoided right now.
Comment 135 Till Adam 2006-10-21 11:34:12 UTC
SVN commit 597659 by tilladam:

Add more uidcache file handling defensiveness and some debugging facilities.
Patch tested by me for a few weeks, initial reactions from the testers seem
to say it helps.
BUG: 104956


 M  +81 -19    kmfoldercachedimap.cpp  
 M  +5 -0      kmfoldercachedimap.h  


--- branches/KDE/3.5/kdepim/kmail/kmfoldercachedimap.cpp #597658:597659
@@ -75,6 +75,7 @@
 #include <globalsettings.h>
 
 #define UIDCACHE_VERSION 1
+#define MAIL_LOSS_DEBUGGING 0
 
 static QString incidencesForToString( KMFolderCachedImap::IncidencesFor r ) {
   switch (r) {
@@ -150,10 +151,22 @@
     mFolderRemoved( false ),
     /*mHoldSyncs( false ),*/ mRecurse( true ),
     mStatusChangedLocally( false ), mAnnotationFolderTypeChanged( false ),
-    mIncidencesForChanged( false ), mPersonalNamespacesCheckDone( true )
+    mIncidencesForChanged( false ), mPersonalNamespacesCheckDone( true ),
+    mFoundAnIMAPDigest( false )
 {
   setUidValidity("");
-  readUidCache();
+  // if we fail to read a uid file but there is one, nuke it
+  if ( readUidCache() == -1 ) {
+    if ( QFile::exists( uidCacheLocation() ) ) {
+        KMessageBox::error( 0,
+        i18n( "The UID cache file for folder %1 could not be read. There "
+              "could be a problem with file system permission, or it is corrupted."
+              ).arg( folder->prettyURL() ) );
+        // try to unlink it, in case it was corruped. If it couldn't be read 
+        // because of permissions, this will fail, which is fine
+        unlink( QFile::encodeName( uidCacheLocation() ) );
+    }
+  }
 
   mProgress = 0;
 }
@@ -306,7 +319,7 @@
   if( uidValidity().isEmpty() || uidValidity() == "INVALID" ) {
     // No info from the server yet, remove the file.
     if( QFile::exists( uidCacheLocation() ) )
-      unlink( QFile::encodeName( uidCacheLocation() ) );
+      return unlink( QFile::encodeName( uidCacheLocation() ) );
     return 0;
   }
 
@@ -317,17 +330,23 @@
     str << uidValidity() << endl;
     str << lastUid() << endl;
     uidcache.flush();
-    fsync( uidcache.handle() ); /* this is probably overkill */
-    uidcache.close();
-    return 0;
-  } else {
-    return errno; /* does QFile set errno? */
+    if ( uidcache.status() == IO_Ok ) {
+      fsync( uidcache.handle() ); /* this is probably overkill */
+      uidcache.close();
+      if ( uidcache.status() == IO_Ok )
+        return 0;
+    }
   }
+  KMessageBox::error( 0,
+        i18n( "The UID cache file for folder %1 could not be written. There "
+              "could be a problem with file system permission." ).arg( folder()->prettyURL() ) );
+
+  return -1;
 }
 
 void KMFolderCachedImap::reloadUidMap()
 {
-  kdDebug(5006) << "Reloading Uid Map " << endl;
+  //kdDebug(5006) << "Reloading Uid Map " << endl;
   uidMap.clear();
   open();
   for( int i = 0; i < count(); ++i ) {
@@ -448,7 +467,8 @@
 {
   killTimer( uidWriteTimer );
   uidWriteTimer = -1;
-  writeUidCache();
+  if ( writeUidCache() == -1 )
+    unlink( QFile::encodeName( uidCacheLocation() ) );
 }
 
 ulong KMFolderCachedImap::lastUid()
@@ -467,10 +487,22 @@
   QMap<ulong,int>::Iterator it = uidMap.find( uid );
   if( it != uidMap.end() ) {
     KMMsgBase *msg = getMsgBase( *it );
+#ifdef MAIL_LOSS_DEBUGGING
+    kdDebug(5006) << "UID " << uid << " is supposed to be in the map" << endl;
+    kdDebug(5006) << "UID's index is to be " << *it << endl;
+    kdDebug(5006) << "There is a message there? " << (msg != 0) << endl;
+    if ( msg ) {
+      kdDebug(5006) << "Its UID is: " << msg->UID() << endl;
+    }
+#endif
+
     if( msg && msg->UID() == uid )
       return msg;
+    kdDebug(5006) << "########## Didn't find uid: " << uid << "in cache athough it's supposed to be there!" << endl;
   } else {
+#ifdef MAIL_LOSS_DEBUGGING
     kdDebug(5006) << "Didn't find uid: " << uid << "in cache!" << endl;
+#endif
   }
   // Not found by now
  // if( mapReloaded )
@@ -482,8 +514,10 @@
   if( it != uidMap.end() )
     // Since the uid map is just rebuilt, no need for the sanity check
     return getMsgBase( *it );
+#ifdef MAIL_LOSS_DEBUGGING
   else
     kdDebug(5006) << "Reloaded, but stil didn't find uid: " << uid << endl;
+#endif
   // Then it's not here
   return 0;
 }
@@ -841,9 +875,14 @@
            to be deleted on the server has been deleted, adjust our local notion of the
            highes uid seen thus far. */
         slotUpdateLastUid();
-        if( mLastUid == 0 && uidWriteTimer == -1 )
+        if( mLastUid == 0 && uidWriteTimer == -1 ) {
           // This is probably a new and empty folder. Write the UID cache
-          writeUidCache();
+          if ( writeUidCache() == -1 ) {
+            resetSyncState();
+            emit folderComplete( this, false );
+            return;
+          }
+        }
       }
     }
 
@@ -1209,9 +1248,10 @@
 void KMFolderCachedImap::slotImapStatusChanged(KMFolder* folder, const QString&, bool cont)
 {
   if ( mSyncState == SYNC_STATE_INITIAL ){
-      kdDebug(5006) << "IMAP status changed but reset " << endl;
+      //kdDebug(5006) << "IMAP status changed but reset " << endl;
       return; // we were reset
   }
+  //kdDebug(5006) << "IMAP status changed for folder: " << folder->prettyURL() << endl;
   if ( folder->storage() == this ) {
     --mStatusFlagsJobs;
     if ( mStatusFlagsJobs == 0 || !cont ) // done or aborting
@@ -1220,6 +1260,7 @@
     if ( mStatusFlagsJobs == 0 && cont ) {
       mProgress += 5;
       serverSyncInternal();
+      //kdDebug(5006) << "Proceeding with mailcheck." << endl;
     }
   }
 }
@@ -1288,15 +1329,24 @@
   // them one by one because the index list can get resized under
   // us. So use msg pointers instead
 
+  QStringList uids;
   QMap<ulong,int>::const_iterator it = uidMap.constBegin();
   for( ; it != uidMap.end(); it++ ) {
     ulong uid ( it.key() );
-    if( uid!=0 && !uidsOnServer.find( uid ) )
+    if( uid!=0 && !uidsOnServer.find( uid ) ) {
+      uids << QString::number( uid );
       msgsForDeletion.append( getMsg( *it ) );
+    }
   }
 
   if( !msgsForDeletion.isEmpty() ) {
-    removeMsg( msgsForDeletion );
+#ifdef MAIL_LOSS_DEBUGGING
+      if ( KMessageBox::warningYesNo(
+             0, i18n( "<qt><p>Mails on the server in folder <b>%1</b> were deleted. "
+                 "Do you want to delete them locally?<br>UIDs: %2</p></qt>" )
+             .arg( folder()->prettyURL() ).arg( uids.join(",") ) ) == KMessageBox::Yes )
+#endif
+        removeMsg( msgsForDeletion );
   }
 
   /* Delete messages from the server that we dont have anymore */
@@ -1370,6 +1420,8 @@
   uidsForDeletionOnServer.clear();
   mMsgsForDownload.clear();
   mUidsForDownload.clear();
+  // listing is only considered successful if saw a syntactically correct imapdigest
+  mFoundAnIMAPDigest = false;
 
   CachedImapJob* job = new CachedImapJob( FolderJob::tListMessages, this );
   connect( job, SIGNAL( result(KMail::FolderJob *) ),
@@ -1415,6 +1467,7 @@
       setReadOnly( access == "Read only" );
     }
     (*it).cdata.remove(0, pos);
+    mFoundAnIMAPDigest = true;
   }
   pos = (*it).cdata.find("\r\n--IMAPDIGEST", 1);
   // Start with something largish when rebuilding the cache
@@ -1432,7 +1485,7 @@
       if( uid != 0 ) {
         if ( uidsOnServer.count() == uidsOnServer.size() ) {
           uidsOnServer.resize( KMail::nextPrime( uidsOnServer.size() * 2 ) );
-          kdDebug( 5006 ) << "Resizing to: " << uidsOnServer.size() << endl;
+          //kdDebug( 5006 ) << "Resizing to: " << uidsOnServer.size() << endl;
         }
         uidsOnServer.insert( uid, &v );
       }
@@ -1451,7 +1504,9 @@
         KMMsgBase *existingMessage = findByUID(uid);
         if( !existingMessage ) {
           if ( mUserRights <= 0 || ( mUserRights & KMail::ACLJobs::Delete ) ) {
-            // kdDebug(5006) << "message with uid " << uid << " is gone from local cache. Must be deleted on server!!!" << endl;
+#ifdef MAIL_LOSS_DEBUGGING
+            kdDebug(5006) << "message with uid " << uid << " is gone from local cache. Must be deleted on server!!!" << endl;
+#endif
             uidsForDeletionOnServer << uid;
           } else {
             redownload = true;
@@ -1490,6 +1545,13 @@
 void KMFolderCachedImap::getMessagesResult( KMail::FolderJob *job, bool lastSet )
 {
   mProgress += 10;
+  if ( !job->error() && !mFoundAnIMAPDigest ) {
+      kdWarning(5006) << "######## Folderlisting did not complete, but there was no error! "
+          "Aborting sync of folder: " << folder()->prettyURL() << endl;
+#ifdef MAIL_LOSS_DEBUGGING
+      kmkernel->emergencyExit( i18n("Folder listing failed in interesting ways." ) );
+#endif
+  }
   if( job->error() ) { // error listing messages but the user chose to continue
     mContentState = imapNoInformation;
     mSyncState = SYNC_STATE_HANDLE_INBOX; // be sure not to continue in this folder
@@ -1741,7 +1803,7 @@
   KMFolderNode *node;
   bool root = ( this == mAccount->rootFolder() );
   if ( root && !mAccount->hasInbox() ) {
-    kdDebug(5006) << "check INBOX" << endl;
+    //kdDebug(5006) << "check INBOX" << endl;
     // create the INBOX
     for (node = folder()->child()->first(); node; node = folder()->child()->next())
       if (!node->isDir() && node->name() == "INBOX") break;
@@ -2216,7 +2278,7 @@
 void
 KMFolderCachedImap::slotAnnotationChanged( const QString& entry, const QString& attribute, const QString& value )
 {
-  kdDebug(5006) << k_funcinfo << entry << " " << attribute << " " << value << endl;
+  //kdDebug(5006) << k_funcinfo << entry << " " << attribute << " " << value << endl;
   if ( entry == KOLAB_FOLDERTYPE )
     mAnnotationFolderTypeChanged = false;
   else if ( entry == KOLAB_INCIDENCESFOR ) {
--- branches/KDE/3.5/kdepim/kmail/kmfoldercachedimap.h #597658:597659
@@ -445,6 +445,11 @@
       mLastUid. See above for details. */
   ulong mTentativeHighestUid;
 
+  /** Used to determine whether listing messages yielded a sensible result.
+   * Only then is the deletion o messages (which relies on succesful
+   * listing) attempted, during the sync.  */
+  bool mFoundAnIMAPDigest;
+
   int mUserRights;
   ACLList mACLList;
 
Comment 136 Till Adam 2006-10-21 11:35:53 UTC
As you can see above the patch (with verbose message boxes turned off) is now in 3.5 branch. Any confirmation that this fixes the mail loss would be much appreciated.
Comment 137 Chris Peikert 2006-10-31 20:28:59 UTC
I think the patch is working!  I have recently encountered several broken-connection situations that sometimes led to mail loss in the past, yet I haven't lost any mail with the new version.

Kudos and many thanks!
Comment 138 Calvin Priest 2006-11-01 20:23:19 UTC
First of all, thanks to the devs for a wonderful application.  Kontact is truly elegant.  I love it.

However, as a person who was just recently bitten by this bug/design issue (I didn't have the patch), I would vote for keeping this bug open until the patch has been tested for several months by a lot of people. This is a VERY VERY serious issue which threatens the reputation of Kontact/Kmail.  I have used a lot of email clients over the years, and have never lost thousands of messages the way I did yesterday.  It leaves a very bad taste in the mouth.  This sort of thing does not go over well in corporate environments, in particular.  It is the sort of thing which is disturbing enough to see in a Beta -- it simply should not exist in a stable tree.

Please don't close until this is truly, 100% resolved.
Comment 139 Richard Hartmann 2006-11-03 11:54:48 UTC
I fully agree. Please do not, repeat not, close this bug any time soon. It is about the worst thing which I ever witnessed on a linux box (apart from file system crash). When debian sid ate my libc I was not near as unhappy as I am with this bug.

And yes, I also love Kontact, thanks for doing all this work :)
Comment 140 Carsten Pfeiffer 2006-11-03 12:12:30 UTC
I think the biggest problem with this is that the issue has not been publicized (besides this bugreport), e.g. on kmail.kde.org. Such bugs are about as critical as security issues, so there should be a channel to inform users about this.

I'm sure there are a lot of users still using old versions of kmail/kontact which are still vulnerable to this problem. I wouldn't want to wait for them to find out about this bug by themselves...
Comment 141 Sebastiaan Janssens 2006-11-04 00:08:48 UTC
I can confirm the foregoing for use of disconnected imap with KMail 1.9.5 on KDE 3.5.5. (standard Arch Linux KDE packages). I noticed my inbox was reduced from 1500 mails yesterday to 144 tonight. Fortunately, mails on the server get backed up on a daily basis so I was able to request a restore. 

I hope the bug will be fixed soon, because KMail is a joy to use (but without this kind of "fun" ;-)
Comment 142 Arne Schmitz 2006-11-20 20:42:38 UTC
Question: why is this bug not reopened, if people still complain about mail losses? Is it or isn't it safe to use DIMAP at the moment?
Comment 143 Tom Albers 2006-11-20 21:25:21 UTC
The bug is not reopened because there are no reports of mail loss _after_ Till has committed the fix.
Comment 144 BJ Blanchard 2006-11-20 21:54:53 UTC
There is still some dimap mail loss issues, even after Till's patch.  I had one folder in which anything "new" in the folder got immediately deleted - but it seemed to only happen after kmail was running a while ( a couple of days without exiting ).

I spoke to Till breifly on IRC on Nov 16th and he suggested an index rebuild might help.  I wasn't brave enough to chance it, so I switched back to online IMAP.

Till suggested it might be a different bug, but the result is "dimap mail loss", so thus, the post here.

The debug output during this deletion was as follows:

kmail: processNextCheck, remaining 1
kmail: for host gentoo-server.dainty.ca current connections=0 and limit is 0
kmail: connection limit reached: false
kmail: processing next mail check for DiMAP Account
kmail: check mail started - connections for host gentoo-server.dainty.ca now is 1
kmail: Deleting 1 sets of messages from server folder /INBOX.AutoFilter.SavedMail/
kmail: connections to server gentoo-server.dainty.ca now 0
kmail: processNextCheck, remaining 0
kmail: account DiMAP Account finished check


This - occuring during interval mail checking of course... update interval of 5 mintues.  I have a server-side sieve script which puts emails from "root" in this INBOX.AutoFilter.SavedMail folder...  as soon as I noticed I wasn't getting anything new since the previous night, I sent a test message, went to my IMAP server, saw the message come in.. and then saw it "go" at the next interval check.

BJ.
Comment 145 Kimmo 2006-11-21 12:10:57 UTC
I have a simple question: As I mentioned in my comment (#46), my mail losses have occurred when using a normal IMAP-account, _not_  a dIMAP. So it is safe at all to use IMAP with KMail??
Comment 146 BJ Blanchard 2006-11-21 14:49:59 UTC
Kimmo - I have been using IMAP with Kmail corporately (with 60 users) without losing mail for over 5 months (having switched to DIMAP personally only because of some crashes related to IMAP).. 
Comment 147 Ferdinand Gassauer 2006-11-21 15:21:39 UTC
kmail (svn and production) rarely crashes or hangs, but I have not lost any imap - mails since summer 2005.
Comment 148 Florian Beier 2006-11-21 16:50:38 UTC
Does anyone know, if opensuse 10.1 (kmail 1.9.5) has this patch included?
Thx, Florian
Comment 149 Richard Hartmann 2006-11-21 17:42:07 UTC
Ferdinand: At least on slow links, where kmail is unable to load IMAP messages completely before you try to open the next email, crashes are the norm. I was rather certain i already sent out a bug report with detailed reproduction steps, but i obviously only relayed that info via irc. I will probably have time to submit this to here in a few days.
Comment 150 Ferdinand Gassauer 2006-11-21 19:57:00 UTC
Richard: OK, this fits to my experience, almost no crashes in the office (1GB network) , some more at home (DSL line)

BTW I have "load attachments" enabled on demand.
Comment 151 Richard Hartmann 2006-11-22 09:18:05 UTC
Ferdinand: I also load attachments on demand, but in most cases my emails do not have any attachments. Still, it crashes very often. With often, i mean every 10 or so emails. For the same reason, i am mainly using pine, in the meantime..
Comment 152 konold 2006-11-22 09:51:43 UTC
Richard: If you can that easily reproduce the crash then please provide a backtrace with debugging symbols.
Comment 153 Till Adam 2006-11-22 09:56:23 UTC
Can we please keep this somewhat on topic, this is going way outside of what this bug is about. There's the kmail-devel mailing list for general development discussion. This is not the place. Thanks.
Comment 154 Richard Hartmann 2006-11-22 10:04:09 UTC
konold: Yes, i will do so once i have the time which will be in half a week. No way to do it before, sorry. Feel free to follow up by email.

Till: Yes, you are completely right.

As i said, i will open a bug soon and point konold to it. Anyone else who wants it should search for my bugs or send me email. I might even put the link in here, to ease the search for it, dunno.


In any case, the sole reason why i replied via this bug is to close that portion of it for anybody reading these messages some time in the future.
Comment 155 Richard Hartmann 2006-11-28 18:37:54 UTC
Great, i start to write my report and all and then i find http://bugs.kde.org/show_bug.cgi?id=126715 which i did not find earlier.. Anyway for anyone who wants to follow this subissue up, please head that way.

This is the last you hear from me on this subject in this thread. Promise.
Comment 156 Adam Porter 2006-12-29 02:01:58 UTC
(Actually I can't seem to reopen this bug in KDE.  Argh.  If someone doesn't notice and reopen it, I'll file a new one and link it to Debian.)

It's with a heavy heart that I reopen this bug on KDE's and Debian's trackers.  Here's what just happened to me (using KMail 4:3.5.5.dfsg.1-4 from Debian):

Every time I reboot, the system changes my mouse evdev device back and forth from /dev/input/event[1|2] to /dev/input/event[1|2], so I end up with no mouse control, having to switch to another VT, edit xorg.conf and restart X.  KMail is usually in my previous session, and I have auto-login set up in KDM, so when I booted, it logged me in, started KMail and downloaded new mail.  When I returned and saw that there was no mouse control, I edited xorg.conf and restarted KDM.  I probably should have used the keyboard to log out of my KDE session first, rather than restarting KDM and causing X and all child processes to be terminated.  But still, should that have caused seven e-mails that were displaying perfectly fine in my inbox and two subfolders to suddenly turn to "No Subject", completely-empty-view-the-source-and-there's-zilch goners?  Then KMail proceeded to upload those empty mails to the IMAP server, erasing the e-mails there as well.  Luckily I forward all my e-mail to GMail as well, so I can still see them there.

The strange part is, after I restarted X and logged back in and started KMail, the e-mails in my inbox were fine, at first.  I read one and replied to it even!  But then when I clicked on the next unread message, the subject and sender of the message I just clicked on turned into "No Subject" and "Unknown".  And then when I clicked on the message that I had *just read and replied to*, it also turned into "No Subject" "Unknown"!  Then when I checked two other folders, the new messages in them were also "No Subject" "Unknown"!

I hate to be the bearer of bad news, but KMail still has this problem.  (Or maybe this is actually a separate bug caused by a separate problem, but the end result is still mail loss in disconnected IMAP accounts.)
Comment 157 Adam Porter 2006-12-29 02:17:51 UTC
Here is a quick thought that would help to mitigate this bug until it can be truly fixed.  KMail should detect if a message is completely-empty (I mean, if you hit V for View Source, there is *nothing* displayed), and should *not* overwrite messages on the IMAP server if the local copy is empty.  In such a case, it should redownload the message from the server, and overwrite the local copy.  And, of course, it should pop up a dialog box and offer to output some debug info to a file for reporting.
Comment 158 m.wege 2006-12-29 12:08:47 UTC
This (what Adam is describing) is actually the problem I had from the beginning. But with the fix the mails do not disappear forever, they reappear after a time. The mails with "No Subject" remain, but another copy is downloaded when synchronising with the server. 
Comment 159 Adam Porter 2006-12-29 12:45:43 UTC
Thanks, Mark; I wish that were the case, but KMail completely erased the messages' contents on my IMAP server.  I logged in with SquirrelMail and found the messages to be completely empty.  There's nothing left to be downloaded from the server.
Comment 160 Adam Porter 2006-12-29 12:47:37 UTC
By the way (sorry for the extra mail), unfortunately, Debian's KDE team has decided not to reopen their report on this bug.  I hope they will change their mind.  In the meantime, it's clear that KMail's dIMAP support is still not completely trustworthy.  I hope that the KMail devs will reopen this bug on the KDE tracker so that all the folks that have been tracking it will know that it's still a problem, even if it doesn't happen as often.
Comment 161 Chris 2007-01-07 15:33:04 UTC
I now AGAIN lost my hole Mail in my INBOX using kmail 1.9.5 from KDE 3.5.5(Kubuntu)! Sorry, but now I am really really angry. I once lost all my mail then i turned back to normal IMAP and now I switched back to dimap just 3 days ago as this has been marked as solved (so KMail should at least then be stable) and now all my mail is gone again. 

What I did:

1. I deleted several mails - mail by mail - in my Inbox. While deleting a mail KMail suddenly crashed (what it does from time to time when deleting mails also with normal imap). I then got this:

(no debugging symbols found)
Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1".
->38 other lines deleted with: (no debugging symbols found)<-
[Thread debugging using libthread_db enabled]
[New Thread -1242100944 (LWP 5170)]
[New Thread -1281852512 (LWP 5175)]
[New Thread -1273459808 (LWP 5174)]
[New Thread -1265067104 (LWP 5173)]
[New Thread -1256674400 (LWP 5172)]
->72 other lines deleted with: (no debugging symbols found)<-
[KCrash handler]
#6  0xb54f3833 in KMHeaders::msgRemoved () from /usr/lib/libkmailprivate.so
#7  0xb54fef39 in KMHeaders::qt_invoke () from /usr/lib/libkmailprivate.so
#8  0xb6fec957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#9  0xb552dfd0 in KMFolder::msgRemoved () from /usr/lib/libkmailprivate.so
#10 0xb552ec8d in KMFolder::qt_emit () from /usr/lib/libkmailprivate.so
#11 0xb6fec92b in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#12 0xb554bbd0 in FolderStorage::msgRemoved ()
   from /usr/lib/libkmailprivate.so
#13 0xb554d06c in FolderStorage::take () from /usr/lib/libkmailprivate.so
#14 0xb55f4b2c in KMFolderMaildir::take () from /usr/lib/libkmailprivate.so
#15 0xb55e488b in KMFolderCachedImap::take () from /usr/lib/libkmailprivate.so
#16 0xb552c88e in KMFolder::take () from /usr/lib/libkmailprivate.so
#17 0xb55f8685 in KMFolderMaildir::addMsgInternal ()
   from /usr/lib/libkmailprivate.so
#18 0xb55e4847 in KMFolderCachedImap::addMsg ()
   from /usr/lib/libkmailprivate.so
#19 0xb554a7a3 in FolderStorage::moveMsg () from /usr/lib/libkmailprivate.so
#20 0xb552c9c5 in KMFolder::moveMsg () from /usr/lib/libkmailprivate.so
#21 0xb567755f in KMMoveCommand::execute () from /usr/lib/libkmailprivate.so
#22 0xb5669949 in KMCommand::slotPostTransfer ()
   from /usr/lib/libkmailprivate.so
#23 0xb5671386 in KMCommand::qt_invoke () from /usr/lib/libkmailprivate.so
#24 0xb567166b in KMMenuCommand::qt_invoke () from /usr/lib/libkmailprivate.so
#25 0xb56716f7 in KMMoveCommand::qt_invoke () from /usr/lib/libkmailprivate.so
#26 0xb567176b in KMDeleteMsgCommand::qt_invoke ()
   from /usr/lib/libkmailprivate.so
#27 0xb6fec957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#28 0xb56699de in KMCommand::messagesTransfered ()
   from /usr/lib/libkmailprivate.so
#29 0xb5672241 in KMCommand::transferSelectedMsgs ()
   from /usr/lib/libkmailprivate.so
#30 0xb56723a7 in KMCommand::slotStart () from /usr/lib/libkmailprivate.so
#31 0xb5671398 in KMCommand::qt_invoke () from /usr/lib/libkmailprivate.so
#32 0xb567166b in KMMenuCommand::qt_invoke () from /usr/lib/libkmailprivate.so
#33 0xb56716f7 in KMMoveCommand::qt_invoke () from /usr/lib/libkmailprivate.so
#34 0xb567176b in KMDeleteMsgCommand::qt_invoke ()
   from /usr/lib/libkmailprivate.so
#35 0xb6fec957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#36 0xb7378f44 in QSignal::signal () from /usr/lib/libqt-mt.so.3
#37 0xb700c8ea in QSignal::activate () from /usr/lib/libqt-mt.so.3
#38 0xb7014300 in QSingleShotTimer::event () from /usr/lib/libqt-mt.so.3
#39 0xb6f83b88 in QApplication::internalNotify () from /usr/lib/libqt-mt.so.3
#40 0xb6f859b7 in QApplication::notify () from /usr/lib/libqt-mt.so.3
#41 0xb76acdb2 in KApplication::notify () from /usr/lib/libkdecore.so.4
#42 0xb6f16389 in QApplication::sendEvent () from /usr/lib/libqt-mt.so.3
#43 0xb6f765d3 in QEventLoop::activateTimers () from /usr/lib/libqt-mt.so.3
#44 0xb6f2aec5 in QEventLoop::processEvents () from /usr/lib/libqt-mt.so.3
#45 0xb6f9e25e in QEventLoop::enterLoop () from /usr/lib/libqt-mt.so.3
#46 0xb6f9e06e in QEventLoop::exec () from /usr/lib/libqt-mt.so.3
#47 0xb6f85731 in QApplication::exec () from /usr/lib/libqt-mt.so.3
#48 0x0805ad81 in ?? ()
#49 0xbf8d7a40 in ?? ()
#50 0x00000001 in ?? ()
#51 0x00000001 in ?? ()
#52 0x00000000 in ?? ()

2. To be secure I then restarted KMail and selected "Lokalen IMAP Zwischenspeicher aktualisieren" (what means something like "Refresh local IMAP cache"). - Kmail then crashed a second time with this result:

(no debugging symbols found)
Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1".
->37 other lines deleted with: (no debugging symbols found)<-
[Thread debugging using libthread_db enabled]
[New Thread -1241482448 (LWP 5700)]
[New Thread -1281234016 (LWP 5704)]
[New Thread -1272841312 (LWP 5703)]
[New Thread -1264448608 (LWP 5702)]
[New Thread -1256055904 (LWP 5701)]
->79 other lines deleted with: (no debugging symbols found)<-
[KCrash handler]
#6  0xb558a833 in KMHeaders::msgRemoved () from /usr/lib/libkmailprivate.so
#7  0xb5595f39 in KMHeaders::qt_invoke () from /usr/lib/libkmailprivate.so
#8  0xb7083957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#9  0xb55c4fd0 in KMFolder::msgRemoved () from /usr/lib/libkmailprivate.so
#10 0xb55c5c8d in KMFolder::qt_emit () from /usr/lib/libkmailprivate.so
#11 0xb708392b in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#12 0xb55e2bd0 in FolderStorage::msgRemoved ()
   from /usr/lib/libkmailprivate.so
#13 0xb55e406c in FolderStorage::take () from /usr/lib/libkmailprivate.so
#14 0xb568bb2c in KMFolderMaildir::take () from /usr/lib/libkmailprivate.so
#15 0xb567b88b in KMFolderCachedImap::take () from /usr/lib/libkmailprivate.so
#16 0xb55c388e in KMFolder::take () from /usr/lib/libkmailprivate.so
#17 0xb568f685 in KMFolderMaildir::addMsgInternal ()
   from /usr/lib/libkmailprivate.so
#18 0xb568ffa7 in KMFolderMaildir::addMsg () from /usr/lib/libkmailprivate.so
#19 0xb55e17a3 in FolderStorage::moveMsg () from /usr/lib/libkmailprivate.so
#20 0xb55c39c5 in KMFolder::moveMsg () from /usr/lib/libkmailprivate.so
#21 0xb56103a6 in KMFilterMgr::process () from /usr/lib/libkmailprivate.so
#22 0xb5582a42 in KMAccount::processNewMsg () from /usr/lib/libkmailprivate.so
#23 0xb567cca0 in KMFolderCachedImap::addMsgInternal ()
   from /usr/lib/libkmailprivate.so
#24 0xb574723d in KMail::CachedImapJob::slotGetNextMessage ()
   from /usr/lib/libkmailprivate.so
#25 0xb5744812 in KMail::CachedImapJob::qt_invoke ()
   from /usr/lib/libkmailprivate.so
#26 0xb7083957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#27 0xb6b6477e in KIO::Job::result () from /usr/lib/libkio.so.4
#28 0xb6ba4a8d in KIO::Job::emitResult () from /usr/lib/libkio.so.4
#29 0xb6bb875e in KIO::SimpleJob::slotFinished () from /usr/lib/libkio.so.4
#30 0xb6bb8e6d in KIO::TransferJob::slotFinished () from /usr/lib/libkio.so.4
#31 0xb6ba46ba in KIO::TransferJob::qt_invoke () from /usr/lib/libkio.so.4
#32 0xb7083957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#33 0xb70843fc in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#34 0xb6b5effc in KIO::SlaveInterface::finished () from /usr/lib/libkio.so.4
#35 0xb6bc4720 in KIO::SlaveInterface::dispatch () from /usr/lib/libkio.so.4
#36 0xb6bc275a in KIO::SlaveInterface::dispatch () from /usr/lib/libkio.so.4
#37 0xb6b7343c in KIO::Slave::gotInput () from /usr/lib/libkio.so.4
#38 0xb6bb2360 in KIO::Slave::qt_invoke () from /usr/lib/libkio.so.4
#39 0xb7083957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#40 0xb708426e in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#41 0xb7410cdb in QSocketNotifier::activated () from /usr/lib/libqt-mt.so.3
#42 0xb70a6516 in QSocketNotifier::event () from /usr/lib/libqt-mt.so.3
#43 0xb701ab88 in QApplication::internalNotify () from /usr/lib/libqt-mt.so.3
#44 0xb701c9b7 in QApplication::notify () from /usr/lib/libqt-mt.so.3
#45 0xb7743db2 in KApplication::notify () from /usr/lib/libkdecore.so.4
#46 0xb6fad389 in QApplication::sendEvent () from /usr/lib/libqt-mt.so.3
#47 0xb700cf81 in QEventLoop::activateSocketNotifiers ()
   from /usr/lib/libqt-mt.so.3
#48 0xb6fc1ea7 in QEventLoop::processEvents () from /usr/lib/libqt-mt.so.3
#49 0xb703525e in QEventLoop::enterLoop () from /usr/lib/libqt-mt.so.3
#50 0xb703506e in QEventLoop::exec () from /usr/lib/libqt-mt.so.3
#51 0xb701c731 in QApplication::exec () from /usr/lib/libqt-mt.so.3
#52 0x0805ad81 in ?? ()
#53 0xbfdfe760 in ?? ()
#54 0x00000001 in ?? ()
#55 0x00000001 in ?? ()
#56 0x00000000 in ?? ()

After this step my cached Inbox was completely erased except one single mail (the first one in the folder). - I then checke my webmail access to this account. But there all mails were still in the folder.

3. I then tried to start KMail a third time and also tried to again refresh the local IMAP cache. But it again crashed with this:

(no debugging symbols found)
Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1".
->39 other lines deleted with: (no debugging symbols found)<-
[Thread debugging using libthread_db enabled]
[New Thread -1241797840 (LWP 6136)]
[New Thread -1281549408 (LWP 6141)]
[New Thread -1273156704 (LWP 6140)]
[New Thread -1264764000 (LWP 6139)]
[New Thread -1256371296 (LWP 6138)]
->68 other lines deleted with: (no debugging symbols found)<-
[KCrash handler]
#6  0x08398834 in ?? ()
#7  0xb58460b8 in ?? () from /usr/lib/libkmailprivate.so
#8  0xb553d94e in KMHeaders::msgRemoved () from /usr/lib/libkmailprivate.so
#9  0xb5548f39 in KMHeaders::qt_invoke () from /usr/lib/libkmailprivate.so
#10 0xb7036957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#11 0xb5577fd0 in KMFolder::msgRemoved () from /usr/lib/libkmailprivate.so
#12 0xb5578c8d in KMFolder::qt_emit () from /usr/lib/libkmailprivate.so
#13 0xb703692b in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#14 0xb5595bd0 in FolderStorage::msgRemoved ()
   from /usr/lib/libkmailprivate.so
#15 0xb559706c in FolderStorage::take () from /usr/lib/libkmailprivate.so
#16 0xb563eb2c in KMFolderMaildir::take () from /usr/lib/libkmailprivate.so
#17 0xb562e88b in KMFolderCachedImap::take () from /usr/lib/libkmailprivate.so
#18 0xb557688e in KMFolder::take () from /usr/lib/libkmailprivate.so
#19 0xb5642685 in KMFolderMaildir::addMsgInternal ()
   from /usr/lib/libkmailprivate.so
#20 0xb5642fa7 in KMFolderMaildir::addMsg () from /usr/lib/libkmailprivate.so
#21 0xb55947a3 in FolderStorage::moveMsg () from /usr/lib/libkmailprivate.so
#22 0xb55769c5 in KMFolder::moveMsg () from /usr/lib/libkmailprivate.so
#23 0xb55c33a6 in KMFilterMgr::process () from /usr/lib/libkmailprivate.so
#24 0xb5535a42 in KMAccount::processNewMsg () from /usr/lib/libkmailprivate.so
#25 0xb562fca0 in KMFolderCachedImap::addMsgInternal ()
   from /usr/lib/libkmailprivate.so
#26 0xb56fa23d in KMail::CachedImapJob::slotGetNextMessage ()
   from /usr/lib/libkmailprivate.so
#27 0xb56f7812 in KMail::CachedImapJob::qt_invoke ()
   from /usr/lib/libkmailprivate.so
#28 0xb7036957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#29 0xb6b1777e in KIO::Job::result () from /usr/lib/libkio.so.4
#30 0xb6b57a8d in KIO::Job::emitResult () from /usr/lib/libkio.so.4
#31 0xb6b6b75e in KIO::SimpleJob::slotFinished () from /usr/lib/libkio.so.4
#32 0xb6b6be6d in KIO::TransferJob::slotFinished () from /usr/lib/libkio.so.4
#33 0xb6b576ba in KIO::TransferJob::qt_invoke () from /usr/lib/libkio.so.4
#34 0xb7036957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#35 0xb70373fc in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#36 0xb6b11ffc in KIO::SlaveInterface::finished () from /usr/lib/libkio.so.4
#37 0xb6b77720 in KIO::SlaveInterface::dispatch () from /usr/lib/libkio.so.4
#38 0xb6b7575a in KIO::SlaveInterface::dispatch () from /usr/lib/libkio.so.4
#39 0xb6b2643c in KIO::Slave::gotInput () from /usr/lib/libkio.so.4
#40 0xb6b65360 in KIO::Slave::qt_invoke () from /usr/lib/libkio.so.4
#41 0xb7036957 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#42 0xb703726e in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#43 0xb73c3cdb in QSocketNotifier::activated () from /usr/lib/libqt-mt.so.3
#44 0xb7059516 in QSocketNotifier::event () from /usr/lib/libqt-mt.so.3
#45 0xb6fcdb88 in QApplication::internalNotify () from /usr/lib/libqt-mt.so.3
#46 0xb6fcf9b7 in QApplication::notify () from /usr/lib/libqt-mt.so.3
#47 0xb76f6db2 in KApplication::notify () from /usr/lib/libkdecore.so.4
#48 0xb6f60389 in QApplication::sendEvent () from /usr/lib/libqt-mt.so.3
#49 0xb6fbff81 in QEventLoop::activateSocketNotifiers ()
   from /usr/lib/libqt-mt.so.3
#50 0xb6f74ea7 in QEventLoop::processEvents () from /usr/lib/libqt-mt.so.3
#51 0xb6fe825e in QEventLoop::enterLoop () from /usr/lib/libqt-mt.so.3
#52 0xb6fe806e in QEventLoop::exec () from /usr/lib/libqt-mt.so.3
#53 0xb6fcf731 in QApplication::exec () from /usr/lib/libqt-mt.so.3
#54 0x0805ad81 in ?? ()
#55 0xbf9ffb60 in ?? ()
#56 0x00000001 in ?? ()
#57 0x00000001 in ?? ()
#58 0x00000000 in ?? ()

Now all mails also were erased from the server and so I lost my Inbox again.

I work for a radio station and do all important communications via Email using KMail so this behaviour is a big issue for me. (And it is even more severe as KMail does not even have a feature to backup and restore Mail folders.)
Comment 162 Chris 2007-01-08 21:42:05 UTC
Now I had another mail loss with my second account and dimap. I had 13 mails in my inbox and was reading several mails in this folder (GMX Account). I then right clicked my inbox and selected "Troubleshoot IMAP cache" -> "Rebuild Index". I got a message that rebuilding index was successful. Everything still looked fine.

After the next synchronize 10 of my 13 mails were deleted from the server and by the second synchronize they also were deleted locally. I did this several times now by copying some mails in my inbox, rebuilding index and synchronizing twice. Each time many of the mails in my inbox were deleted after synchronizing twice.
Comment 163 Adam Porter 2007-01-09 13:19:00 UTC
Would a KDE dev please reopen this bug report?
Comment 164 Richard Hartmann 2007-01-09 22:29:09 UTC
Apart from reopening this bug, there is an issue that is much more important:

KDE, as a whole, is production level software. It is stable, reliable and in productive everyday use in lots of places. The complete system is of a very high, if not superior, quality.

Kontact, more specifically KMail, stands out in two very negative aspects. While the frequent and easily reproducable crashes (http://bugs.kde.org/show_bug.cgi?id=126715) are highly annoying, they can be coped with and do not lose data (unless you are writing an email). On the other hand, the issue we are discussing in this bug report has been losing real data of real people for over one and a half years. While some people might argue that backups are vital for all parts of our digital lives, these are meant for emergencies, not the common case. Making these emergencies the common case does not help anyone.

My suggestion consists of three parts:

1) Remove the option of creating dimap accounts, bury it in some deeper option or _at least_ warn everyone wanting to create a dimap account or anyone using one currently of the very real chance of them losing all their mail, in this order of preference.

2) Force the user to manually confirm every deletion of mail, especially if KMail wants to batch delete anything and/or a large portion of the mails in any given folder/account. To make this less annoying, KMail could wait until the end of the session with that question (which is bad if KMail dies or if the user walks away immediately when closing his KDE session) or move mail into another folder instead of deleting (which is bad if there is not enough free storage left). Yes, no matter what is done, this is still a pain.

3) Keep a detailed log of all actions and decisions the mail handler takes and for what reason it takes them. Make it easy (http upload, email sending, whatever, just make it a one-click affair) for those logs to be sent to a central site where they can be looked at accordingly.


I am fully aware that all of these three steps are disruptive. Still, this issue is amongst the very worst ever to haunt KDE and has been open for way too long. Let me stress that this is not an attack against anyone, it is just a desperate call for action.

I know of no single bug, or group of bugs, that is as devastating to the whole experience of KDE for any user as this one. Please change this situation.

Richard Hartmann
Comment 165 Julian Mehnle 2007-01-09 23:03:27 UTC
Richard Hartmann wrote:
> 1) Remove the option of creating dimap accounts, bury it in some deeper
> option or _at least_ warn everyone wanting to create a dimap account or
> anyone using one currently of the very real chance of them losing all
> their mail, in this order of preference.

Removing DIMAP would suck because the groupware functionality doesn't work with normal IMAP anymore -- support for that was dropped in KDE 3.3 or something (don't ask me why).

(I'm currently using IMAP for my normal mail use and DIMAP on the _same_ IMAP account for the calendar and contacts.)

> [...] this issue is amongst the very worst ever to haunt KDE and has been
> open for way too long.

But don't you understand??  This bug is not open anymore!  It is closed now!
</sarcasm>

> Let me stress that this is not an attack against anyone, it is just a
> desperate call for action.

I second this wholeheartedly. *sigh*
Comment 166 Bastian Venthur 2007-01-10 13:55:16 UTC
Here are some minimalistic ways to reproduce this bug.


Some observations:

1) KMail does not show what's on the server but rather what should be on
the server:

When moving mails around in KMail, the mails aren't actually moved on
the server directly. KMail does this later (when?). Even when you close
and reopen KMail the mail is not moved on the server, although KMail
pretends that it has moved the mail! (Which is the root of the problem
as we will see later)

2) Pressing F5 inside a folder commits open changes for this folder on
the server.

2.a) Pressing F5 inside folder foo commits changes for foo but not for
folder bar (which is so dangerous that I cannot believe this was
actually implemented)

3) Rebuilding cache (sorry translated from my German version) seems to
re-read all the data from the server and overwrites KMails yet
uncommitted actions -- for this folder!

4) People who use IMAP tend to check their mails from more than one machine.

5) People have more than one account and sometimes move files from one
folder to another one on a different account.


Here is what I've used:

- Current KMail 1.9.5 from Sid
- current Dovecot from etch on the server side
- Initial layout: Four mails in inbox and two empty folders:

|-- Inbox
|   |-- mail 1
|   |-- mail 2
|   |-- mail 3
|   `-- mail 4
|-- folder 1
`-- folder 2



Ok let's get to the bug. I found different methods to reproduce this
bug. Here is the first and simplest method: I'll delete a mail by moving
a mail from one folder to another:

1) Drag mail 1 to folder 1 (move).
2) Now take a look on the server: Inbox has still 4 mails, folder 1 is
still empty. The changes are not committed to the server although KMail
pretends to.

3) Now go into your inbox and press F5 (refresh)
4) On the serverside mail 1 is now deleted from your Inbox but still not
present in folder 1. In other words if you'd close KMail now and delete
all it's configuration files from this box, mail 1 is lost!

Of course mail 1 is still living somewhere in KMails config files an
pressing F5 folder 1 will bring it back, but only on this machine.
Checking your mail from another box, it will look like your mail just
disappeared. Which is actually true since it is not on the server anymore.

Think this scenario is unrealistic? I for one often move a mail from my
inbox to another folder. And even more often press F5 in my inbox to
fetch new mail.


Here a second way, even funnier:

1) Move mail 1 (by dragging) from folder 1 to folder 2

The changes are not yet committed on the server! But KMail pretends to.

2) Rebuild cache on folder 2 (KMail will delete the just moved mail from
this folder since it's not present on server)

3) Refresh folder 1 with F5. KMail will remove the mail from the server
(which was not visible inside KMail)

Now you have effectively deleted mail mail 1 from KMail and the Server.


I'm sure there are many other ways to exploit this behavior. I've not
seen a single waring or error from KMail it just did what I say and
happily removed mail precious mails. I found another way to lose mails
in combination with two different boxes using the same account and
moving mail, and (ab)using F5 and rebuilding the cache. But I forgot to
write the steps down and am to lazy now to try to reproduce it again.

I even found a bug where KMail duplicates mails which is relevant to
this bug since it's the same problem.

I've not yet played with different boxes sharing the same account (home,
work) or one box having two accounts (and moving mails), but I hope my
setup inspired a few people to do some testing themselves. Please let me
know if you need some help or further information.


So please urge to upstream to reopen it's relevant bugreport(s) (or
better to fix them), I don't have the nerve to deal with upstreams BTS.

And for Debian: please disable DIMAP completely or remove KMail from etch.



Cheers,

Bastian


-- 
Bastian Venthur                                      http://venthur.de
Debian Developer                                 venthur at debian org
Comment 167 Modestas Vainius 2007-01-10 15:18:27 UTC
Is anyone from upstream going to work on fixing the behaviour described in the comment #166 soon or should Debian disable dimap? Any pointers how to do the latter for existing dimap accounts? By the way, this bug turns into "dimap is broken by design" bug which I agree with.
Comment 168 Bastian Venthur 2007-01-10 15:26:29 UTC
Please merge this bug with http://bugs.kde.org/show_bug.cgi?id=114163 which is open since Oct 2005 and still normal (KDE's Bug Tracking System really sucks) and refers to the very same problem.
Comment 169 Carsten Pfeiffer 2007-01-10 20:44:46 UTC
On Wednesday 10 January 2007 13:55, Bastian Venthur wrote:

> 1) KMail does not show what's on the server but rather what should be on
> the server:
>
> When moving mails around in KMail, the mails aren't actually moved on
> the server directly. KMail does this later (when?). Even when you close
> and reopen KMail the mail is not moved on the server, although KMail
> pretends that it has moved the mail! (Which is the root of the problem
> as we will see later)


Isn't that actually the purpose of dimap (*isconnected IMAP)? I.e. you operate 
in a disconnected mode, delete mails, move them around, etc. and only when 
you explicitly "synchronize", e.g. by pressing F5, then your changes will be 
committed to the server (and changes on the server will be propagated to 
your).

> 2) Pressing F5 inside a folder commits open changes for this folder on
> the server.


Yes, that's a feature, AFAICS.
Comment 170 Richard Hartmann 2007-01-11 01:27:05 UTC
> Isn't that actually the purpose of dimap (*isconnected IMAP)? I.e. you operate
> in a disconnected mode, delete mails, move them around, etc. and only when
> you explicitly "synchronize", e.g. by pressing F5, then your changes will be
> committed to the server (and changes on the server will be propagated to
> your).

In my book, the main point of disconnected IMAP is the local storage of emails. Of course, dimap should work when fully offline. Still, while online, at least moving and deleting is almost without cost.


>> 2) Pressing F5 inside a folder commits open changes for this folder on
>> the server.
>
> Yes, that's a feature, AFAICS.

It might be a nice-to-have feature when you press shift-f5 or some such. Having partial sync as standard behaviour does not make sense to me, though. How often do you edit a subfolder and want to submit changes of this one branch, only? If you did not edit anything in one branch syncing will not hurt. On the other hand, if you changed things in there, chances are you want them to be applied to the server, too.


Still, your two replies are not even the point here. The direct consequence of the behaviour above is lost mail for users. Even if the design was made deliberately, it is inherently broken and needs to be fixed. You can either introduce clutches like tracing potentially harmful actions like the delete part of a move and such or treating any copy, delete cycle as one atomar action which can not be split up no matter what accounts it may span (and needs to be reverted or aborted if any part of it does not work) or you can fix the basic assumptions and redesign from there.

In any case, no matter if this behaviour is intended, no matter what the rationale for this behaviour is, no matter how you look at it, this issue is making an otherwise brilliant piece of software something I have to warn each and everyone I know about.
'Hey, you are using KDE? Cool. You are using Kontact? Even better! You are not using dimap though, are you? Well, it is losing mail. Yes, it is a known issue and has been open for more than 18 months..' - Not fun.
Comment 171 Chris Peikert 2007-01-11 01:58:19 UTC
> In my book, the main point of disconnected IMAP is the local storage of emails.

Others have pointed this out as well, but in my book the main point of dIMAP is for sharing contacts/todos/calendar/... among many machines.  I'd gladly use plain IMAP (and tried to, once) if all my kde-pim info stayed sync'ed and useable.

This is the great downside of disabling dimap at this point: it will kill significant functionality that can't be had any other way.



Comment 172 Rob Kaper 2007-01-11 02:14:46 UTC
On Thursday 11 January 2007 01:58, Chris Peikert wrote:
> I'd gladly use plain IMAP (and tried to, once) if all my kde-pim info stayed
> sync'ed and useable.


And I would not, because offline mail data is crucial for people who have a 
laptop and will want to access their e-mail when travelling or when working 
at a remote location where there might be no Internet connectivity.

> This is the great downside of disabling dimap at this point: it will kill
> significant functionality that can't be had any other way.


True. Offline requirements for just e-mail could easily be met with POP.

On a sidenote though, I no longer experience this bug - or at best it's 
dormant and not frequent like it could be in the past. Then again my ISP 
changed mail server software as well a couple of months ago so I can't rule 
out there's a server-side variable in the behaviour.

Still I think it's best if this bug would remain open until all of the ID 
error catching as thoroughly describe by (was it Till? Adam? I forgot.) and 
there is a better argument for closing it than "it seems to be dormant".

Rob
Comment 173 Ferdinand Gassauer 2007-01-11 07:40:58 UTC
FYI - I am using a Kolab server to store my contacts centraly and so far I have not been loosing anything. May be that ths syncing issue is solved better there?
Comment 174 Carsten Pfeiffer 2007-01-11 20:33:43 UTC
On Thursday 11 January 2007 01:27, Richard Hartmann wrote:

> Having partial sync as standard behaviour does not make sense to me,


I for one never sync partially. I automatically synchronize periodically every 
some minutes and if I need to, I synchronize manually. It was quite obvious 
to me, that I would lose uncommitted changes when rebuilding the folder 
index, so I never did that (why would I anyway?).

So maybe your habits of dealing with mails does not fit well with the idea 
that the kmail developers had, when implementing dimap.

In any case, kmail/kontact should never crash and never lead to data loss. 

The report in comment #166 however is IMO not a bug at all. That's just how 
dimap works in kmail. You commit your changes only during synchronization, 
and not upon every single action. Rebuilding the index obviously clears all 
the locally stored changes, so it's no wonder they are lost.

The crashes in #161 and #162 seem to be valid though, so I'm gonna reopen this 
bug (note that I'm not a kmail developer).
Comment 175 Carsten Pfeiffer 2007-01-11 20:36:02 UTC
Reopening, see comment #161 and comment #162
Comment 176 Ferdinand Gassauer 2007-01-11 21:02:57 UTC
So may be kmail could issue a message that partial sync or rebuilding index WITHOUT prior syncing might loose mails. 

Do you want to sync your (dimap) folders prior to <action> to avoid possible loss of mail?
default YES

BTW I doubt, that normal users will be able to understand "dimap". 
Comment 177 Ferdinand Gassauer 2007-01-11 21:05:49 UTC
Or even better in kmail settings

* Allow partial syncing - default NO

that's what I would want to have for my users.
Comment 178 Chris 2007-01-11 21:36:36 UTC
Reply to #177: I personally would prefer being on the safe side and doing a full sync with F5 in dimap-mode instead. As it may be destructive partial sync should not be a standard feature in my opinion. 
Comment 179 Carsten Pfeiffer 2007-01-11 21:54:59 UTC
> Reply to #177: I personally would prefer being on the safe side and doing a
> full sync with F5 in dimap-mode instead. As it may be destructive partial
> sync should not be a standard feature in my opinion.


As long as this is not more visible to users, you could remove the F5 shortcut 
and set it as an alternate shortcut for the full synchronization action 
(Ctrl-L by default).
Comment 180 Ferdinand Gassauer 2007-01-11 22:19:36 UTC
IMHO kmail should NOT be configured / distributed in a way that loss of mail is possible. 
If an "expert" changes his settings, it is his problem. On the other hand, we have seen to many "experts" messing up EDP Systems.
Comment 181 Carsten Pfeiffer 2007-01-11 22:29:15 UTC
> IMHO kmail should NOT be configured / distributed in a way that loss of
> mail is possible. If an "expert" changes his settings, it is his problem.
> On the other hand, we have seen to many "experts" messing up EDP Systems.


I only wanted to suggest a workaround for the time that this problem is not 
dealt with properly.

Maybe F5 should refresh
- the current folder
- all locally modified folders
Comment 182 groot 2007-01-14 13:06:06 UTC
There is a bug in the KDE Bugzilla, #104956 [1] which has attracted an 
enormous amount of comments and it has been closed and reopened several 
times. This bug report has become a kind of magnet for *all* cases of mail 
loss (real or perceived) where DIMAP is vaguely involved. The KMail 
developers believe that the *original* bug has been solved -- but not 
necessarily all other issues related to DIMAP. However, because the bug is 
still attracting comments (over 180 comments as of this writing) it is 
horribly confusing. Especially since the bug now contains *several* different 
and unrelated issues (except they all have DIMAP as common factor). At the 
Osnabrueck meeting this weekend the KMail developers and the rest of the KDE 
PIM team talked about the bug and its resolution and came to the following 
conclusion:

- the bug will be closed again.
- people with current issues with DIMAP [2] are asked to open *new* bugs.
- further comments on bug 104956 will be ignored.

This measure has everything to do with keeping the bug system usable for 
developers as well. The bug system is *not* a place for long discussion, and 
this bug has become unmanageable as a result. For instance, the crashes 
reported in comment #161 are best dealt with as a separate issue instead of 
buried in such a long discussion. Those are related to messages being 
deleted "out from under" other KMail parts, not DIMAP loss due to flaky 
internet connection which is what the original bug was and which has been 
fixed. Again: distinct bugs which share only the common factor DIMAP should 
not be bunched together in the bug tracker.

This is being posted to the kde-pim mailing list so that any discussion can 
happen *there*, also in public. We believe in the good will and integrity of 
those people who have commented on the bug and the various other bugs in 
there; we ask that they *continue* to watch over DIMAP bugs, just not this 
one -- because the bug report itself is broken.


[ade], writing on behalf of the KMail and KDE PIM developers (Ingo, Till, 
Cornelius, Will, Volker, Thorsten, Martin, Frank and Tobias, to name a few).


[1] http://bugs.kde.org/show_bug.cgi?id=104956

[2] For instance, you can effectively shoot yourself in the foot with a 
combination of F5 (sync only this folder) and message moving plus destroying 
the local cache. I will illustrate this with an example from SVN to show how 
this foot-shooting works. Suppose you have a SVN checkout with directories a/ 
and b/ and a file f in b/ .

	$ cd b/
	$ svn mv f ../a/

Now f is marked as to-be-deleted in b/ and to-be-added in a/

	$ svn commit -m "Move f"

This commits *only* the delete in b/, because you're in b/. The use of F5 is 
comparable. Now throw away your local changes on a/ :

	$ cd ..
	$ rm -rf a
	$ svn up

*BLAM!*
Comment 183 Chris 2007-01-14 14:30:44 UTC
Now I am really shocked about how severe bugs are dealt with. They are just closed because "too complicated" instead of analyzed in deep. I agree in one point that the F5 issue is a differen problem. 

My #161 and #162 are in no way related to any F5 problems. They are synchronization issues and they are really severe. I lost lots of important mail through this twice and I tried to report this as detailed as possible and this also was much time. 

Anyhow it has been decided to leave kmail broken until even more important mail is lost. Wow. :-(
Comment 184 Josh Berry 2007-01-14 18:57:23 UTC
"Anyhow it has been decided to leave kmail broken until even more important mail is lost. Wow. :-( "

This is absolutely not the case.  Had you thoroughly read what ade wrote, you would have seen where he *specifically asked* users affected by DIMAP issues to open new bugs.  (Otherwise, devs wouldn't be able to keep all the issues straight.)

I quote:

  - people with current issues with DIMAP [2] are asked to open *new* bugs.
...
  Again: distinct bugs which share only the common factor DIMAP should not be bunched together in the bug tracker.

The bug you are observing is a different bug than the one initially reported here.  The symptoms may be the same, but the causes are apparently different.  Thus, your report belongs in a separate bug.  F5 was merely an example.
Comment 185 Chris Peikert 2007-01-14 19:43:25 UTC
The kmail devs are acting in good faith here -- they want to see bugs fixed, and so do all of us.  It behooves all of us to keep the bug reports organized, so things can be diagnosed and fixed in a systematic way.  It is just simply too complicated to lump all issues having some dimap connection into one bug.  Separate issues need to be kept separate.

If new bugs are opened, I have no doubt that they will be assigned the appropriate severity and given the attention they are due.

(Just another kmail dimap user who wants a more perfect mail application.)
Comment 186 Richard Hartmann 2007-01-14 20:25:57 UTC
My suggestion would be that, while keeping this bug closed, we 'abuse' this specific bug report to keep track of the other bugs. I am _very_ interested in this whole issue and so are probably most of the people who are still subscribed to this bug. Everyone who does not want to have his or her inbox fill with messages, simply unsubscribe to this bug which will not get reopened any more, anyway.

So, I would ask anyone who opens a new bug that concerns mail loss with dIMAP to please add a short note to this bug, stating what bug number you got assigned.


Thanks :)
Comment 187 groot 2007-01-14 21:15:43 UTC
Use thesaurus to "hide" this bug as suggested by Jan de Visser on kde-pim. The point being to avoid the "catch-all" nature of this bug summary line.
Comment 188 Bastian Venthur 2007-01-14 21:32:58 UTC
Ok, then please rename

 http://bugs.kde.org/show_bug.cgi?id=114163

to "sudden mail loss" and set it's severity to something more appropriate like, say: grave. I'm the reporter of this bug and it is open since Oct 2005 (read: 1,5 years)

Thanks.

BTW it is a shame that Users are not allowed to change the severity of a bug or even rename it. Here in Debian, users have those rights and as a Debian Developer I can say, I have never noticed any drawbacks.

Cheers,

- -
Bastian Venthur            http://venthur.de
Debian Developer      venthur at debian oorg
Comment 189 Adam Porter 2007-01-17 22:45:10 UTC
I just had a dIMAP-related crash.  Here is the new bug:

http://bugs.kde.org/show_bug.cgi?id=140213
Comment 190 Adam Porter 2007-01-18 02:58:44 UTC
If you are interested in KMail's dIMAP bugs, please vote for the year-old, oft-reported http://bugs.kde.org/show_bug.cgi?id=117935 .
Comment 191 Donatas Glodenis 2007-09-13 08:25:44 UTC
OK, I see this bug has been marked as fixed, and I am not as good at interpreting, if the bug I have experienced is the same, but here it is:

I am using Kmail 1.9.7 on Kubuntu Feisty (i386 version).

A few days ago I synched my dimap mail account after a long while of not doing it. There were a few hundred mail that needed to be filtered into different dimap folders. Kmail crashed in the middle of the operation.

After restarting it I found some of my dimap folders completely empty, and some 12 email messages in my inbox with headers "No Subject" and completely empty. 

I checked that the empty kmail folders were still with messages on server (though it seems, not all of those), but the "No Subject" messages in KMail's inbox were also empty in the server (checked with squirell mail). So I just deleted my Kmail Dimap account and recreated it using normal imap.

Then in a few days I saw some mail loss in another of my dimap accounts as well. Incidentally I had my birthday the day before, so I opened the folder and saw a "Happy birthday" mail message from my friend, which suddenly turned into "No Subject" mail message once I clicked on it :(  How do I know if I did not loose some important mail for my doctoral studies? I have a feeling that I do loose mail from time to time as kmail crashes (it does once in a while), but those cases leave no sign except for me being unable to find some old message from time to time, or seeing strange gaps in my mailing list folders.

Please DO something about this. I myself really like Kmail and still use it, but for my wife I configured thunderbird without a second thought.
Comment 192 Florian Beier 2007-12-10 10:28:20 UTC
Bug seems to be beack again. Second time that I lost tons of mails. Cannot give any more specific details. But it is definetely kmail & dimap related.
I am using Gentoo, Kmail 1.9.7.