Bug 121866

Summary: Messages in IMAP folders disappear and reappear during mail check
Product: [Applications] kmail Reporter: Lincoln Peters <sampln>
Component: IMAPAssignee: kdepim bugs <kdepim-bugs>
Severity: normal CC: bjoern, brian, gsocker, isleshocky77, no-spam, roman.karlstetter, sven.burmeister, wedgeshot
Priority: NOR    
Version: 1.10.0   
Target Milestone: ---   
Platform: unspecified   
OS: Linux   
Latest Commit: Version Fixed In:
Attachments: Ethereal capture of IMAP session (compressed using bzip2)

Description Lincoln Peters 2006-02-13 06:44:54 UTC
Version:           unknown (using KDE 3.5.1, Debian Package 4:3.5.1-2 (testing/unstable))
Compiler:          Target: i486-linux-gnu
OS:                Linux (i686) release

When I check for mail on an IMAP account (not disconnected), all recently-received messages disappear prior to any mail check, any new messages appear, and then the messages that disappeared earlier reappear.  This does not seem to cause any harm, but it is horrendously inefficient if there are a lot of messages in an IMAP folder so affected.  That gross inefficiency is why I submit this as a bug rather than a wish.

I can tell that the messages that so disappear and reappear are doing so on the IMAP server, since if I have the same account open with a different application when KMail checks it for new mail (I used Mozilla Thunderbird), I can see the messages disappear and reappear there as well.  (Incidentally, since Thunderbird doesn't seem to exhibit this behavior, it completes a mail check MUCH faster than KMail.)

I use both an always-online IMAP account (on my local network) and a disconnected IMAP account (over the Internet), and I experience this behavior with the always-online account but not the disconnected account.
Comment 1 Carsten Burghardt 2006-02-13 19:42:01 UTC
This is not the normal behaviour of kmail. Any chance to get an ethereal log 
of the communication when this happens?
Comment 2 Lincoln Peters 2006-02-14 04:10:40 UTC
I'll post an Ethereal log shortly.

I did think of something that might help to explain this unexpected behavior.  Since I was essentially trying to migrate from an old account that used POP3 to a new account that uses IMAP, I copied a bunch of old messages from my local "inbox" folder to the IMAP "inbox" folder.  These messages are the only ones that don't disappear and reappear when I check my e-mail on the IMAP account.
Comment 3 Lincoln Peters 2006-02-14 04:31:08 UTC
Created attachment 14682 [details]
Ethereal capture of IMAP session (compressed using bzip2)

This file should contain all of the data that went back and forth when I
checked the IMAP account for new mail (there was 1 new message).  It looks like
there may be some extra data resulting from an IMAPS session (apparently I had
forgotten to shut down Thunderbird, but you should be able to tell the sessions
apart because Thunderbird's connection was encrypted whereas KMail's was not).
Comment 4 Juha Tuomala 2006-02-14 18:35:46 UTC
I confirm this. I've very slow and long connection to my server and I can clearly see when messages are downloaded and removed and then put back to inbox. If kmail crashes during this messages can be duplicated.

It's also annoying if you have big attachments and you would like to delete those mails, it first downloads the message before you can try to delete it. But I guess that's a another issue.
Comment 5 Lincoln Peters 2006-02-15 07:42:33 UTC
By any chance, is your IMAP server running Dovecot?  My IMAP server is running Dovecot, and it might explain why this bug doesn't seem to have been reported until now (Dovecot is not a common IMAP server).

Then again, I'm just guessing at possible explanations for lack of any other clues.
Comment 6 Juha Tuomala 2006-02-15 08:02:34 UTC
Yes, I'm running dovecot.
Comment 7 mjh-kde-bugs 2006-02-15 16:03:01 UTC
I've seen the same thing happen sometimes with the novell netware imap server, and if the connection is broken whilst doing the update you can lose the messages. (I've not seen this in a while though, but that could just be my connection being more stable).
Comment 8 Lincoln Peters 2006-02-17 19:52:30 UTC
Maybe then there is something peculiar about the way that Dovecot and the NetWare IMAP server behave that triggers a bug in KMail?

I'll check if the Dovecot developers have noticed anything similar (but not right now, as one of my hard disks failed yesterday).
Comment 9 Lincoln Peters 2006-02-18 06:09:58 UTC
Looks like the closest thing the Dovecot project has to a bug-reporting system is its mailing list.  I can't find any accounts of a bug even remotely resembling this one, but that may be because there are many different ways that this bug could be described (I searched for messages containing the words "KMail" and/or "disappear").

Perhaps I should send a message about this bug to their mailing list?
Comment 10 Juha Tuomala 2006-02-18 08:55:40 UTC
In duplicate bug 122199, there is a "map-uw-2004g_1,1   University of Washington IMAP4rev1/POP2/POP3" server.
Comment 11 Vincenzo Ciancia 2006-05-21 23:45:16 UTC
I have seen the same behavior on my university account. Message are shown, then they disappear, then they appear again. 
Comment 12 paul.novotny 2006-05-26 03:49:16 UTC
I see this on my university imap account too.
Comment 13 Jerzy Kozera 2006-06-30 13:29:52 UTC
The same behavior occurs for me with KMail 3.5.3 and Courier IMAP server 3.0.8 on localhost.

It's especially frustrating when reading long new mails with interval mail checking enabled - when they disappear, the message's content displayed changes to last not-recently-received mail, which becomes selected in message list until I select another message again. There is a workaround for this - opening messages in new windows by double-clicking them.

I can also see one new duplicate message in mutt (configured to read IMAP's Maildir) marked for deletion for each new message after each mail check.

Upgrading Courier IMAP to 4.1.1 didn't help.
Comment 14 kstars 2006-08-10 20:50:56 UTC
I am seeing this as well, using KMail-1.9.4 on KDE-3.5.4.  The imap server is Cyrus-imapd 2.1.16.

As others have described, when checking for new mail, recently-received messages disappear momentarily and then reappear.  The other annoying thing is that when the messages reappear, the listbox does not scroll back down to show them, so I can't see any new messages that may be received until I scroll the listbox.

Comment 15 Paul Berg 2006-10-27 09:49:22 UTC
*** This bug has been confirmed by popular vote. ***
Comment 16 S. Burmeister 2007-01-15 22:52:00 UTC
Still not fixed and this can lead to data-loss (see bug 140010).
Comment 17 tropikhajma 2007-06-09 22:44:44 UTC
still present in kmail 1.9.7
Comment 18 Thomas McGuire 2007-08-12 13:02:07 UTC
*** Bug 122199 has been marked as a duplicate of this bug. ***
Comment 19 Thomas McGuire 2007-08-12 13:07:05 UTC
See also bug 125084, might be the same.
Comment 20 Stephen Ostrow 2008-08-19 01:22:09 UTC
Still present:
Qt: 4.4.0
KDE: 4.1.00 (KDE 4.1.0)
KMail: 1.10.0
Comment 21 Bob Cavey 2008-11-12 17:01:18 UTC
I've noticed this behavior but only in the past month ... and today is really bad as every e-mail after Sept. 18th up until yesterday is missing in my IMAP Inbox in Kmail but in Thunderbird all is OK.   I'm on FC8 talking to an Exchange server via SSL, sorry I can't try non-SSL.  I usually run both because I have to use Entrust PKI and I've only been able to get Thunderbird working with S/Mime.

The problem is very predictable that when I come into office in the morning to read e-mail and all the message data starts the download again.

Sep 29 10:04:11 Updated: 6:kdelibs-3.5.10-1.fc8.i386
Sep 29 10:04:37 Installed: 1:kdeaccessibility-3.5.10-1.fc8.i386
Sep 29 10:04:43 Installed: 6:kdeutils-3.5.10-1.fc8.i386
Sep 29 10:05:56 Installed: 6:kdebase-libs-3.5.10-2.fc8.i386
Sep 29 10:06:07 Updated: 6:kdebase-3.5.10-2.fc8.i386
Sep 29 10:06:10 Installed: 7:kdenetwork-libs-3.5.10-1.fc8.i386
Sep 29 10:06:15 Updated: 6:kdemultimedia-libs-3.5.10-2.fc8.i386
Sep 29 10:06:17 Updated: 7:kdegraphics-libs-3.5.10-1.fc8.i386
Sep 29 10:06:20 Updated: 6:kdepim-libs-3.5.10-1.fc8.i386
Sep 29 10:06:29 Updated: 6:kdepim-3.5.10-1.fc8.i386
Sep 29 10:06:38 Updated: 7:kdegraphics-3.5.10-1.fc8.i386
Sep 29 10:06:45 Updated: 6:kdemultimedia-3.5.10-2.fc8.i386
Sep 29 10:06:50 Updated: kdeartwork-3.5.10-1.fc8.i386
Sep 29 10:07:00 Updated: 7:kdenetwork-3.5.10-1.fc8.i386
Oct 22 08:14:06 Updated: 6:kdepim-libs-3.5.10-2.fc8.i386
Oct 22 08:14:37 Updated: 6:kdepim-3.5.10-2.fc8.i386
Nov 06 08:51:29 Updated: kde-filesystem-4-19.fc8.noarch
Comment 22 George Socker 2008-11-30 01:33:39 UTC
Same problem here - It downloads, then deletes and re-uploads the messages in the mailbox.
KMail 1.10.3 - KDE 4.1.3 compiled on Gentoo.
Remote server is running Cyrus-imapd.
Comment 23 Bob Cavey 2008-12-19 02:18:57 UTC
This problem has now gotten worse since upgrading to Fedora 10. kmail version reports 1.10.3. I hope I don't have to switch over to some other mailer but the pain level is getting too high.  I clicked the arrow(bottom right corner) to show the details and when clicking on a message sometimes it downloads the message three times but sometimes it's eight!!!. I blew away my .kde directory and .kderc just for giggles and started from scratch with no luck at all. This all started back in October.

On average I keep 3500 messages which is about two months worth of e-mail on the server and have local folders with 20 filters that sort messages.  Maybe I'll try Evolution as I think they support maildir format or even claws and hopefully report back if they "don't cut the mustard". Maybe this problem is OBE(Overcome By Events) and also since IMAP users might be the minority might.... who knows??

Again, all my connections are SSL or TLS so a trace is not possible. I tried to run an strace the other day but was too much to look at.
Comment 24 Bob Cavey 2008-12-19 05:16:23 UTC
Wow ... just did a bugzilla search and there are 161 open bugs kmail/kontact imap/dimap version 1.9 through the latest 1.10 release. Looks like it's time to bail out.

Bummer was a good client for over a year now .. guess I get to try out something else now for a while hopefully. Heck I'll probably just bail entirely on KDE in general and check out the haps with enlightenment, xfce or fvwm.
Comment 25 Roman K. 2009-01-05 14:06:01 UTC
I can confirm this bug with kmail 1.10.3 (openSUSE 11.1). It first shows the messages, then they diappear, and after a (long) while they appear again and I can read my messages.
Comment 26 Roman K. 2009-01-05 15:39:11 UTC
I think there is a problem with the spam-filter-settings. I deactivated all spam-filters for all imap-accounts and now it seems that the problem is gone. But if I want to filter imap-accounts for spam, it is very unusable. Can somebody confirm that the problem is gone with a deactivated the spam-filter for imap-accounts?
Comment 27 Björn Ruberg 2009-12-25 00:52:36 UTC
Those filter problems in conjunction with IMAP are fixed in KDE 4.3. I close this. Please report if you still have problems in the recent kmail versions.