Bug 85013 - folder overview titles get mixed up
Summary: folder overview titles get mixed up
Status: RESOLVED REMIND
Alias: None
Product: kmail
Classification: Applications
Component: IMAP (show other bugs)
Version: 1.6.2
Platform: unspecified Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords: triaged
: 142920 (view as bug list)
Depends on:
Blocks:
 
Reported: 2004-07-12 16:49 UTC by Wiebe Cazemier
Modified: 2008-11-07 15:59 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Screenshot with correct information in messagelist. (28.62 KB, image/png)
2004-07-12 21:22 UTC, Wiebe Cazemier
Details
screenshot with wrong title in messagelist (24.02 KB, image/png)
2004-07-12 21:24 UTC, Wiebe Cazemier
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Wiebe Cazemier 2004-07-12 16:49:50 UTC
Version:           1.6.2 (using KDE 3.2.2, Gentoo)
Compiler:          gcc version 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)
OS:                Linux (i686) release 2.4.26

On my IMAP account (IMAP server courier-imap 3.0.2 resides on localhost), the titles in the folder overview (frame which shows all message's subjects, senders, etc) are getting mixed up. Header-information of mails from folder A are used to generate a title of mails in folder B. When I reselect the message with the wrong title, the right title is displayed. Unselecting and selecting it again results in the wrong title again. The header in the message-pane always shows the correct data.

I've tried removing all the indices in .kde/share/apps/kmail several times. Always the same messages are affected, except when re-generating kmail's indices, then other messages are affected.
Comment 1 Wiebe Cazemier 2004-07-12 18:58:29 UTC
By folder overview, I meant message list. The titles in the message list are getting mixed up.
Comment 2 Carsten Burghardt 2004-07-12 20:11:02 UTC
Please start ethereal and provide and log when you click on those emails that 
do not get displayed correctly. Apart from that please make a screenshot from 
that case.

Comment 3 Wiebe Cazemier 2004-07-12 21:22:42 UTC
Created attachment 6643 [details]
Screenshot with correct information in messagelist.
Comment 4 Wiebe Cazemier 2004-07-12 21:24:08 UTC
Created attachment 6644 [details]
screenshot with wrong title in messagelist
Comment 5 Wiebe Cazemier 2004-07-12 21:41:44 UTC
I hope you can forgive the aristic expressions in the screenshot to protect some privacy information :) I left the initials to be seen.

As you can see, the same mail is shown, but in one screenshot with correct information about the mail in the messagelist, and in the other screenshot with wrong information. When wrong information is displayed, unfocusing and focussing the window is enough to update the messagelist to the correct information.

About the ethereal log, is it sufficient to know that no faulty information travels over the socket? The data that should not be displayed in the messagelist of folder B does not travel over the socket. It's gonna take some time to remove privacy information from that log, you see.

I've found out a few new things:
For one, I said that the message-pane itself always displayed the correct mail. This is not the case. I can't really reproduce it, but it seems that when I select the mail from which the faulty information is coming, that that mail is displayed when I select the second folder.

And the second thing: When I remove the indices of kmail, I let kmail recreate them by clicking all folders from top to bottom. Now, it seems that when there is an error, only erroneous data from the previous folder is displayed in the current messagelist, not from other folders. It seems that Kmail makes a few mistakes when generating indices, perhaps not clearing them in memory when a folder is done or something like that. If that's the case, then it's probably not IMAP-related, is it?

I checked the index file in ~/.kde3.2/share/apps/kmail/imap/.localhost.directory/.INBOX.directory/[foldername]. It contains the erroneous data I see in the messagelist in the first entry.
Comment 6 Carsten Burghardt 2004-07-13 07:18:11 UTC
It's probably not imap related. Please upgrade to 3.2.3 if possible (or even better to kdepim 3.3), delete the account and create it again. Then this should be gone.
Comment 7 Wiebe Cazemier 2004-10-09 17:06:11 UTC
I finally upgraded to kde-3.3 (including kdepim 3.3). I deleted my old .kde dir. The bug remains in kmail 1.7.
Comment 8 Bernd Schubert 2004-12-23 13:59:54 UTC
Same problem here, IMHO it also shows subjects of mails that have been already fetched via pop3, so are not existing any more on the IMAP server.

Also just upgraded to kdepim 3.3.2 and recreated my IMAP account. I will report later on, when I got new mails, if it is fixed.

Bernd
Comment 9 Bernd Schubert 2004-12-24 01:29:44 UTC
Hmm, interesting just noticed that the OP (Wiebe) already had this bug with kde-3.2. I definitely know that this first occured with kde-3.3.0.

Stupid me, I accidentely fetched the mails I got today via pop3, so I still don't know if it is gone with kdepim-3.3.2. And the next 9 days I can't check.

Bernd
Comment 10 Bernd Schubert 2005-01-03 23:52:16 UTC
So, now I definitely know that the problem still exits, so what now? How can I help to debug this problem? Shall I capture the imap connection with ethereal?

Bernd
Comment 11 Bernd Schubert 2005-01-03 23:52:58 UTC
So, now I definitely know that the problem still exits, so what now? How can I help to debug this problem? Shall I capture the imap connection with ethereal?

Bernd
Comment 12 Bernd Schubert 2005-01-10 18:39:00 UTC
So, in the mean time I think this is a kmail-caching problem. Deleting the
$HOME/.kde/share/apps/kmail/imap/.{arb. number}.directory/INBOX file before starting kmail fixes it for this session.
Just a guess/theory what happens:

1.) Some mails are read via IMAP, those (all?) IMAP mail headers are cached in this INBOX file.

2.) The mail are fetched via pop3.

3.) The IMAP-folder is checked again, but kmail doesn't detect, that the mails being now in this folder differ from those cached in the INBOX-file.

4.) Mail subjects shown in the IMAP-folder are taken from the cache of the INBOX-file, only when clicking on the mail the subject is taken from the network.
Are you still with me? ;-)


So, I so far havn't looked into the source code and I'm not (yet) familiar with C++, only with C. If there will be no patches from someone else until the weekend, I will look into the source. 
However,  I would prefer if someone would at least point me to the files/functions I should look into.

Bernd 
Comment 13 Carsten Burghardt 2005-01-10 18:56:12 UTC
> 1.) Some mails are read via IMAP, those (all?) IMAP mail headers are cached
> in this INBOX file.

Correct.

> 2.) The mail are fetched via pop3.

???
Are you fetching the same mails via pop and imap? Why?

> 3.) The IMAP-folder is checked again, but kmail doesn't detect, that the
> mails being now in this folder differ from those cached in the INBOX-file.

Why should they differ?
Bailing out :-(

Carsten

Comment 14 Bernd Schubert 2005-01-10 19:23:14 UTC
referring to 2.) 

>Are you fetching the same mails via pop and imap? Why?

I'm subscribed to rather many mailinglists, to get an overview over all my mails, I usually first check the headers via IMAP and sometimes also read the some of them.
At least ones a day I fetch the mails via pop3 and let them sort into the local folders (currently 39).
I can give further explanations, but the way I'm reading my mails shouldn't matter.

referring to 3.)

>Why should they differ?

O.k., more in detail: 
After fetching the mails via pop3 the IMAP-INBOX cache file should become updated, right? It now should be empty as there are no more mails in the folder. Unfortunately it isn't. After some time new mails go to my account, but the IMAP INBOX cache file still holds the old entries and it just doesn't update itself. 

I really know nothing about the IMAP protocol itself, but I somehow think the there is no working (or none at all) cache file validation mechanism. So I guess kmail assumes that e.g. the first entry in the cache file, is also the first entry in the network IMAP folder. 
Maybe the cache file gets updated, when only working with IMAP, so when one directly deletes/moves/etc. its mail via IMAP, but I'm not doing it this way.

Is it clear now?

Bernd
Comment 15 Carsten Burghardt 2005-01-10 19:54:50 UTC
> O.k., more in detail:
> After fetching the mails via pop3 the IMAP-INBOX cache file should become
> updated, right? It now should be empty as there are no more mails in the
> folder. 

When you click again on your imap inbox after you fetched your mails with pop.

> Unfortunately it isn't. After some time new mails go to my account, 
> but the IMAP INBOX cache file still holds the old entries and it just
> doesn't update itself.

Are you still using kmail 3.2? Then you should upgrade to kmail from kde 3.3.


Carsten<html><head><meta name="qrichtext" content="1" /></head><body style="font-size:12pt;font-family:Arial">
<p>&gt; O.k., more in detail:</p>
<p>&gt; After fetching the mails via pop3 the IMAP-INBOX cache file should become</p>
<p>&gt; updated, right? It now should be empty as there are no more mails in the</p>
<p>&gt; folder. </p>
<p></p>
<p>When you click again on your imap inbox after you fetched your mails with pop.</p>
<p></p>
<p>&gt; Unfortunately it isn't. After some time new mails go to my account, </p>
<p>&gt; but the IMAP INBOX cache file still holds the old entries and it just</p>
<p>&gt; doesn't update itself.</p>
<p></p>
<p>Are you still using kmail 3.2? Then you should upgrade to kmail from kde 3.3.</p>
<p></p>
<p></p>
<p>Carsten</p>
</body></html>
Comment 16 Bernd Schubert 2005-01-10 21:25:49 UTC
>When you click again on your imap inbox after you fetched your mails with pop.

Nope, doesn't help, the cache is not updated.

>Are you still using kmail 3.2? Then you should upgrade to kmail from kde 3.3.

No, already have hole kdepim-3.3.2.
Comment 17 Bernd Schubert 2005-01-10 21:29:04 UTC
>>When you click again on your imap inbox after you fetched your mails with pop. 

>Nope, doesn't help, the cache is not updated.

I mean this $HOME/.kde/share/apps/kmail/imap/.{arb. number}.directory/INBOX file

Even closing and re-opening kmail doesn't help, the old entries just won't be deleted from this file.
Comment 18 Ingo Klöcker 2005-01-11 00:56:53 UTC
Bernd, do you probably know what kind of IMAP server you are using? Is it probably web.de? Usually each message on the IMAP server has a unique id (uid). If a message is deleted (or fetched via POP3) then KMail will notice that there no longer is a message with a certain uid. Of course, this only works if the server plays nice. But if the server reuses the uids then KMail has no way to know that this is a new message. In order to find the problem we either need access to the IMAP server or we need a few logs of the communication between KMail and the server. Carsten? What about the index file? Would it be helpful to have a look at it (with a hex editor or with a to-be-written tool)?
Comment 19 Bernd Schubert 2005-01-11 01:45:14 UTC
> Bernd, do you probably know what kind of IMAP server you are using? Is it
> probably web.de? Usually each message on the IMAP server has a unique id

Actually the problematic server is imap.gmx.net, I'm also using (not so often) 
web.de and never had a problem with this server.

> (uid). If a message is deleted (or fetched via POP3) then KMail will notice
> that there no longer is a message with a certain uid. Of course, this only

O.k., this explains why it works with web.de. Can you tell me where this those 
uid's are stored in? The *.index.ids files?

> works if the server plays nice. But if the server reuses the uids then
> KMail has no way to know that this is a new message. In order to find the
> problem we either need access to the IMAP server or we need a few logs of

What kind of access do you need? I wouldn't like to post my password in the 
public, though we could also encrypt our mails. But maybe I can do all this 
stuff myself? Capturing the transfer via ethereal really shouldn't be a 
problem.

I could also complain at gmx about their problematic IMAP server, but I guess 
there are problems with others servers as well. So from my point of view 
there should be at least the possibility to enable a more reliable way.

Thanks all for your help,
 Bernd

Comment 20 Carsten Burghardt 2005-01-11 11:57:08 UTC
Bernd Schubert sagte:
> O.k., this explains why it works with web.de. Can you tell me where this
> those
> uid's are stored in? The *.index.ids files?

Directly in the .index file but that won't help.

>> works if the server plays nice. But if the server reuses the uids then
>> KMail has no way to know that this is a new message. In order to find
>> the
>> problem we either need access to the IMAP server or we need a few logs
>> of
>
> What kind of access do you need? I wouldn't like to post my password in
> the
> public, though we could also encrypt our mails. But maybe I can do all
> this
> stuff myself? Capturing the transfer via ethereal really shouldn't be a
> problem.

An ethereal log would be good indeed. Start the capture before you click
on the folder.


Comment 21 Wiebe Cazemier 2005-01-11 13:56:18 UTC
As the reporter, I guess I should mention I use courier-imap 3.0.8 and kmail 1.7.2, from kde 3.3.2. In my first post you can see the versions I had at the time.

When I delete all the index-files in .kde/share/apps/kmail/, kmail has to regenrate the indices, obviously, so I click each folder in turn to see the messages. After a number of folders, the first message of a folder often displays as one of the previous folder in the message list, when selecting it.

I thought i'd mention this again, because I don't know if it's the same problem Bernd Schubert is experiencing, because I don't do anything with POP.
Comment 22 Bernd Schubert 2005-01-12 02:05:34 UTC
On Tuesday 11 January 2005 11:57, Carsten Burghardt wrote:
> Bernd Schubert sagte:
> > O.k., this explains why it works with web.de. Can you tell me where this
> > those
> > uid's are stored in? The *.index.ids files?
>
> Directly in the .index file but that won't help.

Is there anywhere a description of the file format of the .index file?

>
> An ethereal log would be good indeed. Start the capture before you click
> on the folder.

O.k., here is the download link:

http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/kmail/gmx_imap_capture.ethereal


Cheers,
 Bernd

Comment 23 Carsten Burghardt 2005-01-12 09:15:44 UTC
>> Directly in the .index file but that won't help.
>
> Is there anywhere a description of the file format of the .index file?

http://webcvs.kde.org/kdepim/kmail/DESIGN?rev=1.19&view=markup

>> An ethereal log would be good indeed. Start the capture before you click
>> on the folder.
>
> O.k., here is the download link:

Will check.

Comment 24 Carsten Burghardt 2005-01-13 11:05:07 UTC
Can you explain what you did (and see) to get this ethereal log?


Carsten

Comment 25 Bernd Schubert 2005-01-13 13:45:51 UTC
>
> ------- Additional Comments From burghardt kde org  2005-01-13 11:05
> ------- Can you explain what you did (and see) to get this ethereal log?
>

I opened kmail, started the ethereal capture and clicked on the gmx-imap inbox 
folder. Unfortunately I can't remember any more what I have seen then ;)

So I just uploaded a second capture 
( http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/kmail/ )

This time its a full capture, so I started capturing before opening kmail. 
After opening kmail I clicked on the gmx-imap inbox folder. 
This time I know for sure that then the active message in the message list has 
a wrong subject. The subject of the active message in the message-list is 
"[reiserfs-list] Severe problem ...", but the message shown in the preview 
window has the subject "[DRBD-user] The need for Speed 2".

Bernd


PS: Don't worry about the password, for the period of capturing I changed it 
to this insecure phrase, and already switched back to the previous one. I'm 
usually also using encrypted transfers.


Comment 26 Carsten Burghardt 2005-01-13 22:32:43 UTC
On Thursday 13 January 2005 13:45, Bernd Schubert wrote:
> This time I know for sure that then the active message in the message list
> has a wrong subject. 

The requests look normal. There are 7 emails in your inbox and kmail simply 
requests the last shown email with the uid 2. As this changed the server 
obviously reuses the uids when you delete the old mails. This is only ok when 
it announces this. Please compile kmail from 3_3_BRANCH or HEAD yourself and 
use --enable-debug=full. Then start kmail from the console and see it you get 
a message "uidValidty changed".

Comment 27 Bernd Schubert 2005-01-20 00:21:35 UTC
Sorry for my late response, really didn't have the time and also sorry for my dumb questions, do you know if the following is o.k.?

apt-get source kdepim
cd kdepim-3.3.1
cvs updates  (it updates a lot)

Which version will cvs update to? 3_3_BRANCH or HEAD? This cvs branch stuff always confuses me. For my project I have switched to subversion therefor. ;-)

The adjustments of the of the debian/kde.mk is easy to enable full debuggin.

Thanks,
   Bernd
Comment 28 David Faure 2005-01-20 00:25:29 UTC
That should be KDE_3_3_BRANCH. 
You can check with "cvscheck" from kdesdk (in a subdirectory, to get a faster result)
or cvs status <filename>.

Comment 29 Bernd Schubert 2005-01-30 01:56:56 UTC
So, finally I got these kmail packages build (forgot to run 'apt-get build-dep kdepim), got no compiling errors, but only errors on packaging the debs and first didn't know whats the reason for it, stupid me.

Well, I just checked, kmail is reporting rather many messages now, but there's nothing about  "uidValidty changed", though the subject was definitely changing for some messages. Log file is here:

http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/kmail/kmail-debug-messages.txt

Should I also install other kdepim components with enabled debug messages.

Cheers,
   Bernd

PS: Thanks David, its the 3_3 branch.
Comment 30 Carsten Burghardt 2005-01-30 16:59:58 UTC
Not very good as this would mean that the gmx server reuses uids without
telling the client.
You can do the following to check this. Telnet to your server on port 143
and send this:

1 login <login> <password>
2 select INBOX
3 uid fetch 1:* (envelope)
4 logout

The output from the second and the third command are interesting. Do this
with new mails in your inbox and after that a second time when the mails
changed. Please note that your password is sent unencrypted in this case
and it's probably better to change it afterwards.


Comment 31 dag 2006-07-05 21:01:46 UTC
Hi,

I am using 3.5.3 and am still seeing problems with the IMAP cache. As I have a MH mailbox format and am using other mailclients to read the mails now and then packing the MH mailbox will change the UID of the mail:s.  This confuses kmail as the old headers for the mails are cached. I can understand and live with that if I only found a way to refresh the cache. Pressing the "refresh IMAP cache doesn't seem to change anytthing :-( .
In earlier kmail versions clicking on the mail and getting the new headers seemed to fix the cache for that mail at least, but in this newest version even that doesn't change anythomg anymore...
Comment 32 Thomas McGuire 2007-03-13 20:27:30 UTC
*** Bug 142920 has been marked as a duplicate of this bug. ***
Comment 33 Tristan Miller 2007-09-02 15:24:25 UTC
Confirming bug still occurs with KDE 3.5.7 (KMail 1.9.6).
Comment 34 Michael Leupold 2008-08-31 12:17:19 UTC
It seems pretty hard to reproduce this bug without access to a certain IMAP server. Could you please recheck if this bug still occurs with a recent KMail (eg. 1.10.0 from KDE 4.1)?
Comment 35 Michael Leupold 2008-11-07 15:59:40 UTC
Closing due to no feedback. Please reopen if the bug still exists.