Bug 37898 - Compact folders : Still enlarging after delete
Summary: Compact folders : Still enlarging after delete
Status: RESOLVED INTENTIONAL
Alias: None
Product: kmail
Classification: Applications
Component: general (show other bugs)
Version: 1.3.2
Platform: Mandrake RPMs Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords: triaged
: 83525 (view as bug list)
Depends on:
Blocks:
 
Reported: 2002-02-09 05:18 UTC by ripleyinfo
Modified: 2012-01-08 22:58 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description ripleyinfo 2002-02-09 05:15:31 UTC
(*** This bug was imported into bugs.kde.org ***)

Package:           kmail
Version:           1.3.2 (using KDE 2.2.2 )
Severity:          normal
Installed from:    Linux-Mandrake 8.0 i586 - Traktopel
Compiler:          gcc version 2.96 20000731 (Mandrake Linux 8.0 2.96-0.64mdk)
OS:                Linux (i686) release 2.4.16
OS/Compiler notes: 

When I "Compact All Folders" on kmail 1.3.2
I still got a folder that doesn't stop growing.
I only have 30 messages (witout attachment)
in kmail and my file un ~/Mail is 86Mb!

I tried the "Consistent=true" trick but it
doesn't work.

(Submitted via bugs.kde.org)
(Called from KBugReport dialog)
Comment 1 Ingo Kl 2002-02-09 10:00:21 UTC
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 09 February 2002 06:15 ripleyinfo@iquebec.com wrote:
> When I "Compact All Folders" on kmail 1.3.2
> I still got a folder that doesn't stop growing.
> I only have 30 messages (witout attachment)
> in kmail and my file un ~/Mail is 86Mb!
>
> I tried the "Consistent=3Dtrue" trick but it
> doesn't work.

The following will work:
1. Create a new folder
2. Move all messages to this new folder
3. Empty the old (and now empty) folder
4. Move the messages back

Regards
Ingo
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE8ZPM1GnR+RTDgudgRAhMXAJsE3ds3dSYj/qU1lsDXRJvfmHABtQCgvyGq
iS6BEveDu5PrvNkWW4oZsYk=3D
=3Da3Jh
-----END PGP SIGNATURE-----
Comment 2 Roger Larsson 2003-11-28 22:24:43 UTC
I can not see that problem (and related bug 37898) is FIXED
- problem is still there.
Described above is a workaround, the user should atleast
get a hint that there is a problem.

I tried to do a backup of my home directory but it did
not fit... but by using FSView (what a great tool!)
I found out that I had a 619 MB(!) linux-kernel folder.
I deleted all messages (with expire) folder is now empty,
compacted...
FSView still showed 619 MB... what???

Searched bugs.kde.org and found this closed bug...

I understand that it is dangerous to automatically
compact folders that might be corrupt. But the user
should at least get an indication of the problem!

Either dim/disable the folder menu item "Compact"
[why show the item when it can not be used?]
or rename it to something like "Forced Compact",
"Dangerous Compact", "Compact (possibly dangerous)"
Comment 3 David Faure 2004-05-26 13:58:55 UTC
There is some strange code that disables compacting on a folder (for ever!)
after a problem was found.... It seems to trigger quite easily... I'm currently debugging it.
Comment 4 David Faure 2004-07-05 14:58:56 UTC
*** Bug 83525 has been marked as a duplicate of this bug. ***
Comment 5 Rudolf Lohner 2004-08-17 14:22:55 UTC
I have the same problem and after some experimenting I found out that it seems to be a configuration issue:

In ~/.kde/share/config/kmailrc there is an option 'Compactable=true/false' with each mail folder and it is set to 'false' on all folders which cannot be compacted!

Just setting this option to true seems to work. At least the folder can be compacted again. I don't know if it has any negative effects since I have no idea when and how these entries have been switched from true to false or if this might happen again by some miraculous cirumstances.

Can someone confirm that this problem is only caused by the setting of this configuration switch and no deeper (more dangerous) reason?
Comment 6 hn75 2004-09-08 16:39:17 UTC
I can confirm that the proposed resolution from Comment #5 solved my problems as well. I never set Compactable=false by hand an could find a place to set it in the GUI.

Hendrik
Comment 7 Don Sanders 2004-09-09 09:43:17 UTC
On Tuesday 17 August 2004 22:23, Rudolf Lohner wrote:
...
> Can someone confirm that this problem is only caused by the setting
> of this configuration switch and no deeper (more dangerous) reason?

The configuration switch (compaction=false) is activated when KMail 
detects an inconsistency between an index file and the corresponding 
folder file(s). This is because when the index file is inconsistent 
performing a compaction may result in mail loss. That is it's a 
safety feature to prevent people losing mail.

The inconvenient but safe way to get around the problem of a folder 
not compacting is to rename the problematic folder, create a new 
folder with the same name as the old folder, and copy all mail from 
the old folder into the new folder.

Don.

Comment 8 steve 2004-09-09 10:39:57 UTC
Surely a better solution would be to rebuild the index and then compact the folder (using the previous index to match as many of the original messages as possible and hence keep their state information)?

I'm not sure why you think the index should be king? Surely the mail folder itself is the definitive reference and the index is merely a convenience? If you want to make sure you tally the correct message with the correct index entry, why not store something like an MD5 sum of the headers for each message in the index?

It shouldn't be up to the user to diddle with an obscure config file in any way.

If in doubt, bring up a warning dialog and ask the user if they want to recover the possibly corrupt folder and then compact. The current situation is almost, but not quite, the worst possible. (The worst would be blindly using the index to munge the folder.)

If anything, the current kmail local mail folder system is one of the least robust I've come across. How is it that programs such as Pine and Elm can work fine with programs delivering into folders in the mail directories and handling live inboxes yet such an advanced program as kmail can't? If only I had the C++ skills I'd see if I could add this sort of thing to kmail, but I don't.
Comment 9 Don Sanders 2004-09-09 11:59:15 UTC
> ------- Additional Comments From steve earth ox ac uk  2004-09-09
> 10:39 ------- Surely a better solution would be to rebuild the
> index and then compact the folder (using the previous index to
> match as many of the original messages as possible and hence keep
> their state information)?

Yes, I think a patch for that would be nice. Maybe it is already 
implemented I remember it being discussed.

> I'm not sure why you think the index should be king?

I don't.

> Surely the 
> mail folder itself is the definitive reference and the index is
> merely a convenience? If you want to make sure you tally the
> correct message with the correct index entry, why not store
> something like an MD5 sum of the headers for each message in the
> index?
>
> It shouldn't be up to the user to diddle with an obscure config
> file in any way.

Ideally Index files shouldn't be corrupted in the first place. 
Historically effort has been focused on not corrupting index files.

> If in doubt, bring up a warning dialog and ask the user if they
> want to recover the possibly corrupt folder and then compact. The
> current situation is almost, but not quite, the worst possible.
> (The worst would be blindly using the index to munge the folder.)
>
> If anything, the current kmail local mail folder system is one of
> the least robust I've come across. How is it that programs such as
> Pine and Elm can work fine with programs delivering into folders in
> the mail directories and handling live inboxes yet such an advanced
> program as kmail can't? If only I had the C++ skills I'd see if I
> could add this sort of thing to kmail, but I don't.

The traditional answer given in reply to this question is that Pine 
and Elm are old style mail clients and don't have index files.

Don.

Comment 10 Rigo Wenning 2004-09-09 12:22:47 UTC
The issue comes if using Mutt from time to time. As KMail has index files being the priority. Mutt changes the mailbox, but does not touch the index. This creates a mess in KMail. Additionally, as I learned now, it messes up the KMail config file. This makes it impossible to use one's email with Mutt and KMail at the same time, which would be my preferred setup. I think there is coordination needed. One solution would be to have the mbox-file or maildir-box being the authoritative source and treat the index as a convenience. It would mean regenerating the index from the mbox or maildir-box as found there in case of inconsistencies. (This is already done) and abandon the paradigm of having the index prevail over the actual mailbox. I hope you could fix that in the future. It would make my life easier. The only thing that prevents me from switching to evolution is the very nice integration of KMail into Kontact. (Though creating only a futile URI when dragging emails to korganizer, hope for a fix here too: we use MID internally in our email-database)

Thanks for all the discussion here

Rigo Wenning, Staff Counsel, W3C
Comment 11 steve 2004-09-09 13:45:20 UTC
> The traditional answer given in reply to this question is that Pine
> and Elm are old style mail clients and don't have index files.
>
> Don.

Which then begs the question: "Are index files actually a backward step?"

The way many mail programs deal with live folders is by adding their own header which holds the message's state information and build themselves a dynamic index internally on the fly rather than having a static one on disk. Also, they work on a copy of the mailbox rather than the primary copy, checking for changes and updating their copy as necessary. When they try to commit changes they check the working copy against the original before committing the changes to the original.

Is there a major structural reason why Kmail couldn't do something similar? It would then allow for multiple programs to access and use the mail folders (but not at the same time) and have programs such as procmail delivering into multiple drop folders without problem.

Being able to use the inbox live from /var/mail or /var/spool/mail would also have the added benefit of being able to access the same mailbox remotely using IMAP as Kmail uses. This may not be an issue for those people using simple, sinlge user Linux boxes but in a large multi-user, single server, multiple client system with people often needing to access their mail remotely via IMAP or telnet access it would help a great deal.

We are trying to make kmail into an enterprise capable mail client and not just a toy for hackers, aren't we?
Comment 12 Don Sanders 2004-09-10 07:40:48 UTC
On Thursday 09 September 2004 21:45, steve@earth.ox.ac.uk wrote:
...
> Which then begs the question: "Are index files actually a backward
> step?"

From an interoperability perspective yes. I think this is why it's 
common for mail clients that use index files (netscape, evolution) to 
store their mail outside of the ~/Mail directory.

> The way many mail programs deal with live folders is by adding
> their own header which holds the message's state information and
> build themselves a dynamic index internally on the fly rather than
> having a static one on disk. Also, they work on a copy of the
> mailbox rather than the primary copy, checking for changes and
> updating their copy as necessary. When they try to commit changes
> they check the working copy against the original before committing
> the changes to the original.

Yes this is the old style.

> Is there a major structural reason why Kmail couldn't do something
> similar? 

There is a reason why modern mail clients don't work that way but it 
isn't structural. It's for scalability. I'm really impressed by how 
well mutt handles medium sized mail folders, it's one of the best if 
not the best (old style mail client) at this. But IIRC its handling 
of large folders (over 100,000 messages) is completely unacceptable.

My kde-cvs folder is nearly a gig in size now, it's just not possible 
to read, let alone parse a file of that size sufficiently quickly 
that a user won't notice. I don't believe mutt can handle a folder of 
this size in an acceptable manner.

Even for KMail with index files the slowness of handling large mail 
box files is a problem and very large mail folders (over 1,000,000 
messages per folder) just aren't handled well. There are a 
significant amount of people who complain about this scalability 
issue, more than about the interoperability issue I think.

Beyond scalability the other benefits from using index files are 
superior message searching and tracking abilities. For instance it 
makes search folders possible and allows for instance a message to be 
dragged from KMail into KOrganizer, and a reference from KOrganizer 
back to the message in KMail to be created. This is because index 
files allows a message identifier to message location on disk mapping 
to be created and maintained persistently (e.g. after the application 
has exited).

> It would then allow for multiple programs to access and 
> use the mail folders (but not at the same time) and have programs
> such as procmail delivering into multiple drop folders without
> problem.

You can use KMail with procmail without problems but it's important to 
follow the instructions in the KMail FAQ.

> Being able to use the inbox live from /var/mail or /var/spool/mail
> would also have the added benefit of being able to access the same
> mailbox remotely using IMAP as Kmail uses. This may not be an issue
> for those people using simple, sinlge user Linux boxes but in a
> large multi-user, single server, multiple client system with people
> often needing to access their mail remotely via IMAP or telnet
> access it would help a great deal.

I don't see your point. KMail supports IMAP, so "people often needing 
to access their mail remotely via IMAP" can use KMail for that. The 
IMAP support in KMail is constant improving. There's even 
disconnected support now so that laptop users can sync their mail at 
the office then use their laptop on a train or plane where no 
affordable net access is present.

> We are trying to make kmail into an enterprise capable mail client
> and not just a toy for hackers, aren't we?

The problem is the old style design is inadequate. It was perfectly 
adequate at the time it was created because mail volumes were 
comparatively low. But by todays standards it's a toy design that 
isn't scalable and doesn't provide message tracking/searching 
functionality that is expected in modern enterprise mail systems.

However it may be possible to use index files without losing 
interoperability. I was surprised to learn Evolution supports this, 
but wasn't so surprised to see several people on usenet complaining 
of losing mail after having tried this feature.

Still for maildir folders at least I think theoretically it should be 
possible to make a scalable interoperable solution that is as 
reliable as the underlying file system. Nevertheless there's a 
significant amount of work involved, and I only see a moderate amount 
of demand for this.

Don.

Comment 13 Don Sanders 2004-09-10 08:35:26 UTC
On Thursday 09 September 2004 20:22, Rigo Wenning wrote:
...
> The issue comes if using Mutt from time to time. As KMail
> has index files being the priority. Mutt changes the mailbox, but
> does not touch the index. This creates a mess in KMail.

This issue comes up on the KMail development mailing list from time to 
time also. Especially mutt is problematic as it has the unusual 
characteristic of changing mail files while forcing the modified time 
of files to remain the same.

> Additionally, as I learned now, it messes up the KMail config file.
> This makes it impossible to use one's email with Mutt and KMail at
> the same time, which would be my preferred setup.

(It's possible using IMAP)

> I think there is 
> coordination needed. One solution would be to have the mbox-file or
> maildir-box being the authoritative source and treat the index as a
> convenience. It would mean regenerating the index from the mbox or
> maildir-box as found there in case of inconsistencies. (This is
> already done) and abandon the paradigm of having the index prevail
> over the actual mailbox. I hope you could fix that in the future.
> It would make my life easier. 

I see two problems that need to solved to make this a reality.

The first relates to status information. Currently status information 
is stored in the index files and the mail message on disk isn't 
updated. Making the status information in the index files redundant 
would require updating the mail message on disk. I think this is 
possible to do this safely and quickly for maildir but not mbox.

The second problem is preserving the integrity of the internal KMail 
message identifier to mail message on disk location map. Again I 
think it should be possible to do this safely and quickly for maildir 
but not mbox.

> The only thing that prevents me from 
> switching to evolution is the very nice integration of KMail into
> Kontact. (Though creating only a futile URI when dragging emails to
> korganizer, hope for a fix here too: we use MID internally in our
> email-database)

Sorry I don't really understand what is meant here.

> Thanks for all the discussion here
>
> Rigo Wenning, Staff Counsel, W3C

In the case that the limitations discussed are proving a financial 
burden for individuals or organizations then I do provide an open and 
affordable commercial service for alleviating dissatisfaction.

http://kontact.org/shopping/
http://kontact.org/shopping/sanders.php

But for the existing individuals/organizations I serve this hasn't 
been raised as an issue so currently I don't think it makes sense for 
me to prioritize this issue.

Don.

Comment 14 Rigo Wenning 2004-09-10 11:14:00 UTC
Am Friday 10 September 2004 08:35 verlautbarte Don Sanders :
> > The issue comes if using Mutt from time to time. 
>
> Especially mutt is problematic as it has the unusual characteristic 
> of changing mail files while forcing the modified time  of files to 
> remain the same. 

I contacted some Mutt developer.
>
> > This makes it impossible to use one's email with Mutt and KMail at
> > the same time, which would be my preferred setup.
>
> (It's possible using IMAP)

I don't use IMAP for privacy reasons as I don't want to keep the email 
on the server. It's for me too much overhead to install IMAP on my 
laptop and maintain it there. But this is _my_ convenience.. ;)
>
> > I think there is coordination needed. 
>
> I see two problems that need to solved to make this a reality.
>
> The first relates to status information. Currently status information
> is stored in the index files and the mail message on disk isn't
> updated. Making the status information in the index files redundant
> would require updating the mail message on disk. I think this is
> possible to do this safely and quickly for maildir but not mbox.

Yes, this is definitely a good idea.
>
> The second problem is preserving the integrity of the internal KMail
> message identifier to mail message on disk location map. Again I
> think it should be possible to do this safely and quickly for maildir
> but not mbox.

The issue here is that you make up an identifier instead of using the 
existing one: Message ID is supposed to be unique. In a URI format, 
this would read like this: mid:20040910054053.18966.qmail@ktown.kde.org
instead of a hash. This way you even can't loose your identifier and it 
might even be used beyond KMail. I mean, this is about 
Web-integration ;)
>
> > Though creating only a futile URI when dragging emails to
> > korganizer, hope for a fix here too: we use MID internally in our
> > email-database)
>
> Sorry I don't really understand what is meant here.

I just tested that in KMail 1.7 and the feature seems to work now, but 
still has some trouble (point me to the right bug to continue this 
discussion ;) If you drag an email to the to-do list, this creates a 
new to-do with an attachment. In KMail 1.6, clicking on the identifier 
of the email in the to-do had no effect. In KMail 1.7, it now points 
the focus of KMail to that email. But it is still impossible to 
double-click the reference in the to-do window to open the message in 
its own window. So the reaction of all the people I tested was: You 
click and nothing happens ;)

Secondly, the sort of reference identifier you use (I assume a hash) 
works for what you want to achieve, but will bring you serious 
limitation in the future. 

1/ The identifier does follow the notation of RFC 2396 but could be 
expressed in a better way saying mid:4711some@example.org

2/ By using the hash instead of the mid, you create yet another 
identifier for you only instead of re-using the one already present 
(MID) used by everybody else. This limits your ability to link to the 
outside world and to be linked from the outside world.

Note that mid is already registered as identifier with IANA in RFC 2392, 
so why not use it? 

> In the case that the limitations discussed are proving a financial
> burden for individuals or organizations then I do provide an open and
> affordable commercial service for alleviating dissatisfaction.

Looks like a very nice business model. I will think about it personally.
>
> http://kontact.org/shopping/
> http://kontact.org/shopping/sanders.php
>
> But for the existing individuals/organizations I serve this hasn't
> been raised as an issue so currently I don't think it makes sense for
> me to prioritize this issue.

This statement is a MUST after the introduction above ;)

Best, 

Rigo

Comment 15 Stephen Usher 2004-09-10 12:02:19 UTC
On Friday 10 September 2004 6:40 am, Don Sanders wrote:
> > Which then begs the question: "Are index files actually a backward
> > step?"
> 
> >From an interoperability perspective yes. I think this is why it's 
> common for mail clients that use index files (netscape, evolution) to 
> store their mail outside of the ~/Mail directory.

Indeed.. Maybe it would be an excellent idea if Kmail did the same? After all, 
if these are supposed to be private files used only by kmail surely they 
shouldn't be placed in a directory name with is so commonly used by "legacy" 
mail programs and IMAP servers. (I'll get onto the problems with the IMAP 
system a bit later.)
 
> > Is there a major structural reason why Kmail couldn't do something
> > similar? 
> 
> There is a reason why modern mail clients don't work that way but it 
> isn't structural. It's for scalability. I'm really impressed by how 
> well mutt handles medium sized mail folders, it's one of the best if 
> not the best (old style mail client) at this. But IIRC its handling 
> of large folders (over 100,000 messages) is completely unacceptable.
> 
> My kde-cvs folder is nearly a gig in size now, it's just not possible 
> to read, let alone parse a file of that size sufficiently quickly 
> that a user won't notice. I don't believe mutt can handle a folder of 
> this size in an acceptable manner.
> 
> Even for KMail with index files the slowness of handling large mail 
> box files is a problem and very large mail folders (over 1,000,000 
> messages per folder) just aren't handled well. There are a 
> significant amount of people who complain about this scalability 
> issue, more than about the interoperability issue I think.

But for some sites interoperability *IS* the major consideration and large 
mailboxes aren't. Maybe it's time to give a choice. Normally, I err on not 
wanting excessive choice in ways are done but in this case, where there is a 
genuine dicotomy of usage between large but individual and non-roaming 
mailboxes and interoperability with small mailboxes it may be a good idea. In 
fact, making the large scale mailboxes use a totally different method of mail 
storage would probably be a good idea as well, such as maildir + indexes 
would probably be more efficient than a flat message file and index system. 
The current system is prone to corruption if something screws up during the 
file re-writing stage of compacting (such as disk filling etc.) which 
wouldn't be a problem with a "one message, one file" system.

> Beyond scalability the other benefits from using index files are 
> superior message searching and tracking abilities. For instance it 
> makes search folders possible and allows for instance a message to be 
> dragged from KMail into KOrganizer, and a reference from KOrganizer 
> back to the message in KMail to be created. This is because index 
> files allows a message identifier to message location on disk mapping 
> to be created and maintained persistently (e.g. after the application 
> has exited).

The flip side of large index files is that on a shared system with NFS mounted 
filestore they create more storage problems (user's quotas get filled more 
quickly and accessing files over the network constantly for message look-up 
at run time is slow due to the latency). I can see the benefits for the 
integration with KOrganiser though.

> > It would then allow for multiple programs to access and 
> > use the mail folders (but not at the same time) and have programs
> > such as procmail delivering into multiple drop folders without
> > problem.
> 
> You can use KMail with procmail without problems but it's important to 
> follow the instructions in the KMail FAQ.

True, but in a large multi-user system you can't expect every user to read the 
FAQ when all they want to do is log in and get work done. Expecting this from 
users is naive at best.

> > Being able to use the inbox live from /var/mail or /var/spool/mail
> > would also have the added benefit of being able to access the same
> > mailbox remotely using IMAP as Kmail uses. This may not be an issue
> > for those people using simple, sinlge user Linux boxes but in a
> > large multi-user, single server, multiple client system with people
> > often needing to access their mail remotely via IMAP or telnet
> > access it would help a great deal.
> 
> I don't see your point. KMail supports IMAP, so "people often needing 
> to access their mail remotely via IMAP" can use KMail for that. The 
> IMAP support in KMail is constant improving. There's even 
> disconnected support now so that laptop users can sync their mail at 
> the office then use their laptop on a train or plane where no 
> affordable net access is present.

The main problem with kmail's IMAP support at the moment is that it requires 
local folders for storing send e-mail and such.. we're back to the local 
folder problem again.

If you add to that that many IMAP servers look in ~/Mail for their folders 
(which is where Kmail is doing its thing) you start to get some recursive 
hell going on. I know you *CAN* change kmail's default mail storage directory 
but it's in an obscure setting you have to add to an obscure config file 
hidden away somewhere in the user's .kde directory structure. I argue that 
using ~/Mail as the default storage directory is a VeryBadIdea(tm).

> > We are trying to make kmail into an enterprise capable mail client
> > and not just a toy for hackers, aren't we?
> 
> The problem is the old style design is inadequate. It was perfectly 
> adequate at the time it was created because mail volumes were 
> comparatively low. But by todays standards it's a toy design that 
> isn't scalable and doesn't provide message tracking/searching 
> functionality that is expected in modern enterprise mail systems.

Surely it depends upon the enterprise in question? If your enterprise has 
individuals who get lots of messages but uses the same, local system all the 
time and only one mail client then kmail's method works (mostly). If, 
however, it's a heterogeneous environment with people often needing to access 
their mail remotely, sometimes from an old VT100 terminal in a shak half way 
up a mountain in Tibet (as we do) and they don't get huge volumes of e-mail 
then kmail's folder system is totally inappropriate and the old mbox format 
wins.

There is another good reason you may like to use the two different formats. 
Firstly, the inbox will be in one of the "old formats" and it would be nice 
to be able to use this at least as a live mailbox. Once you've filed your 
e-mail then, and only then would you use all the whizzy indexing. This should 
be a choice. You should also be able to designate certain mailboxes as "live" 
and others as archival or have a setting to suck everything out of the live 
inbox into an archival inbox. The important thing here is that I think there 
should be a choice.

> However it may be possible to use index files without losing 
> interoperability. I was surprised to learn Evolution supports this, 
> but wasn't so surprised to see several people on usenet complaining 
> of losing mail after having tried this feature.

Well, as long as there's the rule "work on a copy and never overwrite the 
original until you're absolutely totally sure the new copy is valid" and do 
file locking properly it shouldn't be a problem. You can always rebuild the 
index if you're not absolutely sure it's correct. OK, for that access it 
could be very slow and a pain but as long as you notify the user what's 
happening and most importantly WHY it's happening it shouldn't be an issue.

> Still for maildir folders at least I think theoretically it should be 
> possible to make a scalable interoperable solution that is as 
> reliable as the underlying file system. Nevertheless there's a 
> significant amount of work involved, and I only see a moderate amount 
> of demand for this.

Yes, I can see it being a lot of work but it would make Kmail *THE* leading 
mail client if it can be pulled off (well, other than one other gripe to do 
with large mailing lists and their management :-)). Just because there's 
little demand for it at the moment doesn't mean that once it's implemented 
that people won't know how they lived without it.

At the moment our users just don't trust Kmail not to screw up, especially 
when they get close to their quota limit or when they have a complex 
multi-drop system set up with pre-filtering of e-mail using procmail. They're 
going back to using Dtmail (blurgh), the ancient Sun Mailtool or Pine in a 
terminal window. (Yes, we're not talking Linux boxen here)

I hope you take this as a positive criticism as it's meant to be. I would like 
Kmail to become the best all-round mail client available on Unix platforms.

Steve
Comment 16 jeff pitman 2004-11-24 05:08:16 UTC
Ping. KMail 1.7.1 in KDE 3.3.1 still exhibits the same behavior.  Trying to cleanup is very problematic as it's not very consistent.  For example, when I delete .inbox*, sometimes Compactable goes false, sometimes not.   

I also have the following issues:

1. X-Status: does not get updated with what happened with the message. Only the index maintains the information.  If the index gets deleted, then all emails with X-Status: N will show up as not being read.

2. There is no longer a check on exit to Compact folders.  I believe "compact-all-on-exit=true" in "[General]" in the kmailrc should make this happen, but it doesn't.
Comment 17 David Faure 2004-11-24 11:39:55 UTC
On Wednesday 24 November 2004 05:08, jeff pitman wrote:
> 2. There is no longer a check on exit to Compact folders.  I believe "compact-all-on-exit=true" in "[General]" in the kmailrc should make this happen, but it doesn't.
That's on purpose, compacting is now done while kmail is running
(but this is unrelated to the Compactable=false problem)

Comment 18 Thomas Roessler 2005-09-28 22:14:02 UTC
I think it should be reasonably easy to get interoperability for maildirs, and I would be surprised if it was too hard for mboxes.  The key element to being interoperable for maildirs is to treat the base file name of an individual message as a unique key, i.e., the part before the colon.  (This part is guaranteed to be unique by the way in which maildirs are written.  Message IDs are unique in theory, but not in practice.)  All you have to do to synchronize your index is a usual opendir() readdir() closedir() cycle on the cur/ and new/ subdirs, and parsing the filenames. Likewise, to synchronize a maildir to your index, you have to rename the files accordingly.  There is no need whatsoever to re-parse the messages in this case.

Thomas Roessler
Comment 19 Martin Fitzpatrick 2008-04-05 21:10:54 UTC
No update or duplicates since 2005. Is this fixed? I can't reproduce, but I'm unsure of the exact setup being described. Thanks.
Comment 20 Michael Leupold 2008-11-09 12:22:06 UTC
I'll ask the developers to take a look at that.
Comment 21 Rigo Wenning 2008-11-09 23:36:33 UTC
Noting happened. Kmail still gets out of synch with its indexes if one uses mutt at the same time. Also, attachments to todos or appointments in korganizer still disappear if the mail is moved e.g. by expiry. This is all an issue of using the right identifiers. 
Comment 22 Christophe Marin 2012-01-08 22:58:51 UTC
Closing this report. It is no longer relevant in recent kdepim versions where the storage is handled by Akonadi.