Bug 114163 - dangerous handling of dimap-folders
Summary: dangerous handling of dimap-folders
Status: RESOLVED UNMAINTAINED
Alias: None
Product: kmail
Classification: Applications
Component: IMAP (show other bugs)
Version: unspecified
Platform: Debian testing Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2005-10-10 10:28 UTC by Bastian Venthur
Modified: 2015-04-12 09:44 UTC (History)
6 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Bastian Venthur 2005-10-10 10:28:37 UTC
Version:            (using KDE KDE 3.4.2)
Installed from:    Debian testing/unstable Packages
OS:                Linux

I've allready posted[1] this bug in the debian BTS -- but since I did not find it here after a while, I decided to post it myself. The following text is a verbatim copy of my original bugreport -- I suggest to change the severity of this bug to GRAVE.

---
Hi KDE-maintainers,

I've posted a grave bug against kmail a few weeks ago and now I've found
another one, which is similar but not quite the same. Again it's
dangerous-dimap and it's pretty easy to lose your mails when you move
folders.

I try to reconstruct it the best I can, maybe you create a dimap account
yourself with a few folders and fill them with some spam mails in order
to see the effect.

NOTE: I'm translating the german menu-entrys into english so maybe they
are not 100% correct, I hope you still get the idea.

Assume we have two folders A and B filled with a few mails 

(1) Rightclick a folder A and choose "move to".
(2) Now chose folder B  where folder A  should be moved to.
(3) Now you should see something like this:
   B (filled with mails)
   `-A (filled with mails)

The first problem: A is *not* moved into a at this stage -- allthough the
user thinks so, because he sees it. For the users point of view
everything seemed to work well. But:

(4) Now click "Send & Receive"... A stupid panel pops up, asking what
kind of data is stored into this (which?!) folder: folders or messages.

Here are two problems:
	(a) The user has absolutely no idea whats going on here, since
	    he thinks his folder has allready been succesfully moved.
	(b) Under bad circumstances (happend to my today) you make more
	    than one move operation or even create a new subfolder and
	    then move some folder in the new created one -- now two of
	    those panels pop up -- one for the newly created subfolder
	    and one for the moved one. The problem is, that you cannot
	    see which one kmail means, because its not shown to which
	    folder the question is referring. If you choose "messages"
	    instead of "folders" for the first subdir all your mails in
	    this subdir are lost, but kmail still shows them! AGAIN:
	    kmail shows your mails are still there, but they are gone
	    because of the wrong decision.

(5) Now comes the funniest part. You've clicked "Send Receive" and think
all changes have been commited, but wrong -- you have to *restart* kmail
in order that the new mails are uploaded to the new folders. This is the
second time in one operation where kmails pretends to have done some
work but actually has not. Let's assume you forget to restart kmail and
just close it. After a while you start kmail on a different machine
(maybe laptop) but you are accessing the same dimap-account -- guess
what happens to your not-yet-uploaded mails...


I hope you where able to follow my steps please don't hesitate to ask if
I was unclear in certain points -- I know my english is not the best,
exspecially when I've lost some mails this day :)


Kind regards

Bastian
---



[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=332473
Comment 1 Yazz D. Atlas 2005-12-06 20:09:29 UTC
The above has happened to me also. With the sync of the dimap I have also lost e-mail if the dimap sync happens on a filled partition (aka /home 100%). You aren't warned and kmail will try to sync and only create empty "No Subject" e-mails. 

The imap server get the change and leaves you with "No Subject" empty e-mails, I have lost more then my fair share of e-mail just because of running out of space on /home that way. Maybe my bug should be a new ticket.
Comment 2 Bastian Venthur 2005-12-06 20:22:40 UTC
Please note, that my /home was not filled (not even nearly). There exists another bug about losing mails if /home is full.

I wonder, why the kmail-devels don't bother do fix at least the grave/RC-Bugs. There are bugs in kmail/dimap about losing mails, which are still(!) open over 1/2 year.
Comment 3 Adam Porter 2006-04-13 11:01:44 UTC
FYI, since upgrading to KDE 3.5.2 (Debian), my KMail IMAP problems and crashes have gone away.  If you have had this problem, please test with 3.5.2.  This bug may have been fixed as well.
Comment 4 Julian Mehnle 2006-04-13 12:04:21 UTC
You should wait for at least some _explicit_ feedback that the issue has really been resolved before closing this bug.  Also, some users may not be using self-compiled upstream source directly but prepared distributions, which usually take a while before they distribute new releases.  For example, I'm using Debian/Testing where it could take a month or two before KDE 3.5.2 becomes available.
Comment 5 MartinG 2006-09-24 13:53:23 UTC
I experience the same as #1 - getting "No subject", "unknown" emails in my dIMAP folders (with free space on /home). I guess this is related to the (grave) bug 104956. 

KDE: 3.5.4-5.0.fc5.kde
KMail: 1.9.4
Linux: 2.6.17-1.2187_FC5
Comment 6 MartinG 2006-09-24 13:57:05 UTC
*** This bug has been confirmed by popular vote. ***
Comment 7 Adam Porter 2006-09-25 06:40:16 UTC
I also get the occasional "No Subject, Unknown" empty mails in my inbox.  But I've never been able to figure out if I've actually lost any e-mails, or if the empty mail has been added to the inbox.  That could almost be worse, losing mail but not even knowing it.
Comment 8 Philippe Cloutier 2007-01-10 16:57:49 UTC
The submitter asked on #104956 to "merge" these bugs. So please mark this bug as a duplicate of #104956 (after it's reopened, of course).
Comment 9 Bastian Venthur 2007-01-10 20:11:11 UTC
Here are some minimalistic ways to reproduce this bug.


Some observations:

1) KMail does not show what's on the server but rather what should be on
the server:

When moving mails around in KMail, the mails aren't actually moved on
the server directly. KMail does this later (when?). Even when you close
and reopen KMail the mail is not moved on the server, although KMail
pretends that it has moved the mail! (Which is the root of the problem
as we will see later)

2) Pressing F5 inside a folder commits open changes for this folder on
the server.

2.a) Pressing F5 inside folder foo commits changes for foo but not for
folder bar (which is so dangerous that I cannot believe this was
actually implemented)

3) Rebuilding cache (sorry translated from my German version) seems to
re-read all the data from the server and overwrites KMails yet
uncommitted actions -- for this folder!

4) People who use IMAP tend to check their mails from more than one machine.

5) People have more than one account and sometimes move files from one
folder to another one on a different account.


Here is what I've used:

- Current KMail 1.9.5 from Sid
- current Dovecot from etch on the server side
- Initial layout: Four mails in inbox and two empty folders:

|-- Inbox
|   |-- mail 1
|   |-- mail 2
|   |-- mail 3
|   `-- mail 4
|-- folder 1
`-- folder 2



Ok let's get to the bug. I found different methods to reproduce this
bug. Here is the first and simplest method: I'll delete a mail by moving
a mail from one folder to another:

1) Drag mail 1 to folder 1 (move).
2) Now take a look on the server: Inbox has still 4 mails, folder 1 is
still empty. The changes are not committed to the server although KMail
pretends to.

3) Now go into your inbox and press F5 (refresh)
4) On the serverside mail 1 is now deleted from your Inbox but still not
present in folder 1. In other words if you'd close KMail now and delete
all it's configuration files from this box, mail 1 is lost!

Of course mail 1 is still living somewhere in KMails config files an
pressing F5 folder 1 will bring it back, but only on this machine.
Checking your mail from another box, it will look like your mail just
disappeared. Which is actually true since it is not on the server anymore.

Think this scenario is unrealistic? I for one often move a mail from my
inbox to another folder. And even more often press F5 in my inbox to
fetch new mail.


Here a second way, even funnier:

1) Move mail 1 (by dragging) from folder 1 to folder 2

The changes are not yet committed on the server! But KMail pretends to.

2) Rebuild cache on folder 2 (KMail will delete the just moved mail from
this folder since it's not present on server)

3) Refresh folder 1 with F5. KMail will remove the mail from the server
(which was not visible inside KMail)

Now you have effectively deleted mail mail 1 from KMail and the Server.


I'm sure there are many other ways to exploit this behavior. I've not
seen a single waring or error from KMail it just did what I say and
happily removed mail precious mails. I found another way to lose mails
in combination with two different boxes using the same account and
moving mail, and (ab)using F5 and rebuilding the cache. But I forgot to
write the steps down and am to lazy now to try to reproduce it again.

I even found a bug where KMail duplicates mails which is relevant to
this bug since it's the same problem.

I've not yet played with different boxes sharing the same account (home,
work) or one box having two accounts (and moving mails), but I hope my
setup inspired a few people to do some testing themselves. Please let me
know if you need some help or further information.


So please urge to upstream to reopen it's relevant bugreport(s) (or
better to fix them), I don't have the nerve to deal with upstreams BTS.

And for Debian: please disable DIMAP completely or remove KMail from etch.



Cheers,

Bastian

- - 
Bastian Venthur         http://venthur.de 
Debian Developer    venthur at debian org 
Comment 10 Reinhold Kainhofer 2007-01-15 01:29:03 UTC
Dear Bastian,

dIMAP means *disconnected IMAP* and by definitions all changes are done only locally and only synced to the server later. Everything that you do locally on a dimap account will by definition not immediately be done on the server. That's the whole purpose of that type of account. That's why it is not a "normal" IMAP account, but "disconnected". Sure, if you only send some of the local changes to the server, but rever the others, you will loose those other changes. That seems to be one part of the confusion here.

The other part of the confusion is about the F5 key (which is the shortcut for the menu item "Folder -> Check mail in this folder", which is called like that and placed in the "Folder" menu for a reason), which will really only sync the current folder, nothing else. If you want to sync everything, do "Ctrl+l", which is the shortcut for "File -> check email".

And the third point: When rebuilding the cache, kmail warns you that you will loose all your local changes to that folder...

So I suppose that this is mainly an issue of confusion about what DIMAP really is.
Of course, you can shoot yourself in the foot with everything. As you are a Debian developer, Adriaan's example how to shoot yourself in the foot with svn might be a nice read and is really the same issue, only with a different tool: http://lists.kde.org/?l=kde-pim&m=116877634320916&w=2

Cheers,
Reinhold

PS: Ade's example with svn can even be made simpler:
  svn mv test1 test2  # similar to the folder move)
  svn commit test1    # similar to F5 in the old folder)
  svn revert test2 && rm test2  # similar to re-building the cache of the
                                # receiving folder)
Oops, neither test1 nor test2 are there (since you only committed the removing, but deleted the file to be added...).
Comment 11 S. Burmeister 2007-01-15 08:48:06 UTC
> PS: Ade's example with svn can even be made simpler:
>   svn mv test1 test2  # similar to the folder move)
>   svn commit test1    # similar to F5 in the old folder)
>   svn revert test2 && rm test2  # similar to re-building the cache of the
>                                 # receiving folder)
> Oops, neither test1 nor test2 are there (since you only committed the
> removing, but deleted the file to be added...).


What I do not understand about these fancy examples. How come that just 
because another application has the same fatal "bug" kmail can have it too?

Data loss must be avoided at all expenses, so instead of telling users that 
kmail is not the only app one can achieve data-loss with and advertising it 
as a feature it should be prevented. Especially since kmail is one of those 
applications that everyone uses, no matter how little one knows about 
computers in general.
Comment 12 Reinhold Kainhofer 2007-01-15 14:31:46 UTC
The "bug" is a result of the user explicitly using "Rebuild cache", which warns you that this might result in data loss. That, in combination with a misuse of F5 as opposed to Ctrl+l

It's like pressing "save" in e.g. openoffice and wondering why only the current document, but not the other open documents are saved. If you close the other documents and choose discard changes, you'll also experience data loss. F5 is like the "Save" button in OO, "Rebuild cache" in a different folder is like closing and discarding changes in OO. Would you call that a serious data-loss bug in OpenOffice?

Cheers,
Reinhold

PS: Would it be an improvement to prevent misunderstandings by showing a different/modified folder icon if a dimap folder has uncommitted changes? (Notice, however, that I don't know anything about the internals of the kmail dimap code, so I don't know if that is easily implementable...)

PS2: Maybe the "Troubleshoot IMAP cache..." menu item should not be present in the RMB menu of folders, but only in the Folder menu? That would be an additional indication that it is meant not for daily use, but only when the folder is somehow messed up (or you want to undo all your local changes to the folder after you messed up something).
Comment 13 Laurent Montel 2015-04-12 09:44:11 UTC
Thank you for taking the time to file a bug report.

KMail2 was released in 2011, and the entire code base went through significant changes. We are currently in the process of porting to Qt5 and KF5. It is unlikely that these bugs are still valid in KMail2.

We welcome you to try out KMail 2 with the KDE 4.14 release and give your feedback.