Bug 74017

I've setup a test imap account and could reproduce that problem.

first I created a filter that matches any of "status contains unread" "status contains new". Filter action is pipe through spamc (from spamassassin).

Now when reading messsages on an imap account, I can apply that filter action. And sometimes I see :
"Could not remove folder imap://username@server:143/INBOX/;UID="
and all the emails are deleted...

I'll attach a sample message so that you can try to reproduce all that.

Created attachment 4493 [details]
sample email to reproduce the problem

Arnaud, thanks for informing us. Could you please provide us with the exact filter definition (as stated in your $KDEHOME/share/config/kmailrc) and the imap server you are connecting against? I cannot reproduce the behavior you describe, so I'll need to exactly match your setup.

Just ask for more if you need it ... Arnaud

[Filter #0]
ConfigureShortcut=true
Icon=
StopProcessingHere=true
action-args-0=spamc
action-name-0=filter app
actions=1
apply-on=check-mail,manual-filtering
contentsA=unread
contentsB=new
fieldA=<status>
fieldB=<status>
funcA=contains
funcB=contains
name=Spam Check
operator=or
rules=2

first time I saw the problem was with :

telnet mailbox.epfl.ch 143
Trying 128.178.50.37...
Connected to mailbox.epfl.ch.
Escape character is '^]'.
* OK Messaging Multiplexor (iPlanet Messaging Server 5.2 HotFix 1.18 (built Jul 28 2003))
C00000 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN LANGUAGE XSENDER X-NETSCAPE XSERVERINFO AUTH=PLAIN STARTTLS
C00000 OK CAPABILITY completed

and I could reproduce it many times on a kolab server :

telnet kolab.ouvert.ch 143
Trying 212.74.171.57...
Connected to kolab.ouvert.ch.
Escape character is '^]'.
* OK kolab.ouvert.ch Cyrus IMAP4 v2.1.14 server ready
C00000 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ MAILBOX-REFERRALS NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES IDLE STARTTLS
C00000 OK Completed

Thanks for the info. Can you still reproduce it if you disable load on demand for the account?

I disabled "load attachements on demand" and could reproduce that at my first try...

Arnaud, I have now sent that mail you sent (and a few others) through the filter you specified above quite a few times and am unable to trigger deletion of the inbox.  This is against a UW server and a kolab. Current spamc from a suse 9.0 install. Is there any chance I could get access to your installation or could you send me the debug output of both kmail and kio_imap of a (minimal) execution that triggers it?

I just installed KDE 3.2.0 and had this exact thing happen to me. It deleted every email in my inbox and I have no way of recovering them. I'm really pissed off right now, but how could you release software with a known reproducable bug like this?

Apparently not all data was lost, ~/.kde/share/apps/kmail/imap/.accountname.directory/INBOX still contains the headers for the emails. I'd hope that KMail caches the entire message somewhere else, but I'm afraid I'm just out of luck and this is all that remains of the 600+ emails.

Jason, we are sorry about your loss of mail, but the bug was neither known to us nor can we reproduce it, so far. And no, the mails themselves are not cached anywhere. We are trying hard to resolve this.

Subject: kdepim/kmail

CVS commit by tilladam: 

Don't do an expunge Folder, thereby removing all mails in an imap folder,
when the deleteMessage method is called on a message that does not have
a UID, for whatever reason.

Ouch.

Now we need to find out why the message has no uid.

Ingo, backport, I assume?

CCMAIL: 74017@bugs.kde.org


  M +15 -2     kmfolderimap.cpp   1.170


--- kdepim/kmail/kmfolderimap.cpp  #1.169:1.170
@@ -1213,5 +1213,14 @@ void KMFolderImap::deleteMessage(KMMessa
   KURL url = mAccount->getUrl();
   KMFolderImap *msg_parent = static_cast<KMFolderImap*>(msg->storage());
-  url.setPath(msg_parent->imapPath() + ";UID=" + msg->headerField("X-UID"));
+  QString uid = msg->headerField("X-UID");
+  /* If the uid is empty the delete job below will nuke all mail in the 
+     folder, so we better safeguard against that. See ::expungeFolder, as
+     to why. :( */
+  if ( uid.isEmpty() ) {
+     kdDebug( 5006 ) << "KMFolderImap::deleteMessage: Attempt to delete "
+                        "an empty UID. Aborting."  << endl;
+     return;
+  }
+  url.setPath(msg_parent->imapPath() + ";UID=" + uid );
   if ( mAccount->makeConnection() != ImapAccountBase::Connected )
     return;
@@ -1234,5 +1243,9 @@ void KMFolderImap::deleteMessage(QPtrLis
   for ( QStringList::Iterator it = sets.begin(); it != sets.end(); ++it )
   {
-    url.setPath(msg_parent->imapPath() + ";UID=" + *it);
+    QString uid = *it;
+    // Don't delete with no uid, that nukes the folder. Should not happen, but
+    // better safe than sorry.
+    if ( uid.isEmpty() ) continue; 
+    url.setPath(msg_parent->imapPath() + ";UID=" + uid);
     if ( mAccount->makeConnection() != ImapAccountBase::Connected )
       return;

Jason, couldn't your server admin restore your mails from backups ? Mine could and my 1200+ mails are back ...

I paste here a network dump of such an imap session, bad bad bad...

* OK kolab.ouvert.ch Cyrus IMAP4 v2.1.14 server ready
0 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ MAILBOX-REFERRALS NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES IDLE STARTTLS
0 OK Completed
1 LOGIN "kmail@ouvert.ch" "xxxxxxxx"
1 OK User logged in
2 LIST "" "%"
* LIST (\HasNoChildren) "/" "INBOX"
2 OK Completed (0.000 secs 2 calls)
3 LIST "" "INBOX"
* LIST (\HasNoChildren) "/" "INBOX"
3 OK Completed (0.000 secs 2 calls)
4 STATUS "INBOX" (UNSEEN)
* STATUS INBOX (UNSEEN 0)
4 OK Completed
5 LIST "" "INBOX"
* LIST (\HasNoChildren) "/" "INBOX"
5 OK Completed (0.000 secs 2 calls)
6 LIST "" "INBOX/%"
6 OK Completed (0.000 secs 1 calls)
7 LIST "" "INBOX"
* LIST (\HasNoChildren) "/" "INBOX"
7 OK Completed (0.000 secs 2 calls)
8 SELECT "INBOX"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]  
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1075767684]  
* OK [UIDNEXT 51]  
8 OK [READ-WRITE] Completed
9 UID FETCH 47 (UID RFC822.SIZE INTERNALDATE FLAGS BODY.PEEK[])
* 1 FETCH (FLAGS (\Seen) UID 47 INTERNALDATE " 4-Feb-2004 13:45:42 +0100" RFC822.SIZE 3047 BODY[] {3047}
Return-path: <uj4@bigfoot.com>
... more headers

... more body
</body></html>

)
9 OK Completed
10 NOOP
10 OK Completed
11 NOOP
11 OK Completed
12 UID FETCH 1:* (UID FLAGS)
* 1 FETCH (FLAGS (\Seen) UID 47)
* 2 FETCH (FLAGS (\Seen) UID 48)
* 3 FETCH (FLAGS (\Seen) UID 49)
* 4 FETCH (FLAGS (\Seen) UID 50)
12 OK Completed
13 CLOSE
13 OK Completed
14 UNSUBSCRIBE "INBOX"
14 OK Completed
15 DELETE "INBOX"
15 NO Operation is not supported on mailbox
16 SELECT "INBOX"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]  
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1075767684]  
* OK [UIDNEXT 51]  
16 OK [READ-WRITE] Completed
17 UID STORE 1:* +FLAGS.SILENT (\DELETED)
17 OK Completed
18 CLOSE
18 OK Completed
19 DELETE "INBOX"
19 NO Operation is not supported on mailbox

kio (KSycoca): Trying to open ksycoca from /var/tmp/kdecache-ghort/ksycoca
kio (KIOJob): stat imap://kmail%40ouvert.ch@kolab.ouvert.ch:143/INBOX/;SECTION=UNSEEN
kio (KIOJob): error 38 imap://kmail@ouvert.ch@kolab.ouvert.ch:143/INBOX/;UID=
kio (Slave): killing slave pid=9934 (imap://kolab.ouvert.ch)
kio (Slave): slave died pid = 9934
kio (KIOJob): Job::kill this=0x832ee80 m_progressId=0 quietly=true

these are the only kio outputs I get running kmail...

I started kdebugdialog did "select all" apply then ok. And killed all kio I found.

(I can't reboot nor close my kde session because of running tests...)

*** Bug 70429 has been marked as a duplicate of this bug. ***

Subject: kdepim/kmail

CVS commit by tilladam: 

When an imap message is filtered through spamc and classified as spam, a 
new message is constructed and returned which has a new set of headers. 
Consequently there is no X-UID header anymore. If the message is then left 
in the same folder and not moved somewhere else, the original mail is 
removed from the folder on the server and the new one uploaded. To figure 
out which mail to remove, the (now missing) X-UID header is looked at. 
That is not there -> no uid -> message is not delete (used to be folder is 
expunged). To avoid that, restore the X-UID header after the pipe through
to make sure it's there. Not really correct, technically, but I can't
think of another reliable way to fix this.

Thanks a lot to Arnaud Burlet for helping me track this bugger down and
testing.

Backport?

CCMAIL: 74017-done@bugs.kde.org


  M +9 -1      kmfilteraction.cpp   1.115


--- kdepim/kmail/kmfilteraction.cpp  #1.114:1.115
@@ -505,6 +505,14 @@ KMFilterAction::ReturnCode KMFilterActio
     QByteArray msgText = kmkernel->getCollectedStdOut( &shProc );
 
-    if ( !msgText.isEmpty() )
+    if ( !msgText.isEmpty() ) {
+    /* If the pipe through alters the message, it could very well 
+       happen that it no longer has a X-UID header afterwards. That is
+       unfortunate, as we need to removed the original from the folder
+       using that, and look it up in the message. When the (new) message
+       is uploaded, the header is stripped anyhow. */
+      QString uid = aMsg->headerField("X-UID");
       aMsg->fromByteArray( msgText );
+      aMsg->setHeaderField("X-UID",uid);
+    }
     else
       return ErrorButGoOn;

Subject: Re:  grave imap  bug : All messages of inbox lost

On Friday 06 February 2004 08:35, Till Adam wrote:
...
> When an imap message is filtered through spamc and classified as
> spam, a new message is constructed and returned which has a new set
> of headers. Consequently there is no X-UID header anymore. If the
> message is then left in the same folder and not moved somewhere
> else, the original mail is removed from the folder on the server
> and the new one uploaded. To figure out which mail to remove, the
> (now missing) X-UID header is looked at. That is not there -> no
> uid -> message is not delete (used to be folder is expunged). To
> avoid that, restore the X-UID header after the pipe through to make
> sure it's there. Not really correct, technically, but I can't think
> of another reliable way to fix this.
>
> Thanks a lot to Arnaud Burlet for helping me track this bugger down
> and testing.
>
> Backport?

Yes please. And please update the KMAIL_VERSION in kmversion.h to 
"1.6.1" so that users know whether they are using a version of KMail 
with these critical fixes.

(Updating the version also helps with answering bugs).

Don.
 

Subject: Re:  grave imap  bug : All messages of inbox lost

On Friday 06 February 2004 03:39, Don Sanders wrote:

> > Backport?
>
> Yes please. 

Done.

> And please update the KMAIL_VERSION in kmversion.h to 
> "1.6.1" so that users know whether they are using a version of KMail
> with these critical fixes.
>
> (Updating the version also helps with answering bugs).

Yeah, makes sense. Done.

Till

It would like to know where place I apply patche: kdepim-3.2.0-HOT_FIX_FOR_KMAIL.patch which is the directory? Remembering that I use Slackware 9.1 

I installing the following packages: kdegraphics-3.2.0-i486-2.tgz - kdemultimedia-3.2.0-i486-2.tgz - kdepim-3.2.0-i486-2.tgz I solve this bug?

Hi all,
Why don't I see this bug report before this morning ??? :-(

I've tried the Jason's tip, so i've looked in the directory, but the inbox directory was disappeared :-(((

back to thunderbird, and I will wait a little more time, before retried kontact ... too bad !

Neverless, congratulations to all the dev team for this great app !

I have encountered similar problem with kmail 1.6.2 and offline IMAP. 

After creating offline IMAP account kmail started to synchronize folders and their contents with server. Crashed upon that job. Started kmail again and ordered to check mail (Ctrl+L). All folders that had unread messages in showed error "Could not create folder ...."  (contrary to delete in previous posts here). The result was the same, though: nothing left in folders.

The folder with Sent Items was OK, as there were no unread messages inside. 

No more problems after Folder tree had been correctly created. Crashes while creating IMAP (usual also) folder trees (Exchange Public Folders) are quite usual. After folder tree has been created, kmail is stable.

I had exactly the same behavior with KMail 1.9.5 as Toomas Mõttus. disconnected IMAP, started to synchronize folders, crashed. when I restarted kmail, it correctly sychronized all the folders, but I've confirmed that the inbox is indeed empty.