Bug 278082

Summary: KMail2 can not synchro message's flags with IMAP on MSExchange server 2003
Product: [Applications] kmail2 Reporter: Andrew Kovalev <sailor-02>
Component: generalAssignee: Kevin Ottens <ervin>
Status: RESOLVED FIXED    
Severity: normal CC: abone27, alex, dns_hmpf, gapon007, gspr, kdepim-bugs, notz76, paul_crook, rauchweihe, tmp_prince-promo, vkrause, woebbeking, wolf.behrenhoff, wstephenson
Priority: NOR    
Version: 4.7   
Target Milestone: ---   
Platform: Arch Linux   
OS: Linux   
Latest Commit: Version Fixed In: 4.11.3
Sentry Crash Report:
Attachments: quick and dirty patch

Description Andrew Kovalev 2011-07-19 12:39:57 UTC
Version:           2.1.1 (using Devel) 
OS:                Linux

MSExchange 2003 account on the server is connected via IMAP, the Sent Items folder designated folder on the server.
when sending emails - jumping bug:
"Append failed, server replied: A000182 BAD Protocol Error: Specified set of flags is not valid"
Letters to the server are not synchronized and evaporates completely.

Reproducible: Always

Steps to Reproduce:
connect to IMAP account on MSExchange server 2003

Actual Results:  
every time sending a letter the bug appears

Expected Results:  
Expected that sent letter will be moved to Sent box on server
Comment 1 notz76 2011-09-28 16:20:10 UTC
Have exact the same problem. Also with ms exchange 2003. Makes kmail unuseable.
Comment 2 gapon 2011-11-23 14:32:55 UTC
Not sure whether it is the same or not; cannot store any sent emails on my server (using IMAP), the error is:

Append failed A004632 BAD Invalid flag specified

So, right now, ALL THE SENT EMAILS ARE STORED ONLY LOCALLY (!).
Comment 3 Paul Crook 2011-12-22 14:39:58 UTC
Any progress on this?

Same error message "Append failed, server replied:" "BAD Protocol Error: Specified set of flags is not valid" with KMail Version 4.7.3, Using KDE Development Platform 4.7.3 (4.7.3), on Kubuntu 11.10. Connecting to a Microsoft Exchange 2003 server.

Appears to be more annoying than fatal as the messages are being appended to the folder on the server. Googling of "Specified set of flags is not valid" looks like other mail clients have hit this issue and the simplest explanation is Kmail is possibly not checking what flags that Exchange 2003 supports?
Comment 4 Paul Crook 2011-12-22 15:53:49 UTC
The following blog entry (see below) may be relevant though unfortunately the link to imap2exchange which is supposed to explain what flags Exchange doesn't support is broken (okay found a working link - see below). To summaries it looks like the main thing that Exchange doesn't support is user-defined flags. Is Kmail try to generate user defined flags???

Anyone know how to get logs out of Kmail/Akonadi IMAP server?

http://geek.co.il/wp/2008/06/03/copying-mail-to-ms-exchange-server-using-imap

okay link to imap2exchange is http://code.google.com/p/imap2exchange/wiki/Background

and also of relevance
http://serverfault.com/questions/115769/which-imap-flags-are-reliably-supported-across-most-mail-servers
Comment 5 Paul Crook 2012-01-11 12:30:10 UTC
With reference my comment #3. I was being fooled by the Kmail interface that mail was still being appended to the server. It is not. I now concur completely with comment #2.

In fact the situation is worse. The mails look like they've been append but its just a local cached copy. Accessing the server from another client shows the emails are not on the server. In Kmail, once the cached copy falls away, I get "Remote id is empty or invalid" when trying to open mails that are listed in the "Sent" folder.

I'm getting this problem with Exchange but also with another mail server that I can't identify. The only details I have on its identity is that when you connect it says:
* OK IMAP4 server ready

and its capability statement reads:

CAPABILITY IMAP4rev1 UIDPLUS NAMESPACE QUOTA CHILDREN AUTH=CRAM-MD5 AUTH=DIGEST-MD5 AUTH=PLAIN
Comment 6 Paul Crook 2012-01-11 12:55:35 UTC
flagging information on the mystery server:
FLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent)] flags can be changed
Comment 7 Paul Crook 2012-01-11 13:27:34 UTC
How to get Akonadi IMAP debug logs:
http://techbase.kde.org/Projects/PIM/Akonadi/Debug_IMAP

In summary, from a console do:

> export KIMAP_LOGFILE=/tmp/imap.log
> akonadictl restart

then look through the list of /tmp/imap.log.PID files to find the error.
Comment 8 Paul Crook 2012-01-11 15:12:29 UTC
Okay got some logging information from Akondi. Looks like its sending the flag $SENT when it clearly should be using the flag $MDNSent.


Akondi IMAP logs for the MS Exchange 2003 server:

C: A000035 SELECT "Sent Items"
S: * 190 EXISTS
S: * 0 RECENT
S: * FLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )
S: * OK Permanent flags [ PERMANENTFLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )  ]
.
.
C: A000074 APPEND "Sent Items" (\Seen $SENT) {645}
S: A000074 BAD Protocol Error: Specified set of flags is not valid .
C: A000075 LOGOUT
S: * BYE Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1 signing off.
S: A000075 OK LOGOUT completed.


For my mystery server:

C: A000020 SELECT "Sent"
S: * 101 EXISTS
S: * 0 RECENT
S: * FLAGS ( \Answered \Flagged \Deleted \Seen \Draft $MDNSent )
S: * OK flags can be changed [ PERMANENTFLAGS ( \Answered \Flagged \Deleted \Seen \Draft $MDNSent )  ]
.
.
C: A000074 APPEND "Sent" (\Seen $SENT) {522}
S: A000074 BAD invalid APPEND command syntax, invalid message flags list syntax
C: A000075 UID STORE 4078 FLAGS (\Seen)
S: A000075 OK STORE complete
C: A000076 LOGOUT
S: * BYE disconnecting
S: A000076 OK LOGOUT complete
Comment 9 Paul Crook 2012-01-12 19:43:11 UTC
Created attachment 67750 [details]
quick and dirty patch
Comment 10 Paul Crook 2012-01-12 19:54:57 UTC
Comment on attachment 67750 [details]
quick and dirty patch

Okay here's a quick and dirty patch to fix this. The patch is against the source code for Ubuntu 11.10 package kdepimlibs version 4:4.7.3-0ubuntu0.1.

I say "quick and dirty" because all it does is remove all occurrences of the '$SENT' flat when it appears in the clients APPEND command. This is obviously not ideal as it means the flag is removed for all IMAP servers, even those where its use would have been okay. If someone has time to dig into the Akonadi logic a better solution would be to check the flags allowed by the IMAP server's folder which it is appending messages to and allow only those flags. It possibly should also translate '$SENT' to '$MDNSent' where appropriate.

I modified the method AppendJob::doStart() in the file kdepimlibs-4.7.3/kimap/appendjob.cpp.
Comment 11 Will Stephenson 2012-01-13 21:28:15 UTC
Could this be related to bug 291332? Reassigning to kmail2 as per Kevin's comment on 291332 that the imap resource just passes through the flags given to it.
Comment 12 Paul Crook 2012-01-16 13:06:47 UTC
Looks like it might be the same problem.
Comment 13 themroc 2012-05-02 18:26:05 UTC
I have a similar problem since Kubuntu 11.10 on my aim.com (aol mail account): 

After sending a mail, Kmail2 failed to copy this mail in the sent IMAP Folder: 
"server replied: A000035 No append failed" 

SMTP with smtp.aim.com and Port 587, TLS. Kmail 4.7.3 on Kubuntu 11.10 amd64
Comment 14 themroc 2012-12-09 21:31:25 UTC
Same Problem in Kubuntu 12.10 with KDE 4.9.4
Comment 15 Wolf Behrenhoff 2013-03-27 21:35:00 UTC
I've got the same problem. It is still present in KDE 10.1. 

I just got:
Append failed, server replied: A000373 BAD Protocol Error: Specified set of flags is not valid.
Append failed, server replied: A000201 BAD Protocol Error: Specified set of flags is not valid.

What can I do about these messages? What are they telling me? Does KMail treat this problem correctly or does it just store the message without any flags? The point is, as a user I do not want to see this message 100 times a day, in fact I do not want to see it at all - or only once with a more useful message like "Your IMAP server does not support (explanation what the flags are for). This has the following impact ... [OK]".
Comment 16 Carlo Alberto Sanjust 2013-09-04 14:48:21 UTC
Probably related to: https://bugs.kde.org/show_bug.cgi?id=291332
Comment 17 Andre Woebbeking 2013-09-27 20:22:06 UTC
Git commit c561149538336804f212f5a12ea6cf3d1a3a19f5 by Andre Woebbeking.
Committed on 27/09/2013 at 20:19.
Pushed by woebbe into branch 'KDE/4.11'.

Don't bother IMAP servers with not supported flags

Created a new method ResourceTask::fromAkonadiToSupportedImapFlags() to convert
Akonadi to IMAP flags supported by the server and use it everywhere instead of 
fromAkonadiFlags(). 

fromAkonadiFlags() was made private to make it impossible to call it by accident.
Related: bug 287356
FIXED-IN: 4.11.3
REVIEW:112724

This also reduces the risk to trigger 287356 as the insert on the IMAP server
doesn't fail due to unsupported IMAP flags.

M  +1    -1    resources/imap/additemtask.cpp
M  +2    -2    resources/imap/changeitemsflagstask.cpp
M  +2    -15   resources/imap/changeitemtask.cpp
M  +21   -0    resources/imap/resourcetask.cpp
M  +5    -0    resources/imap/resourcetask.h

http://commits.kde.org/kdepim-runtime/c561149538336804f212f5a12ea6cf3d1a3a19f5