Bug 291332

Summary: kmail uses nonstandard IMAP message flags - does not comply with RFC2060
Product: [Applications] kmail2 Reporter: Stefan Gofferje <stefan>
Component: generalAssignee: kdepim bugs <kdepim-bugs>
Status: RESOLVED FIXED    
Severity: normal CC: blackravener, eva, fischer, g, huwie, kde-bugs-68795, kdebugs, kdepim-bugs, lfranchi, paul_crook, rock8000, sam.carcagno, sven.burmeister, tmp_prince-promo, vkrause, Vojtech.Zeisek, wbauer1, winter, woebbeking, wstephenson
Priority: NOR    
Version: unspecified   
Target Milestone: ---   
Platform: openSUSE   
OS: Linux   
Latest Commit: Version Fixed In: 4.11.2
Attachments: kdepim-runtime patch to use IMAP standard reply flag
kdepim-runtime patch to use IMAP standard reply flag

Description Stefan Gofferje 2012-01-12 06:09:43 UTC
Version:           unspecified (using KDE 4.7.2) 
OS:                Linux

kmail2 uses it's own IMAP flags. Instead of RFC2060 compliant flags, like eg "\Answered", it uses "$REPLIED". That throws off other clients, e.g. Hastymail² webmailer which then cannot display the message status properly.


Reproducible: Always

Steps to Reproduce:
Reply to a message, check message flags


Expected Results:  
Comply with RFC2060
Comment 1 Kevin Ottens 2012-01-12 14:11:18 UTC
The IMAP resource just forwards what the client says, so kmail2 was really the right place.
Comment 2 Will Stephenson 2012-01-13 11:26:15 UTC
Right, it is using Replied instead of Answered (which kmail/akonadi also know about) http://lxr.kde.org/source/kde/kdepim-runtime/agents/maildispatcher/sentactionhandler.cpp#37

Akonadi flags
http://lxr.kde.org/source/kde/kdepimlibs/akonadi/kmime/messageflags.h#49

Which are converted to standard IMAP flags, or else passed through verbatim:
http://lxr.kde.org/source/kde/kdepim-runtime/resources/imap/resourcetask.cpp#317

I guess that since KMail's internal API vocabulary uses the term 'Replied', the Replied flag was used.

Leo, any comment as the author?
Comment 3 Paul Crook 2012-01-16 13:05:13 UTC
As per Will Stephenson comments (in bug 278082) it looks like bug 278082 is related. I.e. the flag $SENT is being passed through and causing APPEND to fail on MS Exchange servers (amoung others).
Comment 4 Vojtěch Zeisek 2012-06-24 21:18:01 UTC
I think it's same issue. On my IMAP server I use RoundCube webmailer. When I forward, reply or mark with IMAP label any message in RoundCube, Thunderbird or K-9 Mail (Android, well it does not support labels), those changes are synchronized among each other and to KMail. So KMail does understand it. But when I reply, forward or mark with IMAP label any message in KMail, those information are not synchronized with webmail, nor other clients. Only marking as important ("starring") works as expected (among all clients in all directions). Is it possible to force KMail to use correct flags? I faced this in openSUSE 12.1 in KDE 4.7.2 as well as 4.7.4.
Comment 5 Chuso Pérez 2012-07-27 18:16:10 UTC
Created attachment 72797 [details]
kdepim-runtime patch to use IMAP standard reply flag

Hi, I've got into this problem too.
As said in previous comments, KMail uses $REPLIED own IMAP flag instead of \Answered standard flag. Well, the problem is actually in kdepim-runtime package.
kdepim-runtime only sets $REPLIED IMAP flag when replying a message but can recognize both flags as a kind of "synonyms". So, it can recognize as replied messages that have been replied from other mail clients but these other mail clients can't recognize as replied messages that have been replied from kdepim-runtime.
I attach a patch that modifies kdepim-runtime to set \Answered IMAP flag instead of $REPLIED when you reply a message. Since kdepim-runtime can recognize both flags, there shouldn't be any side effect and I was using this patch for weeks without noticing any problem. But if you find any problem with it, please report.
Comment 6 Chuso Pérez 2012-09-22 23:27:08 UTC
Created attachment 74101 [details]
kdepim-runtime patch to use IMAP standard reply flag

Updated patch for KDE 4.9.
Comment 7 Allen Winter 2012-09-23 12:59:43 UTC
Chuso,

Would you please post your patch to reviewboard.kde.org?
Comment 8 Chuso Pérez 2012-11-02 16:51:36 UTC
Hi, Allen.
Thanks for the advice. The patch is now posted at reviewboard (107179).
Sorry for the delay, but I didn't receive an e-mail notifying me of your comment as supposed.
Comment 9 Huw 2013-04-11 14:33:12 UTC
Is there any update for this?  I'm getting behaviour that could be related; my sent emails are not being stored on the mail server (which, to me, is a critical failure) and I suspect it's something to do with non-standard flags being used.
Comment 10 Wolfgang Bauer 2013-05-27 22:14:41 UTC
*** This bug has been confirmed by popular vote. ***
Comment 11 Carlo Alberto Sanjust 2013-09-04 11:42:36 UTC
I can confirm this bug in Kmail 4.10.5. The sent messages are not stored on the IMAP server "Sent" folder, but are shown locally in the same folder. The mails are also marked by different icons in the message list window (a closed envelope for the messages saved on the server, and visible to all clients, a stamp for the messages shown only in Kmail).
The problem arises with some IMAP servers only, with some other (Office 365) all works well. Probably some servers can accept also non standard flags for the mails.
Comment 12 Huw 2013-09-04 11:59:05 UTC
Thanks for the additional confirmation Carlo, however the KDE devs apparently don't give a damn about this critical bug.
Comment 13 Carlo Alberto Sanjust 2013-09-04 13:33:23 UTC
In my case the problem is related to the $SENT flag. Akonadi debug log says:

C: A000035 APPEND "Sent" (\Seen $SENT) {422}
S: A000035 BAD invalid APPEND command syntax, invalid message flags list syntax

Whereas the server supports:

FLAGS ( \Answered \Flagged \Deleted \Seen \Draft $MDNSent $fax $urgent $voice )
Comment 14 Carlo Alberto Sanjust 2013-09-04 13:44:07 UTC
In the case of Microsoft Office 365 IMAP server, the answer is:

C: A000150 APPEND "Sent Items" (\Seen $SENT) {559}
S: + Ready for additional command text.

The message is accepted and the unknown flag is ignored, all messages in the "Sent" folder are marked simply as \Seen:

C: A000198 SELECT "Sent Items"
S: * 278 EXISTS
S: * 0 RECENT
S: * FLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )
S: * OK Permanent flags [ PERMANENTFLAGS ( \Seen \Answered \Flagged \Deleted \Draft $MDNSent )  ]
S: * OK UIDVALIDITY value [ UIDVALIDITY 11  ]
S: * OK The next unique identifier value [ UIDNEXT 663  ]
S: A000198 OK SELECT completed. [ READ-WRITE  ]
C: A000199 FETCH 1:278 (FLAGS UID)
S: * 1 FETCH ( FLAGS (\Seen) UID 385 )

Hence the different icon shown in the message list window in Kmail.
Comment 15 Andre Woebbeking 2013-09-13 12:26:46 UTC
Git commit 658056eef812eeebb8b3d995378e178b0ae3c9e6 by Andre Woebbeking.
Committed on 13/09/2013 at 12:21.
Pushed by woebbe into branch 'KDE/4.11'.

Convert the Akonadi flags to IMAP flags in ChangeItemsFlagsTask like it's done in ChangeItemTask
and handle Akonadi::MessageFlags::Replied in ResourceTask::fromAkonadiFlags().
FIXED-IN: 4.11.2
REVIEW:112482

M  +2    -2    resources/imap/changeitemsflagstask.cpp
M  +1    -1    resources/imap/resourcetask.cpp

http://commits.kde.org/kdepim-runtime/658056eef812eeebb8b3d995378e178b0ae3c9e6
Comment 16 Chuso Pérez 2013-09-19 09:10:02 UTC
Andre's commit worked for me.
Comment 17 Christian Mollekopf 2013-11-20 10:58:41 UTC
*** Bug 322346 has been marked as a duplicate of this bug. ***
Comment 18 Christian Mollekopf 2013-11-20 10:59:22 UTC
*** Bug 292399 has been marked as a duplicate of this bug. ***
Comment 19 Christian Mollekopf 2013-11-20 12:05:18 UTC
*** Bug 309946 has been marked as a duplicate of this bug. ***
Comment 20 eva 2014-03-21 16:29:24 UTC
In version 4.11.5 this issue is fixed for new emails. However, I still have a bunch of old sent mails that the server (Kerio) does not accept for uploading. How can I fix the flag for those old "sent" mails?
Comment 21 Wolfgang Bauer 2014-03-21 20:13:02 UTC
Just move them to a different folder and back again. That should be enough I think. 
If not, move them to a different resource (your "Local Folders" f.e.) and back.

You could of course also remove the offending flag with akonadiconsole. Either do that on the "Resources" tab, or use some SQL statement to remove the "$SENT" flag everywhere ("DB console").
The mails should then be uploaded to the server when the folder gets synchronized the next time.

I guess the first thing is easier to do especially for not so experienced users... ;-)
Comment 22 Andre Woebbeking 2014-03-22 13:04:51 UTC
> --- Comment #20 from eva@kde.org ---
> In version 4.11.5 this issue is fixed for new emails. However, I still have
> a bunch of old sent mails that the server (Kerio) does not accept for
> uploading. How can I fix the flag for those old "sent" mails?

Hi Eva, can you please try a 4.12.x version. My fix here was not complete :-( 
but is in 4.12. If your server reports which flags it supports.


Cheers,
André
Comment 23 Larry CK 2015-01-10 13:47:58 UTC
(In reply to Andre Woebbeking from comment #22)
> > --- Comment #20 from eva@kde.org ---
> > In version 4.11.5 this issue is fixed for new emails. However, I still have
> > a bunch of old sent mails that the server (Kerio) does not accept for
> > uploading. How can I fix the flag for those old "sent" mails?
> 
> Hi Eva, can you please try a 4.12.x version. My fix here was not complete
> :-( 
> but is in 4.12. If your server reports which flags it supports.
> 
> 
> Cheers,
> André

BUG 292399 report sais that it is resolved by resolving a duplicate bug 291332 (i.e.) this one. There also is a Bug 286692 that in esssence is the same as 292399 but is not resolved.
Basicaly, I've tried this on OpenSuse 13.1 and 13.2 and kmail 4.14.2 and 4.14.3 and attachment icon is till not shown. i've also tried in on kubuntu 14.10 with kmail 4.14.1 and issue is present there as well. I've tried with MS Excahge mail account connecting over DavMail and I've tried Gmail through IMAP directly and attachment icons are not displayed. So what's the status on this? This is very annoying bug! I'd love to use kmail for by business, but not this one bug is holding me back.
Comment 24 Andre Woebbeking 2015-01-11 14:59:07 UTC
Well, I only fixed that Exchange hasn't to cope with unknown IMAP flags and complains with errors. And unfortunately there is no IMAP flag like "has attachment". Other IMAP servers allow any flags (also self defined as Akonadi's $ATTACHMENT).

If you select a mail with an attachement Akonadi sees it and flags it. But with the next sync this is overwritten :-( This is probably caused by the fact that the Akonadi developers only work with IMAP servers allowing any flags, so the flags in Akonadi and the IMAP server are the same and can be synced without information loss.
Comment 25 Larry CK 2015-01-11 15:07:53 UTC
(In reply to Andre Woebbeking from comment #24)
> Well, I only fixed that Exchange hasn't to cope with unknown IMAP flags and
> complains with errors. And unfortunately there is no IMAP flag like "has
> attachment". 

From what you are saying it sounds like it is not going to be fixed (unless kmail stops depending on Akonadi, which is not going to happen any time soon), not any time soon and not for all systems anyways. Am I misunderstanding you?
Comment 26 Andre Woebbeking 2015-01-11 21:35:51 UTC
(In reply to Larry CK from comment #25)
> Am I misunderstanding you?

Yep :-) I'm no Akonadi expert but I think it's fixable without changing Akonadi. But as usual someone has to do the work.