Bug 263761

I have test the akonadi and see that the problem is in kmail2, because received message with cyrillic texts shows normally.

Kmail2 must convert the text of "From" and "To" fields to Base64 encoding before saving it in akonadi resouce:

From: Алексей Test Корепов <mymail@mail.ru>
convert to: =?UTF-8?B?0JDQu9C10LrRgdC10LkgVGVzdCDQmtC+0YDQtdC/0L7Qsg==?= <mymail@mail.ru>


To: Иван Test Иванов <mail2@mail.ru>
convert to: =?UTF-8?B?0JjQstCw0L0gVGVzdCDQmNCy0LDQvdC+0LI=?= <mail2@mail.ru>

like the subject filed are encoded.
Because "Subject" files saves and sends normally, the problem is only in From and To fields.

If I open the akonadi console, and manually replace From and To strings to the Base64-encoded values, id displays correctly in Akonadi and Kmail2, but when I try to open it for editing, it broke those strings again :(

Hmm, I get correctly encoded From header using kmail 2 git from this week (forgot the exact date)

I download today snapshot of kdepim from git repository (git clone http://anongit.kde.org/kdepim), compile, run and test it.
And I can repoduce this but on it too.

And how I can look at the build version of current kmail binary? -v gives me only the version, not current build number:

$ /usr/local/bin/kmail -v
Qt: 4.7.0
KDE Development Platform: 4.5.95 (4.6 RC2)
KMail: 2.0.89

*** Bug 259043 has been marked as a duplicate of this bug. ***

Same problem if I save the message as draft, so the problem is in kmail message text generating process.

Torgny Nyblom, can you on your version of Kmail2 create the message with fields that I describe in first post, and send it to me on murznn(at)gmail(dot)com? 
And save it as draft, save to file and attach to bug?

Created attachment 56271 [details]
Message saved to drafts with wrong fields

I create the message that I describe in first post, save it to drafts and save to file, attaching this file to bug.

Created attachment 56272 [details]
Message with manually corrected fields

I have manually correct the "From" and "To" fields in this file, and after that kmail2 opens and displays it correctly.

I try to fresh install Kubuntu and OpenSUSE with KDE 4.6 and KDEPIM 4.5.94.1 with default settings, and can reproduce this bug:

http://wstaw.org/m/2011/02/04/plasma-desktopbm2641.jpg

Language is set to Englins, locale settings:
> locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

How to reproduce:
1. Open kmail with default settings.
2. Press "New", fill in "To" field text "Вася Пупкин <vasya@example.com>"
3. Press "File -> Save as draft".
4. Open this message in "drafts" folder. You will see "Вася Пупкин <vasya@example.com>" in message list, but in message body window you will see "???? ?????? <vasya@ya.ru>".

Here is the screenshot: http://wstaw.org/m/2011/02/04/plasma-desktopbm2641.jpg

I see that all headers kmail must convert to latin1 encoding via some function.
And, I think, kmail2 miss this converting when save "From" and "To" fields, because after save message as draft, I see in akonadi database field "Subject" converted normally (=?UTF-8?B?0JDQu9C10LrRg....) but From and To contains ??? without "=?UTF-8?" prefix.

Can anybody tell me the name of the function that do this converting?
From: Алексей Test Корепов <mymail@mail.ru>
converts to: 
From: =?UTF-8?B?0JDQu9C10LrRgdC10LkgVGVzdCDQmtC+0YDQtdC/0L7Qsg==?=
<mymail@mail.ru>

And the name of the .cpp or .h file where this convertion is occurs on "Save as draft" message action?

I have a problem that looks like this.

I'm using Kmail-2.0.89
When I create a new e-mail and use Chinese characters in the 'From' field the resulting e-mail will not have the 'From' header. I don't have "???" the e-mail just misses the From header. In the Messagelist in the outbox or draft folder this new e-mail will have "Unknown" as sender.
Like Murz said, the 'Subject' and other headers (like 'Organization') is showing the non-ascii characters correctly.

This bug may be related: https://bugs.kde.org/show_bug.cgi?id=266608

This bug is reproducible on fresh install of Kubuntu 10.10 with daily nighly version (from project-neon) and on OpenSUSE.

How to reproduce (the easiest way):

1. Open Kmail, "New message".
2. Fill "to" with text "Вася <vasya@example.com>"
3. Press "Save as Draft".
4. Go to "Drafts" folder and open the message in new window.

You will see "???? <vasya@example.com>" instad of "Вася <vasya@example.com>"

I have exactly the same proble with kmail 2.0.89.

This bug seems fairly well documented. Does the development team need any more info?

Created attachment 58353 [details]
This patch is just a dirty hack that fixes the problem for the 'from' field

It also highlights the problem that is causing the bug.
When KMime::Message::assembleHeaders() is called the header object contains an UTF-8 string but it's RFC2047 charset is set to ISO 8859-1 (Latin 1) causing the bad serialization as a 7bit string.
The quick hack is to set the RFC2047 charset to UTF-8 of the 'from' header element before serializing the header.

Created attachment 58808 [details]
Patch for convert From To CC BCC fields to UTF-8

Cristian Onet, many thanks for patch! It helps.
But the problem is not only with "From" field. To, CC, BCC fields have this problem too.
I extend this patch for convert those fields too.

Kubuntu users can use my PPA for install patched KDEPIM with this issue solved: https://launchpad.net/~murznn/+archive/kdepim-4.6-git+patches

(In reply to comment #18)
> Created an attachment (id=58808) [details]
> Patch for convert From To CC BCC fields to UTF-8
> 
> Cristian Onet, many thanks for patch! It helps.
> But the problem is not only with "From" field. To, CC, BCC fields have this
> problem too.
> I extend this patch for convert those fields too.

I made the patch just to highlight the problem and didn't add the To, CC, BCC fields because I don't consider that the problem should be fixed at that point. I am waiting for the kdepim developers to fix this properly.

To make the review process easier, please post your patch on https://git.reviewboard.kde.org (group 'kdepim')

(In reply to comment #21)
> To make the review process easier, please post your patch on
> https://git.reviewboard.kde.org (group 'kdepim')

Further discussion at https://git.reviewboard.kde.org/r/101109/

Git commit 7b75f8a769ef0e40e59dedb1327ff0ed51e63ba9 by Thomas McGuire.
Committed on 16/04/2011 at 22:56.
Pushed by tmcguire into branch 'master'.

Use UTF-8 if the charset can't encode the string.

Hopefully the call to canEncode() won't affect the performance too
much...

BUG: 263761

M  +5    -0    kmime/kmime_util.cpp     
M  +1    -1    kmime/kmime_util.h     

http://commits.kde.org/kdepimlibs/7b75f8a769ef0e40e59dedb1327ff0ed51e63ba9

Git commit 6d80788587894ee0cf3e087f959226a3d764af95 by Thomas McGuire.
Committed on 16/04/2011 at 22:56.
Pushed by tmcguire into branch '4.6'.

Use UTF-8 if the charset can't encode the string.

Hopefully the call to canEncode() won't affect the performance too
much...

BUG: 263761
(cherry picked from commit 7b75f8a769ef0e40e59dedb1327ff0ed51e63ba9)

M  +5    -0    kmime/kmime_util.cpp     
M  +1    -1    kmime/kmime_util.h     

http://commits.kde.org/kdepimlibs/6d80788587894ee0cf3e087f959226a3d764af95

I have tested this bug in kmail2 version from GIT from 2011-04-17 - all works good with russian letters, thanks!

I have found and report another bug with mail headers: https://bugs.kde.org/show_bug.cgi?id=271192
Maybe it is related to this bug, can you look on it?

It's not fixed!!! ... when the name & e-mail address is a bit different.

Repeating the testcase from comment #14 after applying the patch from comment #24:
To: Вася <vasya@example.com>          ---> works
To: "Вася" <vasya@example.com>        ---> works
To: "Вася bla" <vasya@example.com>    ---> does not work
To: Вася bla <vasya@example.com>      ---> works
To: "Вася Вася" <vasya@example.com>   ---> works

Maybe a related issue, entering the string "Вася" in KJots (or Kontact's notebook) then quitting and restarting KJots/Kontact: the string will be "????".

(In reply to comment #27)
> It's not fixed!!! ... when the name & e-mail address is a bit different.
> 
Did you build from sources with a recent kdepimlibs git checkout ?

(In reply to comment #28)
> Did you build from sources with a recent kdepimlibs git checkout ?

No, I've applied the patches in comment #24 on kdepimlibs-4.6.2 and then recompiled.
But if it is working on the most recent git version then version 4.6.3 will be a good version.

I test those examples on kdepim from today git:

To: Вася <vasya@example.com>          ---> works
To: "Вася" <vasya@example.com>        ---> works
To: "Вася bla" <vasya@example.com>    ---> works
To: Вася bla <vasya@example.com>      ---> works
To: "Вася Вася" <vasya@example.com>   ---> works

And can't reproduce this issue, maybe you must apply some other patches from git. What version of kmail did you use?

(In reply to comment #30)
> And can't reproduce this issue, maybe you must apply some other patches from
> git. What version of kmail did you use?

Glad to hear that it is working, I have a lot of contacts with non-ascii names.

Okay, maybe there are other patches in the git-version that I need to apply. I will wait for version 4.6.3.
I'm using Kmail/Kontact 4.6beta5 (4.5.95).

Sorry for causing worry.

Kmail/Kontact 4.6beta5 (4.5.95) must already have this patch integrated, because it committed before beta5 release. Can you describe your problem more? 
Is it reproducible with only "To" field or "From", "CC" too? When you save as draft and open message again, you see ??? or normal symbols? You can mail me to murznn (at) gmail.com with detailed description of problem, I try to reproduce it, maybe some part of bug is not fixed.

(In reply to comment #32)
> Kmail/Kontact 4.6beta5 (4.5.95) must already have this patch integrated,
> because it committed before beta5 release.

I think the patch in comment #24 is for the package 'kdepimlibs', while Kontact/Kmail are applications from the package called 'kdepim'. The functionality of kdepim depends on kdepimlibs, but the patch is for kdepimlibs.

> Can you describe your problem more? 
> Is it reproducible with only "To" field or "From", "CC" too? When you save as
> draft and open message again, you see ??? or normal symbols? You can mail me to
> murznn (at) gmail.com with detailed description of problem, I try to reproduce
> it, maybe some part of bug is not fixed.

To/From/CC: "Вася bla" <vasya@example.com>   ---> bad: "???? bla" <vasya@example.com>
To/From/CC: Вася bla <vasya@example.com>     ---> good
To/From/CC: "Вася Вася" <vasya@example.com>  ---> good
To/From/CC: Вася <vasya@example.com>         ---> good
Subject and body are always correct.

The reason it's working for you and not for me is probably because of the other changes in the git-version. Version 4.6.3 will be out soon, then I'll test again.

*** Bug 242231 has been marked as a duplicate of this bug. ***

As expected, fixed in kde-4.6.3.