Bug 263761 - Non-ascii symbols in From and To symbols replaced to "?" when sending email
Summary: Non-ascii symbols in From and To symbols replaced to "?" when sending email
Status: RESOLVED FIXED
Alias: None
Product: kmail2
Classification: Unclassified
Component: general (show other bugs)
Version: 2.0.89
Platform: Ubuntu Packages Linux
: NOR normal with 20 votes (vote)
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
: 242231 259043 (view as bug list)
Depends on:
Blocks:
 
Reported: 2011-01-20 16:33 UTC by Murz
Modified: 2011-05-06 19:00 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.6.3


Attachments
Message saved to drafts with wrong fields (674 bytes, application/octet-stream)
2011-01-21 09:44 UTC, Murz
Details
Message with manually corrected fields (746 bytes, application/octet-stream)
2011-01-21 09:46 UTC, Murz
Details
This patch is just a dirty hack that fixes the problem for the 'from' field (657 bytes, patch)
2011-03-26 13:24 UTC, Cristian Oneț
Details
Patch for convert From To CC BCC fields to UTF-8 (853 bytes, patch)
2011-04-11 21:19 UTC, Murz
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Murz 2011-01-20 16:33:18 UTC
Version:           2.0.89 (using KDE 4.5.95) 
OS:                Linux

When I fill fields "From" and "To" with cyrillic (utf-8) symbols, it fills normally (I see in the screen cyrillic letters), but when I send the message, I see the "????" symbols instead of my cyrillic letters! 
And receiver of message got the "????" symbols instead of cyrillic.

Cyrillic symbols in Subject and Body sends normally.

And I see that in message headers there are "????" symbols, but in message list I see normal cyrillic letters (see the screenshot http://img840.imageshack.us/img840/5227/screen4s.png ), so in database this values saves correctly (because in list I see normal text), the problem is when extracting from database.

Maybe this is akonadi bug if the message stores and sent via akonadi service?

Reproducible: Always

Steps to Reproduce:
1. Create new email message.

2. Fill fields:
From: Алексей Test Корепов <mymail@mail.ru>
To: Иван Test Иванов <mail2@mail.ru>
and other fields.

3. Press "Send" button.

4. Look at the message in "Sent" folder and on inbox in receiver.

Actual Results:  
You will see in thouse fields:
From:	??????? Test ??????? <korepov@qseo.ru>
To:	???? Test ?????? <murznn@gmail.com>

Expected Results:  
You must see:
From: Алексей Test Корепов <mymail@mail.ru>
To: Иван Test Иванов <mail2@mail.ru>


OS: Linux (x86_64) release 2.6.35-25-generic
Compiler: cc
Comment 1 Murz 2011-01-20 16:46:23 UTC
I have test the akonadi and see that the problem is in kmail2, because received message with cyrillic texts shows normally.

Kmail2 must convert the text of "From" and "To" fields to Base64 encoding before saving it in akonadi resouce:

From: Алексей Test Корепов <mymail@mail.ru>
convert to: =?UTF-8?B?0JDQu9C10LrRgdC10LkgVGVzdCDQmtC+0YDQtdC/0L7Qsg==?= <mymail@mail.ru>


To: Иван Test Иванов <mail2@mail.ru>
convert to: =?UTF-8?B?0JjQstCw0L0gVGVzdCDQmNCy0LDQvdC+0LI=?= <mail2@mail.ru>

like the subject filed are encoded.
Because "Subject" files saves and sends normally, the problem is only in From and To fields.
Comment 2 Murz 2011-01-20 16:51:47 UTC
If I open the akonadi console, and manually replace From and To strings to the Base64-encoded values, id displays correctly in Akonadi and Kmail2, but when I try to open it for editing, it broke those strings again :(
Comment 3 Torgny Nyblom 2011-01-20 18:38:45 UTC
Hmm, I get correctly encoded From header using kmail 2 git from this week (forgot the exact date)
Comment 4 Murz 2011-01-21 09:11:35 UTC
I download today snapshot of kdepim from git repository (git clone http://anongit.kde.org/kdepim), compile, run and test it.
And I can repoduce this but on it too.
Comment 5 Murz 2011-01-21 09:12:51 UTC
And how I can look at the build version of current kmail binary? -v gives me only the version, not current build number:

$ /usr/local/bin/kmail -v
Qt: 4.7.0
KDE Development Platform: 4.5.95 (4.6 RC2)
KMail: 2.0.89
Comment 6 Murz 2011-01-21 09:15:31 UTC
*** Bug 259043 has been marked as a duplicate of this bug. ***
Comment 7 Murz 2011-01-21 09:17:15 UTC
Same problem if I save the message as draft, so the problem is in kmail message text generating process.
Comment 8 Murz 2011-01-21 09:41:48 UTC
Torgny Nyblom, can you on your version of Kmail2 create the message with fields that I describe in first post, and send it to me on murznn(at)gmail(dot)com? 
And save it as draft, save to file and attach to bug?
Comment 9 Murz 2011-01-21 09:44:58 UTC
Created attachment 56271 [details]
Message saved to drafts with wrong fields

I create the message that I describe in first post, save it to drafts and save to file, attaching this file to bug.
Comment 10 Murz 2011-01-21 09:46:54 UTC
Created attachment 56272 [details]
Message with manually corrected fields

I have manually correct the "From" and "To" fields in this file, and after that kmail2 opens and displays it correctly.
Comment 11 Murz 2011-02-04 07:49:59 UTC
I try to fresh install Kubuntu and OpenSUSE with KDE 4.6 and KDEPIM 4.5.94.1 with default settings, and can reproduce this bug:

http://wstaw.org/m/2011/02/04/plasma-desktopbm2641.jpg

Language is set to Englins, locale settings:
> locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

How to reproduce:
1. Open kmail with default settings.
2. Press "New", fill in "To" field text "Вася Пупкин <vasya@example.com>"
3. Press "File -> Save as draft".
4. Open this message in "drafts" folder. You will see "Вася Пупкин <vasya@example.com>" in message list, but in message body window you will see "???? ?????? <vasya@ya.ru>".

Here is the screenshot: http://wstaw.org/m/2011/02/04/plasma-desktopbm2641.jpg
Comment 12 Murz 2011-02-21 08:58:35 UTC
I see that all headers kmail must convert to latin1 encoding via some function.
And, I think, kmail2 miss this converting when save "From" and "To" fields, because after save message as draft, I see in akonadi database field "Subject" converted normally (=?UTF-8?B?0JDQu9C10LrRg....) but From and To contains ??? without "=?UTF-8?" prefix.

Can anybody tell me the name of the function that do this converting?
From: Алексей Test Корепов <mymail@mail.ru>
converts to: 
From: =?UTF-8?B?0JDQu9C10LrRgdC10LkgVGVzdCDQmtC+0YDQtdC/0L7Qsg==?=
<mymail@mail.ru>

And the name of the .cpp or .h file where this convertion is occurs on "Save as draft" message action?
Comment 13 eric 2011-02-23 01:17:57 UTC
I have a problem that looks like this.

I'm using Kmail-2.0.89
When I create a new e-mail and use Chinese characters in the 'From' field the resulting e-mail will not have the 'From' header. I don't have "???" the e-mail just misses the From header. In the Messagelist in the outbox or draft folder this new e-mail will have "Unknown" as sender.
Like Murz said, the 'Subject' and other headers (like 'Organization') is showing the non-ascii characters correctly.

This bug may be related: https://bugs.kde.org/show_bug.cgi?id=266608
Comment 14 Murz 2011-03-18 07:22:57 UTC
This bug is reproducible on fresh install of Kubuntu 10.10 with daily nighly version (from project-neon) and on OpenSUSE.

How to reproduce (the easiest way):

1. Open Kmail, "New message".
2. Fill "to" with text "Вася <vasya@example.com>"
3. Press "Save as Draft".
4. Go to "Drafts" folder and open the message in new window.

You will see "???? <vasya@example.com>" instad of "Вася <vasya@example.com>"
Comment 15 Cristian Oneț 2011-03-19 11:37:57 UTC
I have exactly the same proble with kmail 2.0.89.
Comment 16 Cristian Oneț 2011-03-24 20:33:02 UTC
This bug seems fairly well documented. Does the development team need any more info?
Comment 17 Cristian Oneț 2011-03-26 13:24:47 UTC
Created attachment 58353 [details]
This patch is just a dirty hack that fixes the problem for the 'from' field

It also highlights the problem that is causing the bug.
When KMime::Message::assembleHeaders() is called the header object contains an UTF-8 string but it's RFC2047 charset is set to ISO 8859-1 (Latin 1) causing the bad serialization as a 7bit string.
The quick hack is to set the RFC2047 charset to UTF-8 of the 'from' header element before serializing the header.
Comment 18 Murz 2011-04-11 21:19:45 UTC
Created attachment 58808 [details]
Patch for convert From To CC BCC fields to UTF-8

Cristian Onet, many thanks for patch! It helps.
But the problem is not only with "From" field. To, CC, BCC fields have this problem too.
I extend this patch for convert those fields too.
Comment 19 Murz 2011-04-11 21:21:15 UTC
Kubuntu users can use my PPA for install patched KDEPIM with this issue solved: https://launchpad.net/~murznn/+archive/kdepim-4.6-git+patches
Comment 20 Cristian Oneț 2011-04-11 21:26:51 UTC
(In reply to comment #18)
> Created an attachment (id=58808) [details]
> Patch for convert From To CC BCC fields to UTF-8
> 
> Cristian Onet, many thanks for patch! It helps.
> But the problem is not only with "From" field. To, CC, BCC fields have this
> problem too.
> I extend this patch for convert those fields too.

I made the patch just to highlight the problem and didn't add the To, CC, BCC fields because I don't consider that the problem should be fixed at that point. I am waiting for the kdepim developers to fix this properly.
Comment 21 Christophe Giboudeaux 2011-04-12 10:33:30 UTC
To make the review process easier, please post your patch on https://git.reviewboard.kde.org (group 'kdepim')
Comment 22 Cristian Oneț 2011-04-13 12:03:31 UTC
(In reply to comment #21)
> To make the review process easier, please post your patch on
> https://git.reviewboard.kde.org (group 'kdepim')

Further discussion at https://git.reviewboard.kde.org/r/101109/
Comment 23 Thomas McGuire 2011-04-16 22:54:47 UTC
Git commit 7b75f8a769ef0e40e59dedb1327ff0ed51e63ba9 by Thomas McGuire.
Committed on 16/04/2011 at 22:56.
Pushed by tmcguire into branch 'master'.

Use UTF-8 if the charset can't encode the string.

Hopefully the call to canEncode() won't affect the performance too
much...

BUG: 263761

M  +5    -0    kmime/kmime_util.cpp     
M  +1    -1    kmime/kmime_util.h     

http://commits.kde.org/kdepimlibs/7b75f8a769ef0e40e59dedb1327ff0ed51e63ba9
Comment 24 Thomas McGuire 2011-04-17 14:42:05 UTC
Git commit 6d80788587894ee0cf3e087f959226a3d764af95 by Thomas McGuire.
Committed on 16/04/2011 at 22:56.
Pushed by tmcguire into branch '4.6'.

Use UTF-8 if the charset can't encode the string.

Hopefully the call to canEncode() won't affect the performance too
much...

BUG: 263761
(cherry picked from commit 7b75f8a769ef0e40e59dedb1327ff0ed51e63ba9)

M  +5    -0    kmime/kmime_util.cpp     
M  +1    -1    kmime/kmime_util.h     

http://commits.kde.org/kdepimlibs/6d80788587894ee0cf3e087f959226a3d764af95
Comment 25 Murz 2011-04-18 06:42:37 UTC
I have tested this bug in kmail2 version from GIT from 2011-04-17 - all works good with russian letters, thanks!
Comment 26 Murz 2011-04-18 07:37:31 UTC
I have found and report another bug with mail headers: https://bugs.kde.org/show_bug.cgi?id=271192
Maybe it is related to this bug, can you look on it?
Comment 27 eric 2011-04-24 18:58:28 UTC
It's not fixed!!! ... when the name & e-mail address is a bit different.

Repeating the testcase from comment #14 after applying the patch from comment #24:
To: Вася <vasya@example.com>          ---> works
To: "Вася" <vasya@example.com>        ---> works
To: "Вася bla" <vasya@example.com>    ---> does not work
To: Вася bla <vasya@example.com>      ---> works
To: "Вася Вася" <vasya@example.com>   ---> works

Maybe a related issue, entering the string "Вася" in KJots (or Kontact's notebook) then quitting and restarting KJots/Kontact: the string will be "????".
Comment 28 Christophe Giboudeaux 2011-04-24 19:46:13 UTC
(In reply to comment #27)
> It's not fixed!!! ... when the name & e-mail address is a bit different.
> 
Did you build from sources with a recent kdepimlibs git checkout ?
Comment 29 eric 2011-04-25 01:16:30 UTC
(In reply to comment #28)
> Did you build from sources with a recent kdepimlibs git checkout ?

No, I've applied the patches in comment #24 on kdepimlibs-4.6.2 and then recompiled.
But if it is working on the most recent git version then version 4.6.3 will be a good version.
Comment 30 Murz 2011-04-26 06:43:55 UTC
I test those examples on kdepim from today git:

To: Вася <vasya@example.com>          ---> works
To: "Вася" <vasya@example.com>        ---> works
To: "Вася bla" <vasya@example.com>    ---> works
To: Вася bla <vasya@example.com>      ---> works
To: "Вася Вася" <vasya@example.com>   ---> works

And can't reproduce this issue, maybe you must apply some other patches from git. What version of kmail did you use?
Comment 31 eric 2011-04-26 14:08:08 UTC
(In reply to comment #30)
> And can't reproduce this issue, maybe you must apply some other patches from
> git. What version of kmail did you use?

Glad to hear that it is working, I have a lot of contacts with non-ascii names.

Okay, maybe there are other patches in the git-version that I need to apply. I will wait for version 4.6.3.
I'm using Kmail/Kontact 4.6beta5 (4.5.95).

Sorry for causing worry.
Comment 32 Murz 2011-04-26 16:07:35 UTC
Kmail/Kontact 4.6beta5 (4.5.95) must already have this patch integrated, because it committed before beta5 release. Can you describe your problem more? 
Is it reproducible with only "To" field or "From", "CC" too? When you save as draft and open message again, you see ??? or normal symbols? You can mail me to murznn (at) gmail.com with detailed description of problem, I try to reproduce it, maybe some part of bug is not fixed.
Comment 33 eric 2011-04-27 02:15:12 UTC
(In reply to comment #32)
> Kmail/Kontact 4.6beta5 (4.5.95) must already have this patch integrated,
> because it committed before beta5 release.

I think the patch in comment #24 is for the package 'kdepimlibs', while Kontact/Kmail are applications from the package called 'kdepim'. The functionality of kdepim depends on kdepimlibs, but the patch is for kdepimlibs.

> Can you describe your problem more? 
> Is it reproducible with only "To" field or "From", "CC" too? When you save as
> draft and open message again, you see ??? or normal symbols? You can mail me to
> murznn (at) gmail.com with detailed description of problem, I try to reproduce
> it, maybe some part of bug is not fixed.

To/From/CC: "Вася bla" <vasya@example.com>   ---> bad: "???? bla" <vasya@example.com>
To/From/CC: Вася bla <vasya@example.com>     ---> good
To/From/CC: "Вася Вася" <vasya@example.com>  ---> good
To/From/CC: Вася <vasya@example.com>         ---> good
Subject and body are always correct.

The reason it's working for you and not for me is probably because of the other changes in the git-version. Version 4.6.3 will be out soon, then I'll test again.
Comment 34 Christophe Giboudeaux 2011-05-02 16:02:16 UTC
*** Bug 242231 has been marked as a duplicate of this bug. ***
Comment 35 eric 2011-05-06 19:00:47 UTC
As expected, fixed in kde-4.6.3.