Bug 10519 - KMail displays non-iso8859 characters as question mark
Summary: KMail displays non-iso8859 characters as question mark
Status: CLOSED FIXED
Alias: None
Product: kmail
Classification: Applications
Component: general (show other bugs)
Version: unspecified
Platform: unspecified Other
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2000-09-13 21:18 UTC by Unknown
Modified: 2008-11-23 15:52 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nikita V. Youshchenko 2000-09-13 21:08:34 UTC
(*** This bug was imported into bugs.kde.org ***)

Package: kmail
Version: from Sep 12 CVS also some earlier versions

I live in Russia and use Russian mail.

In KMail from recent KDE2 CVS I can't read russian both in subjects and
bodies of the messages - there are all displayed as question marks.

Yes I have russian locale set up correctly and Russia and Russian
language are selected in KDE.

An example russian mail that reproduces the error is included below.

The possible reason of the bug is some misuse of unicode QStrings in
KMail - some conversion from 8-bit to QString is incorrect. Perhaps
fromLocal8Bit() should be used ?..

Example mail:
-----
From kde2@sercond Tue Sep 12 23:31:40 2000
Envelope-to: kde2@sercond
Received: from kde2 by sercond with local (Exim 3.12 #1 (Debian))
        id 13YvmO-0001Wo-00
        for <kde2@sercond>; Tue 12 Sep 2000 23:31:40 +0400
To: kde2@sercond
Subject: ôÃÃÃÃÃÃà ÃÃÃÃÃÃÃÃÃ
Message-Id: <E13YvmO-0001Wo-00@sercond>
From: KDE2 test account <kde2@sercond>
Date: Tue 12 Sep 2000 23:31:40 +0400
Status: R
X-Status: N

ôÃÃÃÃÃÃà ÃÃÃÃÃÃÃÃÃ
-----
Comment 1 Don Sanders 2000-09-13 21:40:39 UTC
I can read it with KMail 1.1.94 fine here the subject and the body of the 
message say the same thing I can't read russian but it looks rough like the 
following

TECTOBOE COO6WEHHE

Well the E's look more like e's and the second H looks more like a backwards 
N this is the best I could do using English letters.

KMail might rely on the content-type being set correctly. The following 
content-type is included in your message:
  Content-Type: text/plain; charset=koi8-r

But no content-type is specified in the message you quote.

BFN
Don.

On Wed 13 Sep 2000 Nikita V. Youshchenko wrote:
> Package: kmail
> Version: from Sep 12 CVS also some earlier versions
>
> I live in Russia and use Russian mail.
>
> In KMail from recent KDE2 CVS I can't read russian both in subjects and
> bodies of the messages - there are all displayed as question marks.
>
> Yes I have russian locale set up correctly and Russia and Russian
> language are selected in KDE.
>
> An example russian mail that reproduces the error is included below.
>
> The possible reason of the bug is some misuse of unicode QStrings in
> KMail - some conversion from 8-bit to QString is incorrect. Perhaps
> fromLocal8Bit() should be used ?..
>
> Example mail:
> -----
>
> >From kde2@sercond Tue Sep 12 23:31:40 2000
>
> Envelope-to: kde2@sercond
> Received: from kde2 by sercond with local (Exim 3.12 #1 (Debian))
>         id 13YvmO-0001Wo-00
>         for <kde2@sercond>; Tue 12 Sep 2000 23:31:40 +0400
> To: kde2@sercond
> Subject: ôÃÃÃÃÃÃà ÃÃÃÃÃÃÃÃÃ
> Message-Id: <E13YvmO-0001Wo-00@sercond>
> From: KDE2 test account <kde2@sercond>
> Date: Tue 12 Sep 2000 23:31:40 +0400
> Status: R
> X-Status: N
>
> ôÃÃÃÃÃÃà ÃÃÃÃÃÃÃÃÃ
> -----
> _______________________________________________
> Kmail mailing list
> Kmail@master.kde.org
> http://master.kde.org/mailman/listinfo/kmail
Comment 2 Nikita V. Youshchenko 2000-09-14 04:54:43 UTC
Don Sanders wrote:
> 
> I can read it with KMail 1.1.94 fine here the subject and the body of the
> message say the same thing I can't read russian but it looks rough like the
> following
> 
> TECTOBOE COO6WEHHE
> 
> Well the E's look more like e's and the second H looks more like a backwards
> N this is the best I could do using English letters.

It is correct :-).  But the bug report was composed in Netscape mail ...

> KMail might rely on the content-type being set correctly. The following
> content-type is included in your message:
>   Content-Type: text/plain; charset=koi8-r
> 
> But no content-type is specified in the message you quote.


Unfortunately there ARE plenty of mail clients around that DON'T add
this header. Actually KMail itself is one of those :-( ...

Until people who write me mail will use such clients I DO NEED to be
able to read such mail and KMail that can't display it is useless for
me.

If KMail is intended to be usefull outside english-speaking countries
KMAIL SHOULD ***NOT*** DEPEND ON CORRECTNESS of charset= header !  (and
also it is a good idea to set the appropriate header in mail composed by
KMail itself)

At least there should be way to set "the default" encoding.  And why
not use the local encoding as default (that is - when no encoding is
selected by charset=) ?  Using QString::fromLocal8Bit() will give this
result ...

Nikita
Comment 3 Andreas Gungl 2000-09-14 07:26:41 UTC
On Don 14 Sep 2000 Nikita V. Youshchenko wrote:
> Don Sanders wrote:
> > I can read it with KMail 1.1.94 fine here the subject and the body of
> > the message say the same thing I can't read russian but it looks rough
> > like the following
> >
> > TECTOBOE COO6WEHHE
> >
> > Well the E's look more like e's and the second H looks more like a
> > backwards N this is the best I could do using English letters.

Well I can read it after setting "Message | Set Encoding" to koi-8r.
The letters are okay and as far as I remember to the days I have learned=
=20
Russian at school a long time ago it means something like
"test message". (?)

> It is correct :-).  But the bug report was composed in Netscape mail ...
>
> > KMail might rely on the content-type being set correctly. The following
> > content-type is included in your message:
> >   Content-Type: text/plain; charset=3Dkoi8-r
> >
> > But no content-type is specified in the message you quote.
>
> Unfortunately there ARE plenty of mail clients around that DON'T add
> this header. Actually KMail itself is one of those :-( ...

This is a known problem AFAIK somebody is woking on it - might be solved=
=20
soon. (At least I hope so.)

> Until people who write me mail will use such clients I DO NEED to be
> able to read such mail and KMail that can't display it is useless for
> me.

Can't you change the encoding scheme like I did? Where is the problem with=
=20
that solution?

Andreas
Comment 4 dyp 2000-09-14 07:33:05 UTC
Hello

> I just recently installed this font from my Windows 98 using DrakeFont.
> Problem is that I can't change "Location: Quoted Text - First Level"
> There is no choice only Helvetica.
> I suggest it's a bug.
> 
> On top of Code Page issues. KOI-8r is old aged standard. It's much better to 
> use CP1251 (and CP1253 for Greek etc.). As far as I understand standard 
> algorith in MS Exchange and Outlook is following:

Hey... Where are you from? Looks like you just started using computer...
FYI: KOI8-R is an internet standard for mail exchange in russian. That is all.
Do not try to persuade people who do not know about this (I mean developers
of KMail) that CP1251 is "new" standard... It is a "new" one for Windowz people.
Let them use it... We do not use it anywhere. You will hardly find mail in CP1251
in Internet. And if you will it mean that it is broken as it is not RFC compliant.

> 1) get message encoding (say KOI-8r)
> 2) compare with standard in Windows (CP1251)
> 3) translate charactes from KOI-8R to CP1251
> 4) store message in Inbox list (CP1251 coding)
> Besides many sys Admins make following setup for Exchange when setting 
> outgoing mail out of company intranet;
> 1) translate message text from CP1251 to KOI-8R
> 2) sent message using KOI-8R encoding
> 3) store message in Sent Mail folder using CP1251 coding

Just for your info... In Russia MS Exchange counts not more than 5% of mail
servers... Most of the mail servers are Unix-based...
 
> All mailing inside office uses CP1251 only.
> For outside world it looks like only KOI-8R is used :-). Just for 
> compatibility questions with old MS-DOS 

Hey... Did you EVER worked on MS-DOS?????? MS-DOS uses CP866
but not KOI8-R... And mails goes outside in KOI8-R because they SHOULD
be RFC compliant. Just for you... KOI8-R is used on "old" Unix platforms.
Linux for example.
 
> ****
> Another problem which is not related to KDE but also important. Standard 
> XFonts (TTF PS Type 1) which included in distribution do not support 
> Cyrillic. Neither Adobe URW or whatever.
> How normal user will know that he needs to install fonts? And where to get 
> them?

Sorry but you are completely wrong. Standard XFree86 contains all neccessary
fonts to operate. And this is your fault that you can not make them working (and KDE
sometimes).

In this case with KMail it is the fault of KMail author that he always uses ISO-8859-1 encoding
for KHTML. I already discussed this with him but it was not not fixed yet.

And you get the result because you have used TTF fonts with incorrect encoding: iso-8859-1.
Of cause such fonts are not supplied with XFree86 as the correct ones have koi8-r.

-- 
Sincerely Yours
Denis Perchine

----------------------------------
E-Mail: dyp@perchine.com
HomePage: http://www.perchine.com/dyp/
FidoNet: 2:5000/120.5
----------------------------------
Comment 5 Michael Haeckel 2000-09-14 07:51:12 UTC
On Thursday 14. September 2000 Nikita V. Youshchenko wrote:
>
> At least there should be way to set "the default" encoding.  And why
> not use the local encoding as default (that is - when no encoding is
> selected by charset=3D) ?  Using QString::fromLocal8Bit() will give this
> result ...

There is a way in the "Message" menu: "Set Encoding".

Regards
Michael H=E4ckel
Comment 6 Don Sanders 2000-09-14 10:15:53 UTC
> In this case with KMail it is the fault of KMail author that he always uses
> ISO-8859-1 encoding for KHTML. I already discussed this with him but it
> was not not fixed yet.

It IS fixed now. We received a patch from Igor Yanssen.

The reader window show support non-latin based languages fine now. I am 
having trouble with CJK but this may be because I can't convince my true 
type font server xfstt to support CJK true type fonts.

We still have a couple of problems:
1) The list of message headers isn't translated Igor Yanssen fixed this but 
his patch was too inefficient.

2) Composing doesn't work.

BFN
Don.
Comment 7 Vadim Plessky 2000-09-14 10:59:56 UTC
Well I can read messages sent to me in Russian from Windows. Nikita if you 
want - you can try to send me... I will reply :-)
Problem comes when I make "Reply" Message Text in Russian becomes absolutely 
unreadable.

I have checked configuration.
In Settings->Configuration->Appearance->Fonts
I selected Arial and it has CharacterSetiing KOI-8R and KOI-8U.

I just recently installed this font from my Windows 98 using DrakeFont.
Problem is that I can't change "Location: Quoted Text - First Level"
There is no choice only Helvetica.
I suggest it's a bug.

On top of Code Page issues. KOI-8r is old aged standard. It's much better to 
use CP1251 (and CP1253 for Greek etc.). As far as I understand standard 
algorith in MS Exchange and Outlook is following:
1) get message encoding (say KOI-8r)
2) compare with standard in Windows (CP1251)
3) translate charactes from KOI-8R to CP1251
4) store message in Inbox list (CP1251 coding)
Besides many sys Admins make following setup for Exchange when setting 
outgoing mail out of company intranet;
1) translate message text from CP1251 to KOI-8R
2) sent message using KOI-8R encoding
3) store message in Sent Mail folder using CP1251 coding

All mailing inside office uses CP1251 only.
For outside world it looks like only KOI-8R is used :-). Just for 
compatibility questions with old MS-DOS 

****
Another problem which is not related to KDE but also important. Standard 
XFonts (TTF PS Type 1) which included in distribution do not support 
Cyrillic. Neither Adobe URW or whatever.
How normal user will know that he needs to install fonts? And where to get 
them?

Cheers

Vadim

On Thu 14 Sep 2000 Nikita V. Youshchenko wrote:
|  Don Sanders wrote:
|  > I can read it with KMail 1.1.94 fine here the subject and the body of
|  > the message say the same thing I can't read russian but it looks rough
|  > like the following
|  >
|  > TECTOBOE COO6WEHHE
|  >
|  > Well the E's look more like e's and the second H looks more like a
|  > backwards N this is the best I could do using English letters.
|
|  It is correct :-).  But the bug report was composed in Netscape mail ...
|
|  > KMail might rely on the content-type being set correctly. The following
|  > content-type is included in your message:
|  >   Content-Type: text/plain; charset=koi8-r
|  >
|  > But no content-type is specified in the message you quote.
|
|  Unfortunately there ARE plenty of mail clients around that DON'T add
|  this header. Actually KMail itself is one of those :-( ...
|
|  Until people who write me mail will use such clients I DO NEED to be
|  able to read such mail and KMail that can't display it is useless for
|  me.
|
|  If KMail is intended to be usefull outside english-speaking countries
|  KMAIL SHOULD ***NOT*** DEPEND ON CORRECTNESS of charset= header !  (and
|  also it is a good idea to set the appropriate header in mail composed by
|  KMail itself)
|
|  At least there should be way to set "the default" encoding.  And why
|  not use the local encoding as default (that is - when no encoding is
|  selected by charset=) ?  Using QString::fromLocal8Bit() will give this
|  result ...
|
|  Nikita
|  _______________________________________________
|  Kmail mailing list
|  Kmail@master.kde.org
|  http://master.kde.org/mailman/listinfo/kmail

-- 
Vadim Plessky
Comment 8 Don Sanders 2000-10-07 21:29:35 UTC
fixed
Comment 9 David Monniaux 2001-09-10 12:29:01 UTC
On the same machine same user but different X display (one with the RENDER 
extension the other without):
- with Xft: correct display of Japanese etc... characters in an UTF-8 encoded 
message
- without: a small number of squares appear.
Comment 10 Corrin Lakeland 2001-09-13 20:48:40 UTC
Hi

Firstly I don't read Russian but I'd rather see incomprehensible glyphs than 
question marks :-)

I recieved an email trying to sell me something.  In the main window the 
message is displayed correctly (well the subject is scaled poorly but that 
is almost certainly a problem with me not having a truetype russian font).

However in the message list the subject is ?????... looking at the file in 
a text editor the subject line is:

Subject: [I18n]=?koi8-r?B?0M/Nz8fJ1MUg09DB09TJIM3BzMXO2MvVwCDExdfP3svV?=

So clearly the kio8-r is supposed to tell KMail to swap to kio8-r encoding 
for this subject line which seems to be accepted when displaying the bulk of 
the message with standard headers but not the message list.

More seriously when I hit reply I get ? characters all through the body.  
Clearly I have kio8-r fonts installed or I wouldn't have been able to see the 
message but kmail has decided to use an 8859-1 font for replying.

This is running KMail 2.2 from the RedHat RPMs.

Corrin
Comment 11 Vadim Plessky 2001-09-15 01:02:28 UTC
On Thursday 13 September 2001 20:48 Corrin Lakeland wrote:
|   Hi
|
|   Firstly I don't read Russian but I'd rather see incomprehensible glyphs
| than question marks :-)
|

Hi Corrin!
I speak Russian and use Russin in KMail.
There is no known problem with it (neither body nor subject/name)

text in Russian
--------------------------
éÃÃÃÃÃÃÃà ÃÃÃà ÃÃÃÃÃÃÃÃÃÃÃÃÃÃà ÃÃà (ÃÃÃÃÃ) à ÃÃÃÃÃà ÷à ÃÃÃÃÃà ÃÃÃÃÃÃÃÃÃÃà à 
ftp-ÃÃÃÃÃÃÃÃ
---------------------------

|   I recieved an email trying to sell me something.  In the main window the
|   message is displayed correctly (well the subject is scaled poorly but
| that is almost certainly a problem with me not having a truetype russian
| font).
|

Most likely this message didn't have correct setting for encoding.
Force it (options-set encoding) either to koi8-r or to cp1251 or to UTF8.
One of them should work

P.S. This message I send in koi8-r

|   However in the message list the subject is ?????... looking at the file
| in a text editor the subject line is:
|
|   Subject: [I18n]=?koi8-r?B?0M/Nz8fJ1MUg09DB09TJIM3BzMXO2MvVwCDExdfP3svV?=
|
|   So clearly the kio8-r is supposed to tell KMail to swap to kio8-r
| encoding for this subject line which seems to be accepted when displaying
| the bulk of the message with standard headers but not the message list.
|
|   More seriously when I hit reply I get ? characters all through the body.
|   Clearly I have kio8-r fonts installed or I wouldn't have been able to see
| the message but kmail has decided to use an 8859-1 font for replying.
|
|   This is running KMail 2.2 from the RedHat RPMs.
|
|   Corrin

Here: KMail 1.2.3 from KDE 2.2
(LM 8.0)
-- 

Vadim Plessky
http://kde2.newmail.ru  (English)
33 Window Decorations and 6 Widget Styles for KDE
http://kde2.newmail.ru/kde_themes.html
Do you have Arial font installed? Just test it!
http://kde2.newmail.ru/font_test_arial.html
Comment 12 Médéric Boquien 2008-11-23 15:52:39 UTC
Add the new email of the reporter to the CC list, he cannot access the original one anymore.