Bug 335079 - Text rendering broken for unicode characters
Summary: Text rendering broken for unicode characters
Status: RESOLVED FIXED
Alias: None
Product: frameworks-ktexteditor
Classification: Frameworks and Libraries
Component: general (show other bugs)
Version: 5.1.0
Platform: Fedora RPMs Linux
: VHI major
Target Milestone: ---
Assignee: KWrite Developers
URL:
Keywords:
: 347474 353730 357582 359742 360302 360555 362271 364236 365116 (view as bug list)
Depends on:
Blocks:
 
Reported: 2014-05-20 12:02 UTC by dizieg
Modified: 2020-07-02 23:31 UTC (History)
14 users (show)

See Also:
Latest Commit:
Version Fixed In: KDE Frameworks 5.27


Attachments
Screenshot demonstrating the problem in kwrite 15.12.3 (88.98 KB, image/jpeg)
2016-05-10 12:38 UTC, Riskable
Details
screenshot of the workaround (214.66 KB, image/png)
2016-09-07 23:27 UTC, Bernhard Scheirle
Details
screenshot of commit 4c83bac52716a08a549dca50a76be8aa47c11ca3 (210.74 KB, image/png)
2016-09-08 15:10 UTC, Bernhard Scheirle
Details

Note You need to log in before you can comment on or make changes to this bug.
Description dizieg 2014-05-20 12:02:33 UTC
The underscore character(_) is invisible in Monospace font with size: 10.5 . I can observe this in Konsole and Kwrite. 

Reproducible: Always
Comment 1 Louis Moureaux 2014-07-01 13:58:35 UTC
Works for me using Project Neon packages. I had to close and reopen Kate.
Comment 2 dizieg 2015-05-30 10:17:41 UTC
Still an issue in 5.1
Comment 3 Christoph Cullmann 2015-10-09 08:20:56 UTC
Works for me with current kate master.
Comment 4 Dominik Haumann 2015-10-09 11:35:11 UTC
It's a problem with the font, and still exists. It's mostly the fault of the font, since it paints in regions of the next line, and when the next line is painted, the previously painted underscore is cleaned again.

A proper fix is only to change the way Kate does the painting:
First paint the backgrounds of all lines in the view, then paint dedicated highlighting backgrounds, then paint the text, so a three-pass painting. This is also what Qt Creator does, and there is no other way to properly fix this.

"Works for me" just means that you probably didn't test with other font sizes and/or other fonts.
Comment 5 Christoph Cullmann 2015-10-09 11:57:15 UTC
We can't fix that even with that kind of painting, as then, we would always need to repaint the complete screen. If one line can overlap the next, if I update then line x I need to least repaint all from 0 to x, a they might overlap into it, and perhaps even the ones below, as I might overlap to the next and need to clear that first to have no artefacts.
Comment 6 Louis Moureaux 2015-10-09 17:44:29 UTC
What about drawing half of the line above the current one, and half of the line below it ? Fonts most probably don't draw below the upper half of the next line. If you really want to draw as little as possible, QFontMetrics::boundingRect is the way to go.
Note that this issue affects many Arabic characters in the (default) Oxygen font, such as و (U+0648).
Comment 7 Dominik Haumann 2016-03-24 16:18:11 UTC
*** Bug 353730 has been marked as a duplicate of this bug. ***
Comment 8 Dominik Haumann 2016-03-24 16:18:44 UTC
*** Bug 347474 has been marked as a duplicate of this bug. ***
Comment 9 Dominik Haumann 2016-03-24 16:32:38 UTC
*** Bug 360555 has been marked as a duplicate of this bug. ***
Comment 10 Dominik Haumann 2016-03-24 16:34:52 UTC
*** Bug 359742 has been marked as a duplicate of this bug. ***
Comment 11 Dominik Haumann 2016-03-24 16:37:26 UTC
Still an issue, and definitely valid. One day we need to fix this through multi-pass painting (first background, then other overlays, and latest text). Needs large change in the renderer, though.
Comment 12 Louis Moureaux 2016-03-24 16:50:11 UTC
Regarding comment in the duplicates:
This is not a problem in the fonts. It is normal that some characters extend above the ascent or below the descent, because these are defined for a "typical" character. You'll find that most fonts have characters extending above their ascent (look at the range U+00C0 to U+00DD).
The Java FontMetrics class got it right: it defines getAscent and getMaxAscent. Some characters may extend above the former, but the second is an absolute limit. I didn't find an equivalent in QFontMetrics.
Comment 13 Zhang Jingqiang 2016-03-26 11:15:47 UTC
(In reply to Dominik Haumann from comment #11)
> Still an issue, and definitely valid. One day we need to fix this through
> multi-pass painting (first background, then other overlays, and latest
> text). Needs large change in the renderer, though.

How about just adding some space that user can config when needed between lines as a workaround?
Comment 14 Dominik Haumann 2016-03-26 14:44:17 UTC
Yes, a "line spacing" option would probably work (but may be visually ugly).
Comment 15 Riskable 2016-05-10 12:38:29 UTC
Created attachment 98874 [details]
Screenshot demonstrating the problem in kwrite 15.12.3

This screenshot demonstrates the missing underscore problem in kwrite 15.12.3 (KDE Framework 5.18.0, QT 5.5.1, xcb windowing) on Kubuntu 16.04.
Comment 16 Riskable 2016-05-10 12:45:41 UTC
I just want to note that changing from Dejavu Sans Mono (11pt) to Ubuntu Mono (12pt) resolved the problem for me for plain text but rendering of Unicode characters on a line can still cause underscores to go missing (among other Unicode characters).  Here's some examples where Unicode characters and underscores will go missing:

ಠ︵ಠ
⊙︿⊙
◉︵◉

Coincidentally, those emoji represent how I feel after encountering this bug ಠ_ృ
Comment 17 Dominik Haumann 2016-06-13 21:26:50 UTC
*** Bug 360302 has been marked as a duplicate of this bug. ***
Comment 18 Dominik Haumann 2016-06-13 21:27:11 UTC
*** Bug 357582 has been marked as a duplicate of this bug. ***
Comment 19 Dominik Haumann 2016-06-13 21:27:29 UTC
*** Bug 362271 has been marked as a duplicate of this bug. ***
Comment 20 Dominik Haumann 2016-06-13 21:28:06 UTC
*** Bug 364236 has been marked as a duplicate of this bug. ***
Comment 21 Christoph Cullmann 2016-09-07 19:55:27 UTC
*** Bug 365116 has been marked as a duplicate of this bug. ***
Comment 22 Christoph Cullmann 2016-09-07 21:21:40 UTC
Git commit 2f31454eede42eb99fd8e95834a11397de98ea1e by Christoph Cullmann.
Committed on 07/09/2016 at 21:21.
Pushed by cullmann into branch 'master'.

try to "fixup" rendering of underlines and stuff like that

KTextEditor assumes equal line height, this leads to problems for _ and stuff like that

"fix": spread lines by factor 1.1

please test this, looks "ok" here and avoid the problems (tested with some fonts + CTRL--/+ zooming

if that is "unusable", we can revert it before 5.27

M  +9    -3    src/render/katerenderer.cpp

http://commits.kde.org/ktexteditor/2f31454eede42eb99fd8e95834a11397de98ea1e
Comment 23 Bernhard Scheirle 2016-09-07 23:27:19 UTC
Created attachment 100981 [details]
screenshot of the workaround

Hmm…, I do not like the 1.1 scaling factor. It's quite noticeable and wastes a lot of space.
(True, for some cases this avoids this bug.)

The attached screenshot compares the input of Bug 365116 with and without this workaround (for two different fonts).
As noted in the commit itself this is not a proper fix, and the lines still get cut of as seen in line 4 of the screenshot (underscore not visible).
Therefore I think this Bug should be reopened.


Input:
abcdefghijklmnopqrstuvwxyzäöü_
abcdefghijklmnopqrstuvwxyzäöü_⽳
abcdefghijklmnopqrstuvwxyzäöü_
abcdefghijklmnopqrstuvwxyzäöü_ẞ
abcdefghijklmnopqrstuvwxyzäöü_
abcdefghijklmnopqrstuvwxyzäöü_–
abcdefghijklmnopqrstuvwxyzäöü_
abcdefghijklmnopqrstuvwxyzäöü_—
Comment 24 Eric S 2016-09-08 00:37:06 UTC
Isn't the underlying problem here the painting of text using an opaque background? Wouldn't it be better painting it with a transparent background (mask or such)?
Comment 25 Christoph Cullmann 2016-09-08 06:29:27 UTC
Even with non-opaque background, we will cut of parts on last lines on widget.
But as said, this was the first quick hack :/ More versatile patches are welcome.
Comment 26 Christoph Cullmann 2016-09-08 13:26:55 UTC
The 1.1 line spreading hack has more problems, we are working on the problem:

https://phabricator.kde.org/D2706

This is a solution without any such scaling.
Comment 27 Christoph Cullmann 2016-09-08 14:07:07 UTC
Git commit 4c83bac52716a08a549dca50a76be8aa47c11ca3 by Christoph Cullmann.
Committed on 08/09/2016 at 14:04.
Pushed by cullmann into branch 'master'.

implement better rendering to allow e.g. _ to span into the next line

CHANGELOG: Fix rendering of characters spanning into next line, e.g. underlines are no longer cut off with some fonts/font-sizes

Differential Revision: https://phabricator.kde.org/D2706

M  +4    -24   src/render/katerenderer.cpp
M  +25   -20   src/view/kateviewinternal.cpp

http://commits.kde.org/ktexteditor/4c83bac52716a08a549dca50a76be8aa47c11ca3
Comment 28 Christoph Cullmann 2016-09-08 14:09:10 UTC
=> new fix won't lead to screen space waste + avoids some ugly things we had as rendering artifacts since years
Comment 29 Louis Moureaux 2016-09-08 14:34:46 UTC
It doesn't look like your patch solves Bug #353730 (characters extending above the line) despite it being marked as a duplicate of this one.
Comment 30 Bernhard Scheirle 2016-09-08 15:10:06 UTC
Created attachment 100987 [details]
screenshot of commit 4c83bac52716a08a549dca50a76be8aa47c11ca3

For me commit 4c83bac52716a08a549dca50a76be8aa47c11ca3 results in a way better rendering using the font Hack, but still not 100% correct (slightly overlapping lines [line 4/5 character j ]).

But using the font Courier New now the lines overlap strongly (even the line numbers).

Also independent of the font the last line in the document is still horizontally cut off (If it contains a tall character).

Input:
# <version>
# <font>
abcdefghijklmnopqrstuvwxyzäöü_
abcdefghijklmnopqrstuvwxyzäöü_⽳
abcdefghijklmnopqrstuvwxyzäöü_
abcdefghijklmnopqrstuvwxyzäöü_ẞ
abcdefghijklmnopqrstuvwxyzäöü_
abcdefghijklmnopqrstuvwxyzäöü_–
abcdefghijklmnopqrstuvwxyzäöü_
abcdefghijklmnopqrstuvwxyzäöü_—

pppp 嗨,你好吗?pppp
我很好。pppp
如何凯特呈现此文字?pppp
pppp


Hello, how are you?
I'm fine
吗pppp
Comment 31 Christoph Cullmann 2016-09-08 15:37:46 UTC
I doubt we can fix that for fonts that have this weired height() behavior in QFontMetrics.
I have seen such issues in pure Qt applications too in the past and beside "use some reasonable font" I have no solution.
Comment 32 Dominik Haumann 2016-11-21 21:52:11 UTC
Hm, can we may use the maximum of all tested heights again like it previously was?

Or was there an issue with that as well?
Comment 33 avlas 2017-02-26 14:44:05 UTC
Please see https://bugs.kde.org/show_bug.cgi?id=376565

This bug affects all KDE applications, not only Kate/Kwrite and related...

I had to decrease the font size to workaround the issue, but now it looks too small. Is there a way to make the fix to apply to a bit bigger fonts?

I have a 2560x1440 screen and force 144 dpi font size with default font size to 11 px. In that condition auto spell checking is not visible, I need to decrease it to <= 10 px, which is too small for my system configuration, but for now at least I recover auto spell checking... :-/