Created attachment 117133 [details] Soliterm with cut off cards SUMMARY The unicode playing cards are not displayed correctly (displayed as single-width instead of double-width). The right half of the cards is cut off, and I haven't found a way to make the right half not be cut off. STEPS TO REPRODUCE 1. python3 -c 'print(chr(0x1F0A1))' OBSERVED RESULT left half of an 'Ace of Clubs' card, in a single cell on the text grid EXPECTED RESULT the whole 'Ace of Clubs' card, spanning 2 cells on the text grid SOFTWARE/OS VERSIONS Arch Linux Konsole Version: 18.12.0 KDE Libs version: 5.53 Qt Version: 5.12 ADDITIONAL INFORMATION in the attached screenshot, soliterm (github.com/Ferdi265/soliterm) is shown to illustrate the issue with playing cards
Created attachment 117134 [details] correct display of unicode playing cards in an older version of konsole (todo)
Last known version that works: 18.08.3 First known version that doesn't work: 18.11.80
According to the Unicode database, playing cards have 'Neutral' character width, and the Unicode standard UAX #11 recommends to display them as narrow characters. Database: http://www.unicode.org/Public/UCD/latest/ucd/extracted/DerivedEastAsianWidth.txt UAX: http://www.unicode.org/reports/tr11/
Okay. It still seems weird to cut them off like that. There should be a way to have them display correctly, since having them cut off is ugly. I think saying "playing cards are single-width, your font is broken" is a valid solution, though I've seen quite a few fonts with wider than normal playing card widths. For completeness, the used font is DejaVu Sans Mono
The actual problem is that Konsole needs to rely on the Unicode data instead of being able to query the font. There should be a way to support duospaced fonts, and Konsole would need a fast way to check if a character is single or double-celled in the font that Qt is choosing after font substitution. I am not even sure if freetype supports this property, or if font designers need to mark those fonts as variable-spaced, because they aren't exactly monospaced. https://en.wikipedia.org/wiki/Duospaced_font
There's at least 3 sides to this story: 1. IIRC there used to be a bug in Konsole, it accidentally treated all non-BMP characters (including the playing cards) as double wide ones. This was fixed relatively recently. I'm sorry but I couldn't quickly find the related bug. This bug might have easily led you into falsely assuming that these characters should take up 2 cells. 2. There's the desired _logical_ behavior: Which characters should occupy one logical cell and which characters should occupy two? E.g. if such a 🂱 character is printed from the left margin, followed by "ab", then the cursor positioned in the (1-based) 3rd column and the letter "x" is printed there, is it going to overwrite "a" or "b", that is, will the text be "🂱xb" (the card is _logically_ double wide) or "🂱ax" (the card is _logically_ single wide)? 3. There's the _visual_ behavior: what happens when a glyph is wider than its designated area (that is, 1 or 2 cells)? Should it push the rest of the line visually (but not logically) to the right (Konsole used to do this within runs of identical attributes, as far as I can tell), or cropped (apparently Konsole's new behavior), or shrinked (Kitty does this as far as I recall) or overflow to the next cell (e.g. VTE)? 3 is probably a matter of taste to some extent, every approach has pros and cons. As for 2: For proper positioning of items, e.g. the right border of soliterm, and for proper overwrites in ncurses and friends, it's crucial that the terminal and the app agrees on the logical width (although the former visual strategy of Konsole of pushing the rest of the line to the right isn't compatible with the expectation of the right border properly aligning up). If there's a misunderstanding, chances are that sooner or later everything falls apart big time on the screen. The basis for this is agreement the Unicode property. It would be nice if there was some improvement, e.g. if there was a way to negotiate the behavior between the terminal and the app, and maybe even allow to alter the behavior (I'm not even sure whether it should be the terminal or the app that should be able to initiate such a request; probably the app). If anything like this is done, it should be designed as a cooperation among some key players (i.e. popular terminal emulators), and ideally not something that Konsole (or any other emulator) implements as its own custom extension. The terminal emulator deciding the logical width based on the font doesn't sound like a good direction to me at all (imagine the debugging hell if every font results in a different logical behavior), and is certainly incompatible with headless emulators or ones that can have multiple views (like libvterm, screen, tmux; also konsole if it allowed different fonts in its split view).
For bullet point 1: bug 399795 is what I was looking for.
Created attachment 117139 [details] single width character as double width character demo I've made a small demo. Konsole assumes ⛩ has width 2, zsh assumes it has width 1. What happens: * The text is pasted - konsole writes it to zsh input buffer and on the screen. Konsole places cursor after pasted text - 10 cells from the beginning. * Backspace is pressed. For each press zsh removes last character from its internal buffer, moves cursor 1 cell to the left (remember that for zsh each used character takes 1 cell), and clears the cell. * When all 9 characters are removed from the buffer, it ignores further backspace presses, as there is nothing more to remove. The cursor moved 9 cells to the left. * 1 cell is not reachable - zsh assumes it moved to the starting position. In second case 5 cells are cleared. For konsole the last cleared cell is an unused cell, because the one to the left has double character in it. ⛩ is still on the screen, but not in zsh buffer. This is visible after zsh redraw the text using its buffer. The problem does not exist when the program and Konsole agree about the width. I understand people (including me) want to use various emojis and even custom images stored in private use area, so I want to expand clipping area of all characters to 2 cells, but only if the next cell is empty (has space). This way: * To use single-width character with double-width image, you'll type "⛩ ". Konsole will draw the character on top of space, but will still assume there is the single-width character and a space. The programs will assume the same. * Regular characters just after too wide character won't be overpainted (as it happens in e.g. vim when clipping is off). In this case the too wide character will be clipped as it happens now. This won't help with displaying text files (unless there are spaces after emojis) or other text not controlled by the user, but will allow to use full characters in prompts or scripts. I hope this is primary use case of graphical characters in a terminal.
I'm fairly sure this is a regression, it seems to be reproducible with several of the test cases in the tests/ folder in the konsole repo.