Executing the `clear` command clears the current view and leaves the prompt/cursor at the top of the window, but also adds the remaining number of empty lines to the scrollback buffer. IOW, if you execute it when the cursor is already at the top of a view that's N lines high, you get N empty lines in the scrollback buffer. That's counterproductive. The clear manpage states that the command `clears your screen if this is possible, including its scrollback buffer (if the extended "E3" capability is defined). clear looks in the environment for the terminal type and then in the terminfo database to determine how to clear the screen`. I'm not certain we'd want the scrollback clearing but there must be a way to avoid the empty lines to it. FWIW, I use TERM=xterm, and I note that xterms (v297) behave as I'd expect (`clear` doesn't affect the scrollback buffer contents at all).
Just FYI: gnome-terminal suffers from the same problem, see https://bugzilla.gnome.org/show_bug.cgi?id=506438. I assume gnome-terminal copied konsole's behavior (or the other way around). Back in the days when the scrollback wasn't wiped out, this behavior made more sense (everything was still available in the scrollback, rather than a pageful of data going missing). With the introduction of E3 this probably no longer makes too much sense. (On the other hand, see also https://bugzilla.gnome.org/show_bug.cgi?id=771953.)
Personally I'd guess this is just a side-effect of what Konsole can see of whatever the clear command sends to the tty. I'd hope that both the Konsole and gnome-term authors sat back to consider whether the other(s) implemented a feature the way they'd like to see it before copying it! ;) I think there are at least 2 levels at which an attempt can be made to improve this. Currently Konsole emulates well-known terminals, but it could very well define its own terminal type with the accompanying termcap/terminfo file that modifies an existing entry (or builds on one, if that database supports inheritance). The other possibility might be to support E3 (might also require shipping a dedicated terminfo file!), catch whatever specific `clear` sends to the tty as a result, and the provide a profile option whether or not the scrollback buffer should be cleared. The simple presence of that E3 command indicates that a clear was requested. FWIW, clear sends this to an xterm: `ESC[HESC[2J`
(In reply to RJVB from comment #2) > Currently Konsole emulates well-known terminals, but it could > very well define its own terminal type [...] As far as I know, konsole has been there, it used to work with TERM=konsole or something similar. I guess they had their reason to go for TERM=xterm (or a variant thereof) instead. gnome-terminal has also been there, it used to set TERM=gnome or maybe TERM=vte. It also had its reason to revert. Probably both approaches have their own downsides, but apparently both projects decided that going with TERM=xterm sucks less. > FWIW, clear sends this to an xterm: `ESC[HESC[2J` This is the old fashioned one that does not clear the scrollback. The new version is: ESC[3J -> clears the scrollback ESC[H -> moves the cursor to the upper left corner ESC[2J -> clears the onscreen contents For the first one, I can't think of any other possible reasonable behavior than either clearing the scrollback, or not doing anything. I'll get back to this. For the middle one there's not much to discuss, changing its behavior in any way would probably break tons of apps. Also it's pretty much irrelevant to this bug. For the last one, this is where konsole and VTE (gnome-terminal and friends) differ from xterm and scroll out these lines to the scrollback buffer. I'll get back to this too. > The other possibility might be to support E3 (might also require shipping a > dedicated terminfo file!), catch whatever specific `clear` sends to the tty > as a result, and the provide a profile option whether or not the scrollback > buffer should be cleared. The simple presence of that E3 command indicates > that a clear was requested. So... Let's first see the old behavior where \e[3J wasn't emitted or wasn't yet supported by the terminal emulator, or assume that there's a profile setting implemented where the user disables this feature. Then: - The traditional way, xterm's approach wipes out the onscreen contents, but the scrollback remains intact. That is, effectively, a screenful of lines will go missing from the scrollback. From a usability point of view, this doesn't make any sense to me. - Hence here come konsole and vte which modify this and scroll out these lines. The behavior is now sane, I get to see the proper history in the scrollback. Now let's introduce the "clear scrollback" feature. - The traditional way, xterm's now makes sense when you execute "clear". You pretty much get a "clean start". (It still doesn't make too much sense with let's say hitting Ctrl+L in bash which does an old-fashioned clear, without clearing the scrollback.) - konsole's and vte's behavior no longer make too much sense with the "clear" command, at least a screenful of empty lines appear in the scrollback – this is what your bugreport is about. (It still makes sense with e.g. bash's Ctrl+L.) So we already see a damn complicated support matrix of all possible scenarios with no straightforward way of getting all possibilities work as expected. One idea that occurred to me but I don't find it a good one: If a config option is introduced to enable/disable wiping out the scrollback (i.e. whether to support \e[3J or not), this same option could also change the behavior of \e[2J (current vs. xterm-compatible). This would "fix" the "clear" command, but would probably cause even more breakage and confusion for bash's Ctrl+L. Also, it's not the way terminal emulators should work. It shouldn't be the escape sequence emitted by the app defining how the terminal emulation should be done, it's the other way around. The terminal emulation behavior should determine what the app in turn emits. Also ugly as hell and other unforeseen side effects are also possible. Another possible approach, although I'm really unsure if the relevant person could be convinced to adapt terminfo to konsole's and vte's xterm-incompatible behavior: Alter the sequence emitted by "clear" so that wiping out the scrollback is the last step. (Doing this would indeed be simpler if konsole used its own TERM.) Yet another possibility is to drop \e[3J support, for reasons outlined in the already mentioned https://bugzilla.gnome.org/show_bug.cgi?id=771953.
Egmont Koblinger wrote on 20170901::20:08:49 re: "[konsole] [Bug 384218] clear/^l shouldn't add empty lines to the scrollback buffer" >- The traditional way, xterm's approach wipes out the onscreen contents, but >the scrollback remains intact. That is, effectively, a screenful of lines will >go missing from the scrollback. From a usability point of view, this doesn't >make any sense to me. Hmm, indeed, the lines that were visible never get added to the scrollback buffer. That never bothered me for "clear" but it did for swapping with the alternate screen (e.g. running vi). >- Hence here come konsole and vte which modify this and scroll out these lines. >The behavior is now sane, I get to see the proper history in the scrollback. There is another approach: add each line to the scrollback when it first appears, and don't remove them during a 'clear' unless the scrollback buffer is to be cleared too. Evidently you'd have to change something in the scrollback offset mechanism but I'd hope that should be trivial. Come to think of it, lines are probably already added as soon as they're output, so there must be another way to "push the visible part of the scrollback up off the sreen" than by adding empty lines. At the very least it should be trivial to avoid adding more empty lines than necessary. If the terminal shows 10 lines when the clear command is entered, the scrollback needs only be moved up (flushed) over 10 lines. If that number is 0 (= the prompt already at the top of the screen), then 0 lines need to be flushed. I think that's actually what irks me most. I use Ctrl-L instead of clear (see below) and it happens that I hit it multiple times, unconsciously. I know there's no need for me to do that, but it shouldn't have the side-effect it currently has. >- The traditional way, xterm's now makes sense when you execute "clear". You >pretty much get a "clean start". (It still doesn't make too much sense with >let's say hitting Ctrl+L in bash which does an old-fashioned clear, without >clearing the scrollback.) I've been using *csh shells since before bash was born the 1st time; Ctrl-L is aliased to the clear command. Should be possible in bash too, no? >probably cause even more breakage and confusion for bash's Ctrl+L. Also, it's >not the way terminal emulators should work. It shouldn't be the escape sequence >emitted by the app defining how the terminal emulation should be done, it's the >other way around. The terminal emulation behavior should determine what the app >in turn emits. Then you're not properly emulating an existing physical terminal anymore. I don't see how you can this without dedicated terminfo entry, be it in the on-disk database or via some sort of initial handshake (like baked-in monitor colour profiles). Either way, I'd see this as "the user indicates how a given escape sequence emitted by the app is interpreted by the terminal". >Yet another possibility is to drop \e[3J support, for reasons outlined in the >already mentioned https://bugzilla.gnome.org/show_bug.cgi?id=771953. E3 isn't involved in the issue that irks me so I don't see how dropping support for it would change anything?
(In reply to RJVB from comment #4) > There is another approach: add each line to the scrollback when it first > appears, and don't remove them during a 'clear' unless the scrollback buffer > is to be cleared too. [...] I don't understand what you mean here. Partly there's a confusion in terminology, whether the word "scrollback" includes the normally onscreen contents (I don't think it should, although I might have been sloppy previously). The onscreen contents can randomly change at any time (and it does with quite a few apps, e.g. fullscreen ones that don't switch to alternate screen such as "top"), so I don't think it's feasible to speak about automatically adding these to the scrollback. Anyway, it's unclear to me what the user-visible behavior would be with your suggestion. > [...] so there must be another way to "push the visible part of the > scrollback up off the sreen" than by adding empty lines. That's what happens now. The visible part is pushed up. The visible part sometimes just happens to be empty lines. > At the very least it should be trivial to avoid adding more empty lines than > necessary. If the terminal shows 10 lines when the clear command is entered, > the scrollback needs only be moved up (flushed) over 10 lines. If that > number is 0 (= the prompt already at the top of the screen), then 0 lines > need to be flushed. This is indeed one possibility, which might solve _some_ of the problems, but definitely not all. E.g. with the current E3-aware "clear" command the result would still be silly. You'd get the scrollback wiped out first, but then the currently visible bits (by the time of executing "clear") would be remembered in the scrollback. > I've been using *csh shells since before bash was born the 1st time; Ctrl-L > is aliased to the clear command. Should be possible in bash too, no? I don't know if it's possible to bound a single key into executing a command. I can't recall such a feature. > Then you're not properly emulating an existing physical terminal anymore. My knowledge about physical terminals is quite sparse, but I don't recall any of them having a scrollback buffer at all. Please tell me if I'm wrong. Since the only difference between xterm and konsole is what goes on to the scrollback on "\e[2J", as far as correct emulation of physical terminals is concerned, neither of them are wrong or right, they are just different. > E3 isn't involved in the issue that irks me so I don't see how dropping > support for it would change anything? The current situation is indeed far from perfect. If E3 isn't involved, then xterm's behavior never adds blank lines, however, sometimes wipes out precious data; whereas konsole never wipes out previous data, yet sometimes adds blank lines. I think the latter one (konsole's) is better. However, introducing E3 completely changes the game, and suddenly xterm's behavior makes more sense to me. Of course this is tayloring the emulator's behavior to what apps do, and as such, it's quite a nonsense approach. Ideally all emulators would copy xterm's behavior, and apps such as "clear" or bash's Ctrl+L would make sure to scroll out the contents. Konsole's and vte's workaround is a quite nasty one for the sake of usability (data preservation). This is a quite complex scenario, any modification probably fixes certain things while breaks others. Thus you can't improve the situation if you omit E3 and the "clear" command from the picture. Anyway, it seems to me that a noticeable improvement to this situation would require heavy cooperation of some key players in the game, including xterm, terminfo and bash; maybe even implementing a new escape sequence. As much as I'd love this to happen, I don't see a reasonable chance. That being said, I'm not a konsole developer, so I can't have a word in what it implements. I'm just trying to show you the bigger picture.
*** Bug 392661 has been marked as a duplicate of this bug. ***
> I think there are at least 2 levels at which an attempt can be made to > improve this. Currently Konsole emulates well-known terminals, but it could > very well define its own terminal type with the accompanying > termcap/terminfo file that modifies an existing entry (or builds on one, if > that database supports inheritance)." * local usage - OK * ssh to full desktop GNU/Linux - OK (as long as konsole terminfo is there) * ssh/serial connection to server/embedded system - konsole terminfo not available > provide a profile option whether or not the scrollback buffer should be > cleared. I'm for it. But this should do more than just blocking E3. > As far as I know, konsole has been there, it used to work with TERM=konsole > or something similar. I guess they had their reason to go for TERM=xterm > (or a variant thereof) instead. Git history says TERM=xterm is there since TERM is exported at all. I know there is terminfo file (even more than one), but it doesn't look official. > TERM=xterm sucks less Compatibility with de facto standard is nice, especially if breaking it doesn't give huge advantages. My thoughts: * \e[1J (erase from start to cursor) does not push lines to history. This command with cursor in last line should work exactly like \e[2J. * none of delete* and erase* functions moves something to history. * With "keep history" feature we would need to decide what should be moved to history. Clear? Remove of upper lines? What about lines in the middle of the screen? Or the one that was replaced? Or maybe just disable E3 (and make [2J xterm-like)? * as I said above, xterm/linux/... compatibility would be my choice.
Good news: see update at https://bugzilla.gnome.org/show_bug.cgi?id=506438#c19.
Git commit 7a1116ccff5da88f6ef7e0c4d50ae979b09fed8f by Kurt Hindenburg, on behalf of Mariusz Glebocki. Committed on 15/06/2019 at 21:03. Pushed by hindenburg into branch 'master'. CSI 2J: Do not append current screen to history Summary: This behavior is present in e.g. xterm and linux console. Currently when using CSI 2J escape sequence, screen contents are appended to scrollback (including bottom empty lines). This is breaking `clear` command, which replaces scrollback with current screen contents. Test Plan: * Start Konsole with a shell * `clear` **Expected result:** Cleared display, empty scrollback. **Actual result:** Cleared display, but scrollback contains previous display contents. --- * Start Konsole with a shell * seq 1000 * `echo -ne '\033[2J` **Expected result:** Cleared display, numbers in last scrollback lines. **Actual result:** Cleared display, but scrollback contains previous display contents (i.e. shell prompt in last lines) Reviewers: #konsole, hindenburg Reviewed By: #konsole, hindenburg Subscribers: hindenburg, konsole-devel, #konsole Tags: #konsole Differential Revision: https://phabricator.kde.org/D21282 M +0 -6 src/Screen.cpp https://invent.kde.org/kde/konsole/commit/7a1116ccff5da88f6ef7e0c4d50ae979b09fed8f
Git commit 7a1116ccff5da88f6ef7e0c4d50ae979b09fed8f by Kurt Hindenburg, on behalf of Mariusz Glebocki. Committed on 15/06/2019 at 21:03. Pushed by scmsync into branch 'master'. CSI 2J: Do not append current screen to history Summary: This behavior is present in e.g. xterm and linux console. Currently when using CSI 2J escape sequence, screen contents are appended to scrollback (including bottom empty lines). This is breaking `clear` command, which replaces scrollback with current screen contents. Test Plan: * Start Konsole with a shell * `clear` **Expected result:** Cleared display, empty scrollback. **Actual result:** Cleared display, but scrollback contains previous display contents. --- * Start Konsole with a shell * seq 1000 * `echo -ne '\033[2J` **Expected result:** Cleared display, numbers in last scrollback lines. **Actual result:** Cleared display, but scrollback contains previous display contents (i.e. shell prompt in last lines) Reviewers: #konsole, hindenburg Reviewed By: #konsole, hindenburg Subscribers: hindenburg, konsole-devel, #konsole Tags: #konsole Differential Revision: https://phabricator.kde.org/D21282 M +0 -6 src/Screen.cpp https://commits.kde.org/konsole/7a1116ccff5da88f6ef7e0c4d50ae979b09fed8f
Hi Kurt, Mariusz, > This is breaking > `clear` command, which replaces scrollback with current screen contents. You seem to have missed my latest comment (comment 8). As of ncurses 20180804, the "clear" command correctly clears the entire history in konsole, even without the just committed change. Therefore the claim that I quoted is no longer valid, and does not justify this change. The change makes konsole behave like xterm, which is not necessarily better in this case. For example, pressing Ctrl+L at the bash prompt now wipes out the last screenful of data, while the rest is still available in the scrollback. This behavior (i.e. xterm's behavior) makes little sense to me. In my firm opinion, Konsole's previous behavior (which differed from xterm's) of scrolling out those lines made much more sense: the scrollback buffer contained meaningful data even after a Ctrl+L at the shell prompt. If you really wish to go for this new behavior and be fully compatible with xterm, I'm not trying to stop you. But I'm kindly asking you to test the old and new behavior and evaluate the differences against a new ncurses which fixed clear's behavior, and make your choice based on this, rather than based on an old ncurses as you apparently did.
It might be a while until distros use 20180804; ubuntu is on 20180127. Still is is the consensus to revert this commit?
> ubuntu is on 20180127 Not sure where you got this number from. On packages.ubuntu.com, I get 20180210 (older than the "clear" fix) for the ncurses-base and ncurses-term packages of Cosmic (18.10), and 20181013 (newer than the "clear" fix) for Disco (19.04). I myself am using Ubuntu Disco and discovered the change in clear's behavior on this version. Also, > It might be a while until distros use 20180804 IMO this is not something that you should evaluate on its own, but in combination with when they upgrade to a Konsole recent enough to contain your changes (e.g. how realistic is that a distro will upgrade to Konsole >= 2019-06, but not to terminfo >= 2018-08), also how many users are expected to upgrade Konsole themselves while having an older terminfo; and whether this user base should be the one you tailor your software to rather than those distros that upgrade both software components more or less in parallel. In other words: out of those users who have konsole >= 2019-06 by whatever means, how many are likely to have ncurses prior to vs. newer than 2018-08? Users of konsole < 2019-06 are irrelevant, you can't fix nor break this for them unless you invent a time machine. :) Neither behaviors are seriously wrong or critical to worry about. It's a matter of "okay" vs. "even better" behavior. IMO it's better to target the newer systems, and their presumably continuously growing user base, rather than some the transitional state (presumably for not too many people). But I guess I should stay quiet and let you KDE folks make the call :)
Any chance this decision could be handled in the cmake logic so that the code has the best possible behaviour with either ncurses version, and would it be reasonable to do so (for the time being)? There are lots of distributions out there, and then there are dinosaurs like me who run a now unsupported distro but keep certain essential things uptodate themselves (and build niceties like KF5 stuff themselves too). BTW, I notice I'm just 1 point release behind with my local ncurses build (6.0 vs. 6.1); how do those numbers compare to the date-based version numbers? R.
(In reply to RJVB from comment #14) > Any chance this decision could be handled in the cmake logic so that the > code has the best possible behaviour with either ncurses version, and would > it be reasonable to do so (for the time being)? This could improve the situation; although there are more things to consider: - Currently konsole does not know anything about ncurses, doesn't rely on this library or tools (including the "clear" binary) for compiling or running. This would introduce at least a compile-time dependency. - It's actually the runtime version of ncurses ("clear" in particular) that should matter rather than the compile-time one (and thus I made a mistake in my previous comment referring to the ncurses-{base,term} Ubuntu packages rather than ncurses-bin). I don't know if ncurses exposes its version information via some API, or konsole could test what "clear" does, however, either approach would introduce a run-time dependency. Maybe it could be an optional runtime dependency: if /usr/bin/clear exists then konsole could examine what it prints and subsequently behave according to this. Hack, hack, hack... Is it worth it? I'd let Kurt and friends decide. - These wouldn't cover the case when you ssh to another system with another ncurses version. > BTW, I notice I'm just 1 point release behind with my local ncurses build > (6.0 vs. 6.1); how do those numbers compare to the date-based version > numbers? Seems to me that 20181013 (with the "clear fix") is a patch on top of 6.1.