Bug 327356 - Provide a way to align 2 or more windows in a split fullscreen mode with input devices assigned to either window (Feature Request)
Summary: Provide a way to align 2 or more windows in a split fullscreen mode with inpu...
Status: REPORTED
Alias: None
Product: kwin
Classification: Plasma
Component: multi-screen (show other bugs)
Version: unspecified
Platform: unspecified Linux
: NOR wishlist
Target Milestone: ---
Assignee: KWin default assignee
URL:
Keywords: multiscreen
Depends on:
Blocks:
 
Reported: 2013-11-09 10:44 UTC by Dimitri Nüscheler
Modified: 2024-07-29 21:58 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dimitri Nüscheler 2013-11-09 10:44:23 UTC
There is an endless amount of possible multiseat setups, but only very few are useful.
A split screen mode for games that don't have native split screen is one of the useful ones, in my opinion.
With Wayland we finally get the building blocks for it (I hope).

But there is still the problem of assigning each gamepad/mouse/keyboard to the correct window in an easy way and aligning the windows correctly.

My suggestion is to have some sort of split-screen mode which can be activated/deactivated using a key combination.

As soon as it is activated, every window that requested or requests fullscreen or a selected set of windows gets a tile on an otherwise black screen with user friendly aspect ratio maintained, no decorations.

After activation, input devices first need to be assigned to a window.
This happens by using the keys best suited for navigation (arrow keys, WASD, joystick, mouse cursor) confirmation and cancellation key.

Each input device decorates (not a window-decoration as we know it) a window with a unique combination of color and symbol. Users learn the association of color and input device by changing windows using the keys and seeing which decoration jumped to the other window.
The symbol ideally reflects the type of input device to further ease association.

The decoration fades out after a while, so devices the user decided to not use are no longer visible and don't confuse the user.

Several input devices can be assigned to the same window, completion of association is therefore an additional confirmation.

As soon as all windows have both input devices associated AND got confirmed that input device association is complete, the input is from now on redirected to the clients (Pointers are no longer able to leave the window now).

The compositor should use the windows as scanout buffers, just like in the one-window fullscreen case.

This is not only useful for games.

The full "problem space" of Multiseat setups is still an interesting discussion. I tried to analyze it a bit, here, but as an implementation it is overengineered overkill: https://wiki.ubuntu.com/DimitriNuescheler/MultiseatGraph

What do you think? Is it a cool idea? Is this feasible?

I don't think I can do it. But if I wanted to try to implement this specific feature. Where would I start?

Reproducible: Always
Comment 1 Dimitri Nüscheler 2013-11-09 10:55:17 UTC
Further things:
- There should be a way to escape the split-mode without forgetting about the association on return
- A way to initiate reassociation
- To change the window positions and aspect ratio, especially in case of multiple monitors

I don't need to mention that windows become scanout buffers only after full completion of association, because for the decorations you need compositing.
Comment 2 Martin Flöser 2013-11-12 06:52:46 UTC
As you talk about multi-seat I do not think that KWin is the right component for it. This needs to be a layer below (logind-system-compositor). I don't want to have low-level input control inside KWin.
Comment 3 Dimitri Nüscheler 2013-11-12 08:24:30 UTC
I think the problem is also the terminology.
I see the term multiseat as a super domain to what is described here:
http://www.freedesktop.org/wiki/Software/systemd/multiseat/

I see it as something that has responsibilities throughout the stack to the finest granularity.
If a document has to be edited by multiple users at the same time, then it's the application which has to be able to deal with multiple mouse-pointers. For me this problem also belongs to the multiseat domain, but I think it has nothing to do with logind.

I guess some instance (logind?) has to decide which input hardware is assigned to a "session", but is it wrong to have the "session"(kwin?) then further assign these input devices to specific windows.

So what is low-level input control? Where do you see a conflict with the design and architecture of kwin if you put a splitscreen mode into kwin?

Don't you think that putting it into logind imposes too much of overhead? For every game window you have to start a session, you might even have authentication overhead, wait for it to load a second session? I think it completey kills the added value of the usability-feature I describe.

Maybe it's not bad to have the same functionality in logind as well (with some abstraction added using the same shared logic).

Another alternative is to have another compositor running inside kwin, but then you can't take existing windows into your session unless wayland-clients learn to jump from compositor to compositor. You add another layer and I really see the splitscreen as a window-managerish problem, because the window manager is the information expert on tiling.
Comment 4 Dimitri Nüscheler 2013-11-12 08:29:51 UTC
Edit: Third paragraph should end with questionmark.

I also want to avoid logind in the first place, because I really want to narrow down the "problem domain" to split screen emulation with an innocent focus on games - otherwise you're going to end up in the overgeneralized, overengineered overkill.
Comment 5 Martin Flöser 2013-11-12 09:59:27 UTC
On Tuesday 12 November 2013 08:24:30 you wrote:
> https://bugs.kde.org/show_bug.cgi?id=327356
> 
> --- Comment #3 from Dimitri Nüscheler <dimitri.nuescheler@sunrise.ch> ---
> I think the problem is also the terminology.
> I see the term multiseat as a super domain to what is described here:
> http://www.freedesktop.org/wiki/Software/systemd/multiseat/
I follow the description exactly used by systemd. Everything I know about 
multiseat comes directly from Lennart (presentation at DS 2011), so let's 
please not introduce some other definition. Thank you :-)

> I guess some instance (logind?) has to decide which input hardware is
> assigned to a "session", but is it wrong to have the "session"(kwin?) then
> further assign these input devices to specific windows.
KWin currently does not support the concept of multiple seats. I don't see 
this to change. I do not think that this should have any priority, it clearly 
is not a scratch-your-own-edge use case for me and I think it's an extreme 
corner case usecase up to the point where I question whether it's a valid use 
case at all. During my first work on Wayland support I considered adding 
support for multiseat as supported by the Wayland protocol, but decided 
against it.

I'm open to see this being added to KWin, but I'm not going to work on it ;-)

Considering your usecase of games: I don't think that this is a valid use case 
for KWin at all. From a performance perspective it might be better to have 
games just bypass the session compositor completely and run as an own 
independent session directly on the system compositor. Last but not least we 
will have to see how games will evolve. Maybe everybody who runs games will be 
using SteamOS next year - who knows?
Comment 6 Thomas Lübking 2013-11-12 13:51:24 UTC
no idea how troublesome this would be on wayland, but at least assugning different keyboard focuses to different windows might be beyond cornercase, eg. to allow a remote control on a mediaplayer while writing a mail with the realdeal keyboard (couch case ;-)

(though i will admit that the current solution to assign the rc special keys, bind them to global shortcuts searching for the next matching client and qdbus/xdotool it works fine for me =)
Comment 7 Dimitri Nüscheler 2013-11-12 14:50:39 UTC
Yeah, MultiSeat is no accurate wording.
"human interface device signal and graphics buffer redirection" is better ;-)

So with Unix we have an awesome operating system for I/O redirection.
Thanks to bash we have a shell for it.

And with Wayland we have an awesome display system for "human interface device signal and graphics buffer redirection"
Thanks to the resolution of #327356 we have an awesome shell (for part of it).

Just kidding - but I really believe that the cleverness of Wayland in design is something I haven't seen since Unix (if I understand it correctly). KWin is my favorite Window Manager today and I'm confident it will be a great Wayland compositor, so of course I wanted to bring this idea here.

I understand there are higher priorities, but maybe you can at least give me a suggestion how to start on this? I'm abit of a rookie when it comes to system level programming (sorry to bother you), but I'd love to do this.

I don't think it's feasible with KWin scripting, or is it?
Should I go and ask logind - are they concerned with similar problems?
Should I start with a fresh minimal compositor that runs atop the session compositor? Maybe this is even the right approach. It's some more context switches but for couch gaming the added nano seconds don't matter.
Maybe I can adapt/fork KWin code? Reuse effects?
Comment 8 Martin Flöser 2013-11-12 15:13:01 UTC
> I understand there are higher priorities, but maybe you can at least give me
> a suggestion how to start on this? I'm abit of a rookie when it comes to
> system level programming (sorry to bother you), but I'd love to do this.
as written, I had thought about it when starting the wayland input 
redirection. The idea I had back then was to have one Toplevel per seat 
element (pointer, keyboard). I decided in the end against it as I thought it 
to be too invasive at the moment. In the end it would result in having more 
than one "active" window. One active window per seat. That's really a lot of 
work I expect and too much to integrate it directly into the Wayland port.

But I like the use case Thomas mentioned. This could be a good enough 
solution. If we don't support more than one active application but still allow 
to bind an additional seat to send events to another window it might work.