Bug 263974

Summary:	Unexpected transparency of full-screen windows when compositing is enabled
Product:	[Plasma] kwin	Reporter:	stegerc
Component:	compositing	Assignee:	KWin default assignee <kwin-bugs-null>
Status:	RESOLVED NOT A BUG
Severity:	normal
Priority:	NOR
Version First Reported In:	unspecified
Target Milestone:	---
Platform:	openSUSE
OS:	Linux
Latest Commit:		Version Fixed/Implemented In:
Sentry Crash Report:
Attachments:	Patch for xscreensaver-5.12 Output of StonerView when compositing is not used Output of StonerView when compositing is used

Description stegerc 2011-01-22 15:18:09 UTC

Created attachment 56320 [details]
Patch for xscreensaver-5.12

Version:           unspecified (using KDE 4.4.4) 
OS:                Linux

I am trying to extend xscreensaver 5.12 to use multisampling anti-aliasing (MSAA). Please see the attached patch for xscreensaver 5.12. If the patch is applied, MSAA works as expected when compositing is turned off, both in the windowed versions of any GL screensaver (such as stonerview) as well as in the full-screen version that can be selected in xscreensaver-demo.  If compositing is switched on, however, a very strange behavior results. If the screensaver is run in windowed mode, the output looks correct, except for the fact that MSAA is not used. However, if the full-screen version of the screensaver is executed via xscreensaver-demo, all black pixels become transparent, i.e, the desktop "behind" the full-screen window becomes visible. I am using an ATI Radeon 5850 card with AMD Catalyst 10.12 on OpenSUSE 11.3 (x64).

Reproducible: Always

Steps to Reproduce:
Download xscreensaver 5.12 from http://www.jwz.org/xscreensaver/xscreensaver-5.12.tar.gz.
Apply the attached patch.
Compile xscreensaver 5.12.
Set the PATH variable as follows (csh syntax):
setenv PATH $XSCREENSAVER/driver:$XSCREENSAVER/hacks:$XSCREENSAVER/hacks/glx:$PATH
where XSCREENSAVER points to the directory in which xscreensaver 5.12 was compiled.

First scenario:
Execute
$XSCREENSAVER/driver/xscreensaver-demo --configdir $XSCREENSAVER/hacks/config
and select any OpenGL screensaver (such as StonerView or GLMatrix).

Second scenario:
Execute the screensaver directly:
$XSCREENSAVER/hacks/glx/stonerview


Actual Results:  
First scenario:
When compositing is turned off, the screensaver works correctly.
When compositing is turned on, the black pixels in the screen saver become transparent.

Second scenario:
When compositing is turned off, the screensaver works correctly and MSAA is used.
When compositing is turned on, MSAA is not used.


Expected Results:  
First scenario:
When compositing is turned on, the output should look identical to the output when compositing is turned off, i.e., black pixels should not be transparent.

Second scenario:
When compositing is turned on, the output should look identical to the output when compositing is turned off, i.e., MSAA should be used.

Comment 1 stegerc 2011-01-22 15:19:13 UTC

Created attachment 56321 [details]
Output of StonerView when compositing is not used

Comment 2 stegerc 2011-01-22 15:19:48 UTC

Created attachment 56322 [details]
Output of StonerView when compositing is used

Comment 3 Martin Flöser 2011-01-22 16:22:16 UTC

I don't think there is a kwin bug here. I assume you use an RGBA window and the areas are transparent. In that case it is correct that the background is visible.

Ensure to use RGB window or do a proper clear with glClearColor(0,0,0,1)

Comment 4 Thomas Lübking 2011-01-22 17:03:04 UTC

...and preferably use an RGB window as otherwise the window can not be unredirected (performance matter)

Comment 5 stegerc 2011-01-23 21:38:49 UTC

Thank you very much! Adding an explicit call to glClearColor seems to solve the transparency problem.  However, the fact that MSAA is ignored when compositing is enabled remains.

Comment 6 Thomas Lübking 2011-01-23 22:18:36 UTC

(In reply to comment #5)
> However, the fact that MSAA is ignored when compositing is enabled remains.

it's probably rather related to the drawable, so:
"...and preferably use an RGB window as otherwise the window can not be
unredirected (performance matter)"

Comment 7 Martin Flöser 2011-01-23 22:23:08 UTC

Setting to invalid due to comment #5 and #6.

Comment 8 stegerc 2011-01-24 09:28:07 UTC

On reflecting on comment #3, I still don't understand why the behavior only occurs if I select a visual that supports MSAA, i.e., why doesn't the behavior occur for "regular" visuals without MSAA?  In this case, the full-screen windows don't become transparent. If glClearColor was the true cause of the behavior, I would expect the same behavior, no matter which kind of visual is used.

Comment 9 stegerc 2011-01-24 09:45:10 UTC

Regarding comments #4 and #6: there is no way to select an RGB visual, as opposed to an RGBA visual, in xscreensaver because there are numerous screen savers in xscreensaver that use transparency and because there is a centralized mechanism that selects the visual for all screen savers. Hence, an RGBA visual must be selected.

Without having looked at the code of kwin, my feeling is that MSAA visuals (i.e., those that have GLX_SAMPLE_BUFFERS, GLX_SAMPLE_BUFFERS_ARB, or GLX_SAMPLE_BUFFERS_SGIS set) would have to be mapped to GL_EXT_framebuffer_multisample (assuming that FBOs are used for compositing in kwin) automatically to result in the same visual appearance independent of whether compositing is used or not.

Comment 10 Thomas Lübking 2011-01-24 14:52:47 UTC

(In reply to comment #9)
> Regarding comments #4 and #6: there is no way to select an RGB visual, as
> opposed to an RGBA visual, in xscreensaver because there are numerous screen
> savers in xscreensaver that use transparency and because there is a 

i think this is a misunderstanding.
for some reason you're triggering an ARGB *X11* drawable ("window") (otherwise the window would not be translucent) and i doubt that xscreensaver (or some hack) relies on ARGB X11 windows + active compositing :-)

This prevents the unredirection (which works perfectly with my copy of xscreensaver), and possibly causes the MSAA failure as well.

proofs:
- plain stonerview (xss 5.12, no idea about the hack version) is unredirectable
- MSAA works with compositing:
$ xrandr -s 800x600 #to actually see the aliasing ;-)
$ __GL_FSAA_MODE=0 /usr/lib/xscreensaver/stonerview
-> bloody aliased =)
$ __GL_FSAA_MODE=8 /usr/lib/xscreensaver/stonerview
-> nvidia claims it's 16x (4xMS, 12xCS) - it's apparently aliased in any case and works (and has a black background)
   * redirected or unredirected
   * OpenGL or XRender backend


> (i.e., those that have GLX_SAMPLE_BUFFERS, GLX_SAMPLE_BUFFERS_ARB, or
> GLX_SAMPLE_BUFFERS_SGIS set) would have to be mapped to
> GL_EXT_framebuffer_multisample

afaik KWin could not possibly detect that
a) it's a GL client (read: "window")
b) the GL context has glEnable(GL_MULTISAMPLE)

we only map X11 clients into (RGBA) textures - regardless of what's "in there", and that does not break multisampling here (etracer would be unbearable otherwise ;-)

Alltogether i'm fairly convinced that this is an issue on your side and that multisample contexts just do not work on ARGB X11 clients, what /might/ be a driver limitation, though.

Solution: don't use an ARGB window =)

Comment 11 stegerc 2011-01-28 17:17:34 UTC

I have to admit that I am lost here.  Could you please explain to me how to get an OpenGL visual that has an alpha buffer, but an X11 window that is not an ARGB drawable, i.e., in an RGB X11 drawable?