After figuring out where to find the code for spectacle, I did some more experiments and could pinpoint the issue better.

First of all, by running `spectacle` from the command line, I could see the following output in the terminal:

> qt.qpa.xcb: QXcbConnection: XCB error: 11 (BadAlloc), sequence: 922, resource id: 94371870, major code: 149 (Unknown), minor code: 2
> qt.qpa.xcb: QXcbConnection: XCB error: 9 (BadDrawable), sequence: 923, resource id: 94371879, major code: 149 (Unknown), minor code: 4
> qt.qpa.xcb: QXcbConnection: XCB error: 4 (BadPixmap), sequence: 925, resource id: 94371879, major code: 148 (Unknown), minor code: 1
> qt.qpa.xcb: QXcbConnection: XCB error: 11 (BadAlloc), sequence: 926, resource id: 94371870, major code: 149 (Unknown), minor code: 2
> qt.qpa.xcb: QXcbConnection: XCB error: 9 (BadDrawable), sequence: 927, resource id: 94371881, major code: 149 (Unknown), minor code: 4
> qt.qpa.xcb: QXcbConnection: XCB error: 4 (BadPixmap), sequence: 928, resource id: 94371881, major code: 148 (Unknown), minor code: 1

This hints towards an allocation issue. After a fun little printf (and qDebug) session I was able to isolate this call to setGeometry[1] in the QuickEditor: 
 - If I divide the width and height by 2, rectangular selection works but is displayed on 1/4 of the screen
 - If I also remove the division by the devicePixelRatio in the QML, rectangular selection works (but presumably with a downscaling followed by an upscaling, which is not optimal)

After some more experimenting, it turns out that there is a cut-off when width * height becomes larger than 4Mpx (my screen resolution is 2880 x 1800 = 4.9Mpx): if less, the UI shows up[2], if more, the screen goes black and the above error messages gets printed.  It definitely looks like *something* prevents allocating more than 4Mpx... but somehow it doesn't happen when scaling is set to 1?  I also checked on another computer, and Spectacle works fine, even with a screen scaling factor of 4!

Some more fiddling later (at this point I discovered I could set the QT_SCREEN_SCALE_FACTORS variable instead of logging out/logging in to my session, which made further experiments faster), I realized that if I keep scaling to 1 the problem does manifest itself, but the limit is now at 16Mpx instead of 4Mpx.  This corresponds to a (fake) resolution of 4096x4096.  This is weird, because the value passed in there is always 2880x1800 no matter what scaling factor I use, so it should already be the native resolution; no more scaling should occur.  Finally, trying with 8192x8192 yield the following interesting error:

> i965: Failed to submit batchbuffer: No space left on device

which gives the source of the limit: the GPU driver! Indeed, xrandr reports a maximum of 8192x8192 on my MacBook, and 32767x32767 on my other laptop.

After reading some more documentation on Qt HiDPI (https://doc.qt.io/qt-5/highdpi.html) it looks like:

 - Spectacle is taking a high DPI (native resolution) capture of the screen, then displaying it at low DPI (scaled resolution), which results in a buffer four time as big as it would need to
 - My graphics card driver doesn't like the point above :)
 - Doing `d->mQuickView->setGeometry(0, 0, pixmap.width() / devicePixelRatio, pixmap.height() / devicePixelRatio)` and removing the `Screen.devicePixelRatio` divisions in the QML actually does the right thing (i.e. the final buffer allocated on the device is the size of the screen) and does not suffer from downscaling-then-upscaling
 - In the multi-monitor case, there is not always a single devicePixelRatio!  Given the above, it looks like the solutions are either to use a larger amount of memory than needed or properly display the rectangle selection UI across multiple screens (which I don't know if Spectacle currently does).

Anyways, I think I reached the limits of what I could do, I hope this is helpful.  I am willing to make a patch, but it is not clear to me what exactly said patch should do :)

[1] https://cgit.kde.org/spectacle.git/tree/src/QuickEditor/QuickEditor.cpp?id=e00391c1b1f4a119a3e737a96da9057a70ae3339#n94

[2] Potentially at the wrong scale

Thanks, that is indeed very helpful. Please do feel free to submit a patch, if only for discussion purposes. You can learn how at https://community.kde.org/Infrastructure/Phabricator.

>  - Spectacle is taking a high DPI (native resolution) capture of the screen,
> then displaying it at low DPI (scaled resolution), which results in a buffer
> four time as big as it would need to
This would seem to be a good place to focus your efforts.

Kai, do you think you might be able to give gryzzl1 a hand with this?

I believe the bug does not reproduce anymore.

I believe you're right!

Still occurs for me; Spectacle 21.04.0;

I have three 4K monitors with 200% scaling.
When i try to capture screenshot, all screens are black.
Without scaling everything works as expected

I can provide any other information if needed.

(In reply to sashok724 from comment #5)
> Still occurs for me; Spectacle 21.04.0;
> 
> I have three 4K monitors with 200% scaling.

Three 4k screens is uncommon, just the shear amount of pixel could explain the situation.

> When i try to capture screenshot, all screens are black.
> Without scaling everything works as expected
> 
> I can provide any other information if needed.

Please paste you kinfocenter info.

Do you reproduce with wayland ?

Do you reproduce with 2 screens ? 1 screen ?

Spectacle was very laggy under Wayland with scaling (felt like 2fps), but it worked.

With two screens it works as expected.
Maybe it uses twice the space than needed with 200% scaling? XRandr shows that maximum resolution is 16384x16384, and with three displays i'm using 11520 of its width (*2 = 23040, more than 16384), but with two displays i'm using only 7680 of its width (*2 = 15360, less than 16384)


KInfoCenter info:

Operating System: Arch Linux
KDE Plasma Version: 5.21.4
KDE Frameworks Version: 5.81.0
Qt Version: 5.15.2
Kernel Version: 5.12.0-arch1-1
OS Type: 64-bit
Graphics Platform: X11
Processors: 32 × AMD Ryzen 9 3950X 16-Core Processor
Memory: 62.8 GiB of RAM
Graphics Processor: AMD Radeon RX 5700 XT

XRandr output minus the long list of available modes:

Screen 0: minimum 320 x 200, current 11520 x 2160, maximum 16384 x 16384
DisplayPort-0 connected primary 3840x2160+3840+0 (normal left inverted right x axis y axis) 600mm x 340mm
DisplayPort-1 connected 3840x2160+7680+0 (normal left inverted right x axis y axis) 600mm x 340mm
DisplayPort-2 connected 3840x2160+0+0 (normal left inverted right x axis y axis) 600mm x 340mm

I will be happy to provide any other information. Thanks in advance!

I think i figured out a solution for this problem.

I just replaced this line in QuickEditor.cpp(625):

resize(mScreensRect.width(), mScreensRect.height());

with:

resize(qRound(mScreensRect.width() / devicePixelRatio), qRound(mScreensRect.height() / devicePixelRatio));

So region selector window is created with correct size.

But, it was not the only problem. In region selector, leftmost screen contents was duplicated to central screen. I fixed it by also removing this line from QuickEditor.cpp(662), as width and height of rect appears to be already scaled, unlike X and Y, i do not know why:

rectToDraw.setSize(rectToDraw.size() * devicePixelRatio);

If what i done is correct solution, i can create pull request.

Created attachment 138060 [details]
Proposed patch

(In reply to sashok724 from comment #8)
> I think i figured out a solution for this problem.
> 
> I just replaced this line in QuickEditor.cpp(625):
> 
> resize(mScreensRect.width(), mScreensRect.height());
> 
> with:
> 
> resize(qRound(mScreensRect.width() / devicePixelRatio),
> qRound(mScreensRect.height() / devicePixelRatio));
> 
> So region selector window is created with correct size.
> 
> But, it was not the only problem. In region selector, leftmost screen
> contents was duplicated to central screen. I fixed it by also removing this
> line from QuickEditor.cpp(662), as width and height of rect appears to be
> already scaled, unlike X and Y, i do not know why:
> 
> rectToDraw.setSize(rectToDraw.size() * devicePixelRatio);
> 
> If what i done is correct solution, i can create pull request.

This seems like a plausible way, I encourage you to open a merge request !
https://community.kde.org/Infrastructure/GitLab

Reviewing out of context is hard and we can have more reviewers on invent.kde.org

A possibly relevant merge request was started @ https://invent.kde.org/graphics/spectacle/-/merge_requests/66

Git commit 2bc4d3971d3a34ef7a3177745f17735764daa2b6 by Méven Car, on behalf of Aleksandr Ksenofontov.
Committed on 07/05/2021 at 06:03.
Pushed by meven into branch 'master'.

X11: Use correct width and height for region selection window and its contents when scaling is enabled

M  +4    -2    src/QuickEditor/QuickEditor.cpp

https://invent.kde.org/graphics/spectacle/commit/2bc4d3971d3a34ef7a3177745f17735764daa2b6