Bug 469255 - 5.27.4.1: test suite is failing in 64% of all units
Summary: 5.27.4.1: test suite is failing in 64% of all units
Status: RESOLVED NOT A BUG
Alias: None
Product: kwin
Classification: Plasma
Component: general (other bugs)
Version First Reported In: 5.27.4
Platform: Other Other
: NOR normal
Target Milestone: ---
Assignee: KWin default assignee
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-05-02 10:55 UTC by kloczek
Modified: 2023-05-02 22:12 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed/Implemented In:
Sentry Crash Report:


Attachments
ctest output (745.86 KB, text/plain)
2023-05-02 10:55 UTC, kloczek
Details

Note You need to log in before you can comment on or make changes to this bug.
Description kloczek 2023-05-02 10:55:38 UTC
Created attachment 158620 [details]
ctest output

I suppose that something is missing in my build env or used procedure is not correct.
Neverteless most of the units are failing.

Just summary of the ctest output

+ cd kwin-5.27.4.1
+ xvfb-run -a /usr/bin/ctest --test-dir x86_64-redhat-linux-gnu --output-on-failure --force-new-ctest-process -j48 ' '
Internal ctest changing into directory: /home/tkloczko/rpmbuild/BUILD/kwin-5.27.4.1/x86_64-redhat-linux-gnu
Test project /home/tkloczko/rpmbuild/BUILD/kwin-5.27.4.1/x86_64-redhat-linux-gnu
        Start   1: kwayland-testWaylandOutput
        Start   2: kwayland-testWaylandSurface

[..]

187/187 Test   #3: kwayland-testWaylandSeat ...............................   Passed   16.96 sec

64% tests passed, 68 tests failed out of 187

Total Test time (real) =  16.98 sec

The following tests FAILED:
          4 - kwayland-testShmPool (Subprocess aborted)
         58 - kwin-testDontCrashGlxgears (Failed)
         59 - kwin-testLockScreen (Failed)
         61 - kwin-testDecorationInput-waylandonly (Failed)
         63 - kwin-testInternalWindow-waylandonly (Failed)
         64 - kwin-testTouchInput (Failed)
         66 - kwin-testInputStackingOrder (Failed)
         68 - kwin-testPointerInput (Failed)
         70 - kwin-testDontCrashCancelAnimation (Failed)
         73 - kwin-testTransientPlacement-waylandonly (Failed)
         74 - kwin-testDebugConsole (Failed)
         75 - kwin-testDontCrashEmptyDeco (Failed)
         76 - kwin-testPlasmaSurface (Failed)
         78 - kwin-testMaximized (Failed)
         80 - kwin-testXdgShellWindow (Failed)
         81 - kwin-testXdgShellWindow-waylandonly (Failed)
         82 - kwin-testDontCrashNoBorder (Failed)
         83 - kwin-testDontCrashNoBorder-waylandonly (Failed)
         85 - kwin-testSceneOpenGL (Failed)
         86 - kwin-testSceneOpenGL-waylandonly (Failed)
         87 - kwin-testSceneOpenGLES (Failed)
         88 - kwin-testSceneOpenGLES-waylandonly (Failed)
         89 - kwin-testScreenChanges (Failed)
         93 - kwin-testTabBox-waylandonly (Failed)
         94 - kwin-testWindowSelection (Failed)
         97 - kwin-testPointerConstraints-waylandonly (Failed)
         99 - kwin-testKeyboardLayout-waylandonly (Failed)
        101 - kwin-testKeymapCreationFailure-waylandonly (Failed)
        103 - kwin-testShowingDesktop-waylandonly (Failed)
        104 - kwin-testDontCrashUseractionsMenu (Failed)
        107 - kwin-testKWinBindings-waylandonly (Failed)
        109 - kwin-testLayerShellV1Window-waylandonly (Failed)
        111 - kwin-testVirtualDesktop-waylandonly (Failed)
        113 - kwin-testXdgShellWindowRules-waylandonly (Failed)
        114 - kwin-testIdleInhibition (Failed)
        116 - kwin-testDontCrashReinitializeCompositor (Failed)
        117 - kwin-testDontCrashReinitializeCompositor-waylandonly (Failed)
        119 - kwin-testNoGlobalShortcuts-waylandonly (Failed)
        120 - kwin-testBufferSizeChange (Failed)
        121 - kwin-testBufferSizeChange-waylandonly (Failed)
        123 - kwin-testPlacement-waylandonly (Failed)
        124 - kwin-testActivation (Failed)
        126 - kwin-testInputMethod (Failed)
        129 - kwin-testScreens-waylandonly (Failed)
        130 - kwin-testScreenEdges (Failed)
        133 - kwin-testOutputChanges-waylandonly (Failed)
        134 - kwin-testTiles (Failed)
        137 - kwin-testFractionalScaling-waylandonly (Failed)
        139 - kwin-testVirtualKeyboardDBus-waylandonly (Failed)
        140 - kwin-testNightColor (Failed)
        141 - kwin-testNightColor-waylandonly (Failed)
        145 - kwin-testDontCrashAuroraeDestroyDeco (Failed)
        151 - kwin-testX11Window (Failed)
        152 - kwin-testQuickTiling (Failed)
        161 - kwin-testMinimizeAllScript-waylandonly (Failed)
        163 - kwin-testSlidingPopups (Failed)
        164 - kwin-testShadeWobblyWindows (Failed)
        165 - kwin-testScriptedEffects (Failed)
        166 - kwin-testToplevelOpenCloseAnimation (Failed)
        167 - kwin-testToplevelOpenCloseAnimation-waylandonly (Failed)
        169 - kwin-testPopupOpenCloseAnimation-waylandonly (Failed)
        170 - kwin-testDesktopSwitchingAnimation (Failed)
        171 - kwin-testDesktopSwitchingAnimation-waylandonly (Failed)
        172 - kwin-testMinimizeAnimation (Failed)
        173 - kwin-testMinimizeAnimation-waylandonly (Failed)
        174 - kwin-testMaximizeAnimation (Failed)
        175 - kwin-testMaximizeAnimation-waylandonly (Failed)
        176 - kwin-testLibinputDevice (Failed)
Errors while running CTest


Full ctest output is in attachment
Comment 1 Nate Graham 2023-05-02 17:56:24 UTC
They work in the CI, which is required to pass. So there is indeed something wrong with your local setup. And I'm afraid bug reports aren't the right place to ask for help with your dev setup.
Comment 2 kloczek 2023-05-02 21:29:26 UTC
(In reply to Nate Graham from comment #1)
> They work in the CI, which is required to pass. So there is indeed something
> wrong with your local setup. And I'm afraid bug reports aren't the right
> place to ask for help with your dev setup.

So you don't know what it could be th cause?
Comment 3 Nate Graham 2023-05-02 21:31:00 UTC
Nope, I didn't look at the attachment, because this is a bug tracker, not a place to get dev or configuration assistance. :) I might recommend https://discuss.kde.org/c/development/10 instead.
Comment 4 kloczek 2023-05-02 21:34:17 UTC
(In reply to Nate Graham from comment #3)
> Nope, I didn't look at the attachment, because this is a bug tracker, not a
> place to get dev or configuration assistance. :) I might recommend
> https://discuss.kde.org/c/development/10 instead.

Why I shoud ask people who knows progbably nothing about kwin test suite?🤔
Comment 5 Nate Graham 2023-05-02 21:37:52 UTC
Why are you assuming nobody there will be able to help you?

Regardless, you need to do some investigation yourself to read the logs and figure out why your local setup isn't working properly. Again, the tests are working in the CI, which means if they're failing for you, the problem is a local issue on your machine.
Comment 6 kloczek 2023-05-02 21:53:25 UTC
(In reply to Nate Graham from comment #5)
> Why are you assuming nobody there will be able to help you?

Are there any people who knows about test kwin test suite?

> Regardless, you need to do some investigation yourself to read the logs and
> figure out why your local setup isn't working properly. Again, the tests are
> working in the CI, which means if they're failing for you, the problem is a
> local issue on your machine.

May I ask where is procedure/description of assembly test suite and how it is executed?
Comment 7 Nate Graham 2023-05-02 21:55:23 UTC
(In reply to kloczek from comment #6)
> (In reply to Nate Graham from comment #5)
> > Why are you assuming nobody there will be able to help you?
> 
> Are there any people who knows about test kwin test suite?
Maybe. I don't know. They certainly aren't going to show up here to help you troubleshoot your dev environment, though.

> > Regardless, you need to do some investigation yourself to read the logs and
> > figure out why your local setup isn't working properly. Again, the tests are
> > working in the CI, which means if they're failing for you, the problem is a
> > local issue on your machine.
> 
> May I ask where is procedure/description of assembly test suite and how it
> is executed?
I don't know, and I doubt such a thing exists. Typically test quites aren't extensively documented. You run them and they pass or fail. If they fail, you investigate the failure.
Comment 8 kloczek 2023-05-02 21:58:12 UTC
(In reply to Nate Graham from comment #7)
[..]
> I don't know, and I doubt such a thing exists. Typically test quites aren't
> extensively documented. You run them and they pass or fail. If they fail,
> you investigate the failure.

Looks like you really don;t know to much about testing kwin .. which is a bit odd.
Why did you not wrote about that in first place?

So who knows?
Comment 9 kloczek 2023-05-02 21:58:52 UTC
Do you know at least how test suite is executed?
Comment 10 Nate Graham 2023-05-02 22:03:26 UTC
I'm a bug triager, not a KWin developers. I triaged this bug as "not a bug because it's a question about a local development setup". The purpose of bug triaging is to make sure that bug reports are reporting actual bugs, so we don't waste developers' time when they go looking through bug reports by making them wade through things that they can't work on.

I typically run tests with a bash alias:

function ktest {
    REPO=$( basename $( git rev-parse --show-toplevel ) )
    BUILD_DIR=~/kde/build/$REPO
    make --no-print-directory -C $BUILD_DIR test
}

When I'm in the source dir, I run `ktest` and it runs the tests from the build dir without me having to go there manually.
Comment 11 kloczek 2023-05-02 22:12:07 UTC
(In reply to Nate Graham from comment #10)
> I'm a bug triager, not a KWin developers. I triaged this bug as "not a bug
> because it's a question about a local development setup".

How do you know that this is not a bug?
Did you never saw test suite failing on one system and be perfectly fine on another one even if they have been assembled using the same binary packages caused by actual bug in tested code?

> The purpose of bug
> triaging is to make sure that bug reports are reporting actual bugs, so we
> don't waste developers' time when they go looking through bug reports by
> making them wade through things that they can't work on.
>
> I typically run tests with a bash alias:
> 
> function ktest {
>     REPO=$( basename $( git rev-parse --show-toplevel ) )
>     BUILD_DIR=~/kde/build/$REPO
>     make --no-print-directory -C $BUILD_DIR test
> }
> 
> When I'm in the source dir, I run `ktest` and it runs the tests from the
> build dir without me having to go there manually.

Is that ststem assembled out of Debian or some rpm based duistrinution?
Can you show me at least list of packages installed in that CI system?