Have a CMake test that exits non-zero but outputs something, and a matching PASS_REGULAR_EXPRESSION in the test properties. When this test is run from the command line (ctest -R my_test) it works and is marked as passed, if it is run from the unit test pane in KDevelop it fails (and looks like the expression isn't used at all).