Bug 460450 - KDevelop ignores modifications to CMakeLists.txt performed while KDevelop's CMake configuring is in progress
Summary: KDevelop ignores modifications to CMakeLists.txt performed while KDevelop's C...
Status: REPORTED
Alias: None
Product: kdevelop
Classification: Applications
Component: Build tools: CMake (show other bugs)
Version: git master
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: kdevelop-bugs-null
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-10-14 20:26 UTC by Igor Kushnir
Modified: 2023-09-06 14:35 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Patch to debug CMake outdated warning (695 bytes, patch)
2023-09-06 09:37 UTC, Igor Kushnir
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Igor Kushnir 2022-10-14 20:26:02 UTC
SUMMARY
Probably the recent timestamp optimization https://commits.kde.org/kdevelop/5b1c175bad00d5f087caf49dabbfb565d9da857d didn't take this possibility into account.

STEPS TO REPRODUCE
Note: these steps rely on Bug 460245 being present for easy modification detection, but the bug is general.
1. Open the kdevelop project in KDevelop and build it.
2. Comment out the `ki18n_install(po)` line in the top-level CMakeLists.txt and save the file.
3. While CMake configuring is in progress (see the output in the Build tool view), uncomment the line again.
4. Build the kdevelop project again.

OBSERVED RESULT
The build process is instant and the Build tool view contains the "ninja: no work to do." line.

EXPECTED RESULT
The build process lasts several seconds and the Build tool view contains the "[2/2] Generating ts..." and "[2/2] Generating mo..." lines.

SOFTWARE/OS VERSIONS
Manjaro GNU/Linux, Xfce, X11
KDE Frameworks Version: 5.98
Qt Version: 5.15.6+kde

ADDITIONAL INFORMATION
The steps to reproduce 2-3 are contrived. The actual steps that lead to my discovery of this bug are:
2. git stash the commented-out-`ki18n_install(po)` line diff.
2.5. git rebase --continue
3. git stash pop
Comment 1 Milian Wolff 2022-10-17 07:23:57 UTC
> The build process is instant and the Build tool view contains the "ninja: no work to do." line.

This comes from CMake and as such this is a bug within CMake upstream. I'm inclined to close this report, as I don't see a good solution to handling it locally?
Comment 2 Igor Kushnir 2022-10-17 08:16:00 UTC
(In reply to Milian Wolff from comment #1)
> > The build process is instant and the Build tool view contains the "ninja: no work to do." line.
> 
> This comes from CMake and as such this is a bug within CMake upstream. I'm
> inclined to close this report, as I don't see a good solution to handling it
> locally?
This may be a bug in CMake. But KDevelop surely exacerbates it by running CMake configure as soon as a CMakeLists.txt file is changed. Can KDevelop remember the last time when a CMake reconfiguration of each project started (during the current KDevelop launch, not persistent) and compare *that* to timestamps of cmake source files?
Comment 3 Igor Kushnir 2022-10-17 08:18:56 UTC
(In reply to Igor Kushnir from comment #2)
> and compare *that* to timestamps of cmake source files?
And if KDevelop detects the situation described in this bug, force CMake reconfiguration.
Comment 4 Milian Wolff 2022-10-17 09:51:58 UTC
here's what I added to a project that I'm working on at work:

```
# prevent concurrent cmake runs which could fubar the cmake cache
# really, this should be handled directly in CMake, but for now we can
# at least prevent madness through this manual workaround
# this is required for developers who use IDEs that run cmake automatically
# to update the cache. if a build step is then run in an external terminal
# we could sometimes end up with two cmake processes writing to the cache file
# concurrently... which is a sure recipe for disaster
message(STATUS "Locking build directory ${CMAKE_CURRENT_BINARY_DIR}...")
file(LOCK ${CMAKE_CURRENT_BINARY_DIR} DIRECTORY GUARD PROCESS)
message(STATUS "Build directory locked...")
```

we could add something like this to at least ensure the second cmake configure step waits for the first one to finish

I don't understand how your proposed handling of timestamps would help - we would need to know when cmake is finished and only then could we start another job. since that may happen externally too, we cannot really guard against that only from within kdevelop? we might prevent multiple configure jobs running concurrently from within kdevelop as a starter, which might help your situation... but there would be easy ways to break that too
Comment 5 Igor Kushnir 2022-10-17 11:50:34 UTC
KDevelop happily runs cmake-configure a second time when the user requests build, while automatic cmake-configure is running after a change to CMakeLists.txt. Preventing KDevelop from running multiple cmake-configure for the same project simultaneously is a good idea.

However, the steps in this bug report don't lead to concurrent cmake-configure runs. So a different fix would be needed here. Possibly based on timestamps. I don't think taking external cmake-configure processes into account is particularly important here, because the user is unlikely to start a build in an external terminal and then immediately modify and save a cmake file. Seems to be much less likely than two close-by modifications of a cmake file without explicit build requests in an external terminal.
Comment 6 Igor Kushnir 2023-01-22 16:52:26 UTC
Git commit 419d3a5de17134aadbb26edaf7a4b3a369af6567 by Igor Kushnir.
Committed on 22/01/2023 at 16:28.
Pushed by igorkushnir into branch 'master'.

cmake: improve detection of outdated project data

The current algorithm is simple: consider project data outdated if the
API reply index file was last modified before a possibly generated
source CMake file. Unfortunately this algorithm is unreliable:
1) does not detect source CMake file modifications made during a CMake
   configure or generate step;
2) ignores CMakeCache.txt modifications.

Improvements in this commit:
1. Consider project data invalid if KDevelop's CMake file API client
query file does not exist. KDevelop never removes this file. The absence
of the client query file can mean that it has never been created, in
which case there can be no API reply; or it was removed during a prune
job, which should have removed the API reply as well; or it was removed
for some other reason, possibly an error.
2. Consider project data outdated if the API reply index file was last
modified before the API client query file. This means that a CMake
generate step was canceled - probably because of some error.
3. Consider project data outdated if the API reply index file was last
modified before CMakeCache.txt. This can mean that the user modified
CMakeCache.txt manually but did not run cmake afterwards, or run only a
CMake configure step, but not a CMake generate step, which updates the
API reply.
4. Consider project data outdated if the API client query file was last
modified before a non-generated source CMake file. A non-generated
source CMake file can be last modified between the API client query and
reply index files for two reasons:
  1) the user edited a source CMake file during a CMake configure or
     generate step, in which case project data is out of date;
  2) the user edited a source CMake file, then run CMake configure and
     generate steps outside of KDevelop, in which case project data is
     up to date.
Since KDevelop cannot know the true reason, it should err on the side of
caution and consider project data outdated in this case. The user is not
supposed to edit generated CMake files, so ignoring their modifications
is safe enough. Generated CMake files can be modified during a CMake
configure step, which occurs after KDevelop creates the API client query
file.
5. Optimize by finishing CMake::FileApi::ImportJob as soon as it detects
that project data is out of date, if
ChooseCMakeInterfaceJob::tryDirectImport(), which discards outdated
data, creates the job.

Ignoring modifications to generated CMake files brings outdated project
data detection algorithm closer to the algorithm in
CMakeManager::integrateData(), which decides whether to reload a project
when a file is changed. The remaining and just introduced
inconsistencies between these two algorithms:
1. Modifications to external source CMake files make project data
outdated, but such modifications are not tracked. Watching each
individual external CMake file can be costly and cause issues elsewhere
if a watched item count limit is reached. QFileSystemWatcher's
documentation warns:
  The act of monitoring files and directories for modifications consumes
  system resources. This implies there is a limit to the number of files
  and directories your process can monitor simultaneously.
External source CMake files can be modified when the user updates an
external dependency of a project opened in KDevelop, or when system
updates are installed. The user should be aware of both events and can
reload or reopen the project to reconfigure it, or restart KDevelop.
2. Modifications to CMakeCache.txt make project data outdated, but such
modifications are not tracked. Reloading the project after each
CMakeCache.txt change is risky. CMakeCache.txt can be modified during a
CMake configure step. If this CMake configure step occurs outside of
KDevelop, concurrent CMake configuration runs can corrupt build data.
Even if the CMake configure step that modifies CMakeCache.txt runs
inside KDevelop, reloading the project risks creating a reload loop.

When the user edits a source CMake file during a CMake configure or
generate step, KDevelop prints a kdevelop.plugins.cmake.debug message:
"the project is being reloaded, aborting reload!" but does not
reconfigure the project automatically for now. But at least with this
commit when KDevelop is restarted, it detects that project data is out
of date and reconfigures the project.

M  +75   -8    plugins/cmake/cmakefileapi.cpp
M  +14   -1    plugins/cmake/cmakefileapi.h
M  +21   -16   plugins/cmake/cmakefileapiimportjob.cpp
M  +6    -0    plugins/cmake/cmakefileapiimportjob.h
M  +5    -3    plugins/cmake/cmakemanager.cpp
M  +2    -2    plugins/cmake/tests/test_cmakefileapi.cpp

https://invent.kde.org/kdevelop/kdevelop/commit/419d3a5de17134aadbb26edaf7a4b3a369af6567
Comment 7 Igor Kushnir 2023-01-22 16:52:34 UTC
Git commit 11010a3d4d8ddf3a217f686f20248756108fe43e by Igor Kushnir.
Committed on 22/01/2023 at 16:28.
Pushed by igorkushnir into branch 'master'.

cmake: show a warning if project is configured with outdated data

When project data is outdated right after configuring, KDevelop does not
reconfigure the project to avoid a reload loop in case of a bug or some
upstream CMake change. Project data can be outdated right after
configuring if the user edits a source CMake file during a CMake
configure or generate step. In this case reloading the project will
bring the data up to date. Show a warning that asks the user to do just
that. The user can finish editing source CMake files, reload the
project, and the reminding warning will disappear automatically. If a
reload loop occurs for some reason, the user will eventually cease
reloading and hopefully report a bug.

I plan to implement the added TODO together with future project
reloading fixes.

M  +25   -1    plugins/cmake/cmakemanager.cpp
M  +4    -4    plugins/cmake/cmakemanager.h

https://invent.kde.org/kdevelop/kdevelop/commit/11010a3d4d8ddf3a217f686f20248756108fe43e
Comment 8 Jack 2023-09-05 19:44:10 UTC
I'm not sure if I have the same problem, or just somehow related.  If I open kdevelop (pointed to a local git clone of kmymoney, with out of tree build dir) it gives me the warning about outdated CMake data.  I click Reload, the warning disappears, kdevelop runs cmake, and the warning immediately reappears.  There is no other access to the source folder at the time.  What information can I provide for troubleshooting?
Comment 9 Igor Kushnir 2023-09-06 09:18:40 UTC
(In reply to Jack from comment #8)
> I'm not sure if I have the same problem, or just somehow related.  If I open
> kdevelop (pointed to a local git clone of kmymoney, with out of tree build
> dir) it gives me the warning about outdated CMake data.  I click Reload, the
> warning disappears, kdevelop runs cmake, and the warning immediately
> reappears.  There is no other access to the source folder at the time.  What
> information can I provide for troubleshooting?
I just cloned kmymoney and reproduced this bug. Will investigate soon. 

By the way, the CMake configure step fails with the following error:
```
CMake Error at /usr/lib/cmake/LibAlkimia5-8.1/LibAlkimia5Targets.cmake:65 (set_target_properties):
  The link interface of target "Alkimia::alkimia" contains:

    Qt5::WebEngineWidgets

  but the target was not found.  Possible reasons include:
...
```
Adding WebEngineWidgets in kmymoney's CMakeLists.txt fixes this failure:
find_package(Qt${QT_MAJOR_VERSION} ${QT_MIN_VERSION} REQUIRED COMPONENTS Core DBus Widgets Svg Xml Test PrintSupport WebEngineWidgets)
Comment 10 Igor Kushnir 2023-09-06 09:37:16 UTC
Created attachment 161440 [details]
Patch to debug CMake outdated warning

Patched KDevelop with the attached file and run it as:
    QT_LOGGING_RULES='kdevelop.plugins.cmake=true' kdevelop
The relevant lines of KDevelop's output:
kdevelop.plugins.cmake: writing API client query file at     2023-09-06T12:27:23.972 - within "/path/to/build_kmymoney"
kdevelop.plugins.cmake: last modified API client query file: 2023-09-06T12:27:23.971 - within "/path/to/build_kmymoney"
kdevelop.plugins.cmake: last modified  API reply index file: 2023-09-06T12:27:27.080 - within "/path/to/build_kmymoney"
kdevelop.plugins.cmake: last modified "/path/to/kmymoney/kmymoney/templates/templates.qrc" QDateTime(2023-09-06 12:27:24.920 EEST Qt::LocalTime)
kdevelop.plugins.cmake: last modified "/path/to/kmymoney/kmymoney/icons/kmm_icons.qrc" QDateTime(2023-09-06 12:27:25.070 EEST Qt::LocalTime)
kdevelop.plugins.cmake: last modified     source CMake file: 2023-09-06T12:27:25.070 - within "/path/to/build_kmymoney"
kdevelop.plugins.cmake: API client query file is out of date (last modified before a source CMake file)

Jack, could you please check why the CMake configure step modifies the qrc files and whether it can be avoided?
Comment 11 Jack 2023-09-06 14:26:28 UTC
The cmake issue with alkimia and kmymoney is known.  There is a patch in the libalkimia master branch.  (It's due to an interaction with how Qt5WebEngineWidgets is mentioned in both cmake files if it is not actually used, based on my simplified understanding.)
I'll see what I can find or figure out about rewriting qrc files, as well as testing your patch, but not sure if today or tomorrow.
Comment 12 Igor Kushnir 2023-09-06 14:35:32 UTC
(In reply to Jack from comment #11)
> I'll see what I can find or figure out about rewriting qrc files, as well as
> testing your patch, but not sure if today or tomorrow.
There may be no need for you to test the attached patch. It only makes debug output much more verbose. I have already posted the problematic files revealed by the increased verbosity.