Bug 295361 - Kdevelop freezes while linking
Summary: Kdevelop freezes while linking
Alias: None
Product: kdevelop
Classification: Unclassified
Component: Output Views (show other bugs)
Version: unspecified
Platform: Compiled Sources Linux
: NOR normal (vote)
Target Milestone: 4.2.3
Assignee: kdevelop-bugs-null
Depends on:
Reported: 2012-03-05 12:19 UTC by Gabriele Menna
Modified: 2012-04-11 07:47 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In:

a linking string (15.62 KB, text/plain)
2012-03-30 23:33 UTC, Gabriele Menna

Note You need to log in before you can comment on or make changes to this bug.
Description Gabriele Menna 2012-03-05 12:19:50 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) rekonq Safari/534.34
Build Identifier: 

I'm currently working on a large c++ project. 
Kdevelop interface freezes during linking process, nothing happens until linking has ended.

Reproducible: Always

On my sistem, for each message line produced by compiler, ProcessLineMaker::slotReadyReadStdout() is triggered multiple times, bringing to gui freeze when messages are longer than usual, as it happens in my linking step, where a large number of object files are involved.
Comment 1 Gabriele Menna 2012-03-15 11:31:06 UTC
Update: the freeze happens when MakeOutputModel::addLines() is called passing it a large output string (~16.000 symbols, on my use case).

makeoutputmodel.cpp:222 is called a time for each of the available errorFormats. Regular expression evaluation, which is known to be quite slow in current qt implementation, is called several times, thus causing interface freeze.
Comment 2 Milian Wolff 2012-03-29 18:11:01 UTC
I have some ideas on how to resolve that, but I'm laking a test case... do you have an idea how to test this? maybe with a hand-written make file that uses e.g. echo with many arguments? if you could provide such a test project it would help me tremendously!
Comment 3 Milian Wolff 2012-03-30 00:25:18 UTC
Git commit 12a0e9327119533a07dbbe1d6af833acd4de0ad4 by Milian Wolff.
Committed on 29/03/2012 at 22:28.
Pushed by mwolff into branch '4.3'.

add unit test to benchmark makeoutputmodel

M  +2    -1    projectbuilders/makebuilder/CMakeLists.txt
M  +9    -13   projectbuilders/makebuilder/makeoutputmodel.cpp
M  +11   -1    projectbuilders/makebuilder/makeoutputmodel.h
A  +4    -0    projectbuilders/makebuilder/tests/CMakeLists.txt
A  +50   -0    projectbuilders/makebuilder/tests/test_makeoutputmodel.cpp     [License: LGPL (v2+)]
A  +37   -0    projectbuilders/makebuilder/tests/test_makeoutputmodel.h     [License: LGPL (v2+)]

Comment 4 Milian Wolff 2012-03-30 00:25:18 UTC
Git commit 0a916158c33a5bd2d79c9a4cfaf2521850852af4 by Milian Wolff.
Committed on 30/03/2012 at 02:19.
Pushed by mwolff into branch '4.3'.

prevent UI-lockup when adding lots of stdout lines to make output model

we now add batches of 50 lines and then return to the event loop,
continuing with the rest of the lines via a QueuedConnection

the unit test shows that (on my machine) the event loop is roughly
returning every 80ms. the test ensures that this time does not cross
the threshold of 200ms which is imo too much.

on a further note, it is interesting to see that optimizing the actual
adding of lines is not easily possible, the tons of regexp matching
is simply required... the only possible way would be to make the error
patterns configurable such that e.g. those for unused compilers or
unused toolchains are not queried.

M  +19   -3    projectbuilders/makebuilder/makeoutputmodel.cpp
M  +6    -0    projectbuilders/makebuilder/makeoutputmodel.h
M  +22   -0    projectbuilders/makebuilder/tests/test_makeoutputmodel.cpp

Comment 5 Gabriele Menna 2012-03-30 23:33:58 UTC
Created attachment 70031 [details]
a linking string
Comment 6 Gabriele Menna 2012-03-30 23:38:45 UTC
Nice, but it doesn't fix my particular use case.

test_makeoutputmodel.cpp, benchAddLines(), insert into lines just one string, the one in the attachment, try to run the test. UI lockup jumps FAR beyond 200 ms!

I'm trying to fix this by running ErrorFormat and ActionFormat regexps evaluation in a different thread.
Comment 7 Gabriele Menna 2012-04-04 11:07:12 UTC
I found out that, in my particular use case, most of the time is spent in the evaluation of the first two ErrorFormat::errorFormats regular expressions. If I comment them out, time spent inside MakeOutputModel::addLineBatch() falls from 400 to 0.01 msecs.

Since the patch you committed some days ago seems quite useless in my use case, I'd reopen this issue and try to optimize those regular expressions!
Comment 8 Milian Wolff 2012-04-04 16:16:13 UTC
yes thanks for reopening and giving me some useful data. I'll try to take a look at it again as well now.
Comment 9 Milian Wolff 2012-04-04 16:26:39 UTC
Git commit bed56dbc23ef04fc86d7a7d3054ffbf69b080152 by Milian Wolff.
Committed on 04/04/2012 at 18:25.
Pushed by mwolff into branch 'master'.

add benchmark that tests performance of makeoutputmodel for really long linker lines

it fails currently

M  +44   -0    projectbuilders/makebuilder/tests/test_makeoutputmodel.cpp
M  +1    -0    projectbuilders/makebuilder/tests/test_makeoutputmodel.h

Comment 10 Milian Wolff 2012-04-04 17:01:19 UTC
an easy fix: add a ^ to the start of the two slow regexps.

I'm thinking of commiting that to solve the issue. afaiks, gcc outputs one error per line and the errors always start at the beginning of the line
Comment 11 Milian Wolff 2012-04-04 17:03:02 UTC
Git commit 8c736770ae3ddd173dbe0cce4479c277f75df7ff by Milian Wolff.
Committed on 04/04/2012 at 19:00.
Pushed by mwolff into branch 'master'.

greatly improve performance of our compiler error-detection regexps

errors start at the beginning of a line, by adding this information
to the regexps (^ at the beginning) they can perform much better for
very long lines which can occur during linking of large projects

also adds a unit test with some error messages I found on the net
to "proof" it's still working as intended

M  +6    -6    projectbuilders/makebuilder/outputfilters.cpp
M  +53   -0    projectbuilders/makebuilder/tests/test_makeoutputmodel.cpp
M  +3    -0    projectbuilders/makebuilder/tests/test_makeoutputmodel.h

Comment 12 Gabriele Menna 2012-04-11 07:47:51 UTC
Why don't fix this issue inside 4.3 branch, too?