Parsing speed is being worked on, but it should be _much_ faster already than the time you'r reporting. Your backtrace indicates a performance problem not in the parsing, but rather in the project-manager in the itemForUrlInternal(..) function. Maybe your project has a structure where the itemForUrlInternal(..) function is specifically inefficent?

There is a tool called "duchainify" in kdevplatform/utils/duchainify that can be used to test the parsing-speed under clean conditions.

According to the strace utility, many project files are stat'ed hundreds of times consecutively. 

How do you define "clean conditions"?

By 'clean' conditions I mean that the parsing is isolated out of the IDE, without a project-manager and such.

If the parsing is slow there as well, you could supply a callgrind trace, showing up the reason why it's slow.

The info from strace is not really useful. Also, there has been work in this area, so please update to the current kdevelop from svn and check whether the situation is improved.

There is no "duchainify" in kdevplatform/utils/duchainify - at least for kdevplatform-0.9.96.

Created attachment 38542 [details]
Callgrind log for a 20 minute kdevelop run

I've started KDevelop under callgrind and let it reparse all project files for 20 minutes after which I forced a callgrind dump and aborted kdevelop.

Looking at the log of svn I think you should retry with svn instead of beta6. There's been speedup changes specifically in the "hotspots" of that callgrind output.

kdevelop svn trunk now depends on qt4.6 (which had not been publicly release yet) while opensuse 11.2 is based on qt 4.5.3, so that is not an option for me.

This is not true. KDE Libs trunk (i.e. 4.4) depends on Qt 4.6, but KDevplatform & KDevelop can be build with 4.3 (Qt 4.5.X) just fine.

(In reply to comment #7)
> kdevelop svn trunk now depends on qt4.6 (which had not been publicly release
> yet) while opensuse 11.2 is based on qt 4.5.3, so that is not an option for me.

I'd be interested to know where you heard or read this so we can clarify that.

My mistake, I had mistaken the requirements for kde trunk for requirements for kdevelop. 

Anyhow, I've just built kdevplatform and kdevelop from svn today (revisions 1060767 and 1060770 respectively). The improvements for full scheduled reparse are quite significant - it now takes just under 15 minutes - a factor of 3 improvement over the old 45 minutes. Well done!

However, it is still almost 10 times slower than full reindexing of a makefile-based project in QtCreator 1.3.0 on the same desktop.

If you can spare the time, please retry with r1060781, would be interested on what difference it makes on your project.

Cool - reparse time is down to just under 7 minutes.

I hope what you're doing is really reparsing, not 'updating' ;-)

Did you do "rm ~/.kdevduchain -R" between your tests?

Also, there is a new option to reduce the processing-level, available under "Settings -> Language Support", there you can specify the project size until which the 'full' processing is done. I think the default value is 10000, but it may count the project-files wrongly, so maybe your project was already processed in simplified mode and thus the speedup.. however this mode is still experimental and might still cause problems in code-completion and other duchain stuff.

I've tried both 10000 (default) and 100000 for min project size - it makes no significant difference on reparse time both with and without rm -rf ~/.kdevduchain before reparsing.

FYI, some more stats about the contents of the project directory: ~6000 .h/.c/.cpp files, same number of source control metadata files, ~12000 dependency information files, and ~12000 build artifacts (the project is built twice - once in unicode mode and once in mbcs mode, hence number of dependencies and build artifacts is double the number of source files).

You can easily check whether 'simplified' parsing was used: Type CTRL+ALT+m and check whether all functions are shown as "functionName(...)" instead of "functionName(args)"

It used "simplified" parsing even with 100000 for min project size ... Just out of curiosity I bumped it up to 500000, deleted .kdevduchain, and forced a reparse -  that resulted in "full" parsing.

I was also somewhat surprised to see kdevelop launch make and gcc many times during the reparse. At least that's what top reported.

Hi. Just upgraded to Kdevelop 4 beta 8 (tip of svn) from beta 6, and scheduled project reparsing appears to have a huge performance regression. My custom makefiles project consists of 9317 C++ files. Beta 6 took around 11 minutes. Beta 8 takes 33 minutes, even after a recursive removal of the .kdevduchain folder.   top reports ~75-80% CPU usage from kdevelop.bin, iotop reports many small (10-100K) reads/writes from kdevelop.bin.

Also noticed  numerous (hundreds if not more) gcc and make invocations my kdevelop during parsing - could this be the cause?

Just for comparison, QtCreator 1.3.1 from 2010.2 SDK takes < 2 minutes to reindex the project on same notebook (Intel Core2 2.53Ghz, 6Mb cache, 4GB RAM, 320Gb 7200rpm hdd). iotop reports qtcreator performing reads at over 500Kb - 1.5Mb per second, and virtually no writes.

Similar performance regression here as well.

Could you create a callgrind trace of parsing the project in question?

Will do. I noticed that at least some spawned make commands look like

make -k --no-print-directory -W './path/to/file.h' -n './path/to/file.o'

Is Kdevelop spawning a make process for each .h/.cpp in the project during the reparse?

No joy. If I run "valgrind --tool=callgrind --num-callers=40 $KDEVELOP_HOME/bin/kdevelop  -s mysession", then a 0 byte callgrind log is generated on quit from kdevelop. Adding --trace-children=yes forces callgrind to trace each spawned make process as well, and the system literally freezes. Had to reboot to recover.

Created attachment 40968 [details]
Callgrind trace of kdevelop.bin during a full project reparse

First part of the attachment.

Created attachment 40969 [details]
Callgrind trace of kdevelop.bin during a full project reparse

Second part of the attachment. Use the following to recover the original file

cat callgrind.out.11625.bz2aa callgrind.out.11625.bz2ab  > callgrind.out.11625.bz2
bunzip2 callgrind.out.11625.bz2

Got it - the trick was to set the KDEV_SESSION environment variable, and then profile kdevelop.bin, and not kdevelop.

Hm thanks for the effort, sadly the callgrind doesn't show the bottleneck direclty. But I also think that the includepath builder (the make runs) are to blame, we'll have to see how to improve that.

Makes sense. IMHO, Kdevelop seems like a heavily-multithreaded application, but running it under callgrind will probably have the effect of serializing all threads due to callgrind's emulation, so mutex contention will probably be significantly reduced when running kdevelop under callgrind, and won't show up in the trace. Plus callgrind is not the best tool for profiling apps doing lots of I/O like kdevelop.

*** Bug 228295 has been marked as a duplicate of this bug. ***

SVN commit 1096804 by mwolff:

Disable dumping of context when 'previous parameter declaration context didn't get used??'.

Speeds up parsing of boost libs:

        before    | after
real    7m49.239s | 6m30.723s 
user    6m40.697s | 6m15.995s
sys     0m23.012s | 0m9.513s

David, if you think that this should get fixed proplery, please explain what this warning means.

CCMAIL: zwabel@googlemail.com
CCBUG: 215968


 M  +3 -3      contextbuilder.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1096804

SVN commit 1097117 by mwolff:

utilize new lengthFromIndex function, reduces total instr cost esp. for files like the boost/typeof/vectorXYZ.hpp files. The vector50.hpp e.g. has now a 15% lower cost

CCBUG: 215968

 M  +1 -1      lexer.cpp  
 M  +1 -1      rpp/pp-location.cpp  
 M  +3 -3      rpp/pp-stream.cpp  
 M  +1 -1      rpp/pp-stream.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1097117

I did some more profiling, esp. with the project that bschindler provided me with. Essentially it's just a checkout of boost. To my eyes the custom include path resolver is not the bottleneck, esp. considering that it's already cached per dir. Instead I noticed that esp. files like boost/typeof/vectorXYZ.hpp are really slow at parsing. See their contents for example at:

http://sodium.resophonic.com/git/boost/tree/boost/typeof/vector200.hpp?id=6936053b064a7cef59fdbaca963d49c60708ebf2

I'm not sure how common such files are, but I fear that every project using boost might include them at one point or another, hence improving their parsing should be helpful.

You can take this profile as an example, showing that the positionAt in pp-location.cpp is really slow since it has to lookup the length of many offsets with a method marked as expensive:

http://milianw.de/files/traces/callgrind.out.4895.bz2

No boost in my project, just plain stl.

Still, the root of the problem points to a very generic area that has nothing to do with the boost file. Improving the area would also increase the speed of parsing any other file.

At least the wakeOne() at position 3 should be gone from the callgrind log now.

There is some interesting info in the callgrind log, it also reveals a problem in CodeModel::addItem(..), which seems to be doing too much (that's where all the IndexedQualifiedIdentifier constructor- and destructor-calls come from).

The bottleneck for custom make file projects is now the IncludePathResolver that runs up to 6 make calls for _each_ file :-/

I'll look into it and try to fix it.

SVN commit 1111174 by mwolff:

fix implementation of operator== in ModificationRevisionSet, that was totally broken
and explains why some caches (I saw it in the includepathresolver) didn't apply at all, wonder how this could have slipped so long...

Now that the cache applies, Custom Make Manager projects are much faster to parse since the cached results per directory
apply, and make doesn't have to be called once for every file.

CCBUG: 215968

 M  +1 -1      modificationrevisionset.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1111174

SVN commit 1111176 by mwolff:

merge make runs for different possible targets, should make parsing of some custom make file projects already 3x faster

since I have no clue about unsermake, I don't know whether it works there as well, anyone?

CCMAIL: kdevelop-devel@barney.cs.uni-potsdam.de
CCBUG: 215968

 M  +6 -12     includepathresolver.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1111176

SVN commit 1111177 by mwolff:

merge make command for absolute and relative path, should make certain custom make projects
up to 2x faster to parse

CCBUG: 215968

 M  +10 -26    includepathresolver.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1111177

Running into the same problem as well, but most of my time is in itemsForUrlInternal.

I made a patch that I believe works the same but using a QMultiHash instead of recursion, the only thing I am note sure about is whether using the IndexedString is appropriate for this.

Created attachment 42936 [details]
Replace recursion inside itemsForUrl with a QMultiHash lookup.

@Milian: If caching works, then that change alone won't make anything
3 times faster. The change should be ok though, although I'm worried
about doing this so close before the release, I hope you've tested it
intensively.

@Brandon Ehle: Your patch seems to make a lot of sense. Can someone
with a better overview of the project architecture check, and
eventually apply it?

@ David: this commit was done before I found the error in the caching, the statement in the commit message is not applicable anymore. But imo the change is still approriate and I tested it as much as possible.

@Brandon: Can you share your project or at least the callgrind log files showing the bottleneck? Esp. a comparison would be nice (before <-> after).

Furthermore, I'd appreciate it, if you could put your patch on http://reviewboard.kde.org/ , I have a few comments on it and there I could comment directly on the changes.

@Milian: Ah sorry I didn't look at the date, and thought the change was fresh. :)

Added as http://reviewboard.kde.org/r/3722/

Will do some before / after comparisons tonight after I get back from work.

Git commit 5c67406cbb35012d87c2560ec74366c1701ac288 by Milian Wolff.
Committed on 08/09/2011 at 16:05.
Pushed by mwolff into branch 'master'.

optimize: only use ForegroundLock in ParseJob::readContents when really required

We now use a variant of the double-checked-initialization pattern, as the
trackerForUrl should be threadsafe according to its apidox. Only when we have
a tracker do we need to get the foreground lock.

When opening a project, the majority of files won't be opened in the editor,
and all others can thus be parsed/read without requiring a ForegroundLock

In my benchmarks (using the free non-commercial edition of VTune) this improved
the performance of a duchainify run by 25% or ~10s.

CCBUG: 215968

M  +22   -19   language/backgroundparser/parsejob.cpp

http://commits.kde.org/kdevplatform/5c67406cbb35012d87c2560ec74366c1701ac288

Git commit acabbf5b22030266968bb60a3ae50109c92c00f6 by Milian Wolff.
Committed on 09/09/2011 at 18:02.
Pushed by mwolff into branch 'master'.

optimize: implement hash-based lookup for itemsForUrl

When loading big projects with lots of files, much time was spent in
Project::itemsForUrl (up to a minute in total in one of my tests).

Now we use a QHash<uint, ProjectBaseItem*> lookup table, which brings
down the total time to a mere fraction of a second.

The lookup table is saved in the ProjectModel and is updated on demand
from the ProjectBaseItem.

Introduces a new ProjectModel::itemsForUrl which will return all files
for a url without filtering by project.

CCBUG: 215968

M  +5    -0    language/duchain/indexedstring.cpp
M  +1    -0    language/duchain/indexedstring.h
M  +40   -1    project/projectmodel.cpp
M  +4    -0    project/projectmodel.h
M  +15   -6    shell/project.cpp

http://commits.kde.org/kdevplatform/acabbf5b22030266968bb60a3ae50109c92c00f6

Is there any objective way to close this bug? :)

Many improvements happened over time, I guess it can be closed as fixed.

Congratulations guys!!