Bug 235440 - Parser reparses the whole project every time
Summary: Parser reparses the whole project every time
Status: RESOLVED WORKSFORME
Alias: None
Product: kdevelop
Classification: Applications
Component: general (show other bugs)
Version: unspecified
Platform: Debian testing Linux
: NOR normal
Target Milestone: ---
Assignee: kdevelop-bugs-null
URL:
Keywords: investigated, triaged
Depends on:
Blocks:
 
Reported: 2010-04-26 14:48 UTC by Romain
Modified: 2018-10-21 05:01 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Romain 2010-04-26 14:48:35 UTC
Version:           Version 3.10.0 (using KDevPlatform 0.10.2) (using KDE 4.4.2)
OS:                Linux
Installed from:    Debian testing/unstable Packages

I use KDevelop on a *huge* project (3.6M LOCs) and it's a bit frustrating that it forgets the files it has parsed when I close it and start it again.

Is it by design ?
Comment 1 Andreas Pakulat 2010-04-26 14:52:53 UTC
Kdevelop certainly doesn't do that, at least doesn't do it here on the other developers machines. Please provide more details, in particular which project manager are you using, which language does your project use. I we could get the project sources somewhere that would be very good. Also does it take the same amount of time after each startup or is it faster? Does KDevelop shut down properly when you exit it or does it exit in an unclean way?
Comment 2 Andreas Pakulat 2010-04-26 14:53:46 UTC
Oh and your version information looks incorrect. KDevelop 3.10.2 required KDevPlatform 0.10.2, while you seem to be running against 0.10.0. Thats not supported.
Comment 3 Romain 2010-04-26 15:17:11 UTC
Thanks for the fast answers. I've built kdevelop 3.10.2 and I'll check asap.

In the meantime here are some more details:

* project manager: cmake , out-of-tree build
* language: mostly C++
* source code is proprietary, sorry
* parsing takes the same time each time, as far as I can tell (it's usually a couple of hours)
* the problem happens even KDevelop shuts down properly (I still have a few crashes here and there but I can sometimes get by a whole day without a crash)
* I didn't see anything particular in the stdout/stderr...

I'm going to let it finish its run, shut it down, restart it and I'll see if it reloads the results.
Comment 4 Romain 2010-04-26 17:03:01 UTC
OK, even with a freshly compiled 3.10.2 it still takes about 30+ minutes for the background parser to finish !

I've run kdevelop on its own sources, loading+parsing took 15+ seconds, most notably about 10+ seconds of "background parser". Could it be that actually only loading the cache on my huge project (110k files, 3.6M lines) really takes 30 minutes ? The directory in ~/.kdevduchain for this session is 465MB.


Furthermore, in the logs, I have a lot of "found no languages for url FOO". In some cases it's to be expected (binary data, images, etc) but in some cases it's on a .cpp or .h file. I've checked these files and there are indeed lots of parser errors in system headers (/usr/include , boost), etc. But these errors don't prevent successful parsing of the file, I have all the pretty colors.

I'm a bit lost... What can I do to help ?
Comment 5 Milian Wolff 2010-04-26 17:09:38 UTC
Profile KDevelop while running, either with OProfile or with valgrind:

kdevelop --sessions
=> get the HASH of your session

then

KDEV_SESSION={...} valgrind --tool=callgrind kdevelop.bin

this *will* be slow, but running it for a few minutes (10 or so) and then quitting with ctrl + c will dump the contents. Send us the (non-empty) callgrind.out.%pid file.
Comment 6 Romain 2010-04-26 22:15:01 UTC
All right, here it is:  http://dl.free.fr/dnNTwGH04

It's about 13MB of callgrind data, bzipped to 2.3Mo (still too large for bugzilla). It took about 1.5hrs to load the project and I let it run about 30+ minutes doing the "parsing".

I can try to rebuild kdevplatform with more debugging info, if required.


Oh, and note that the link for the file will stay alive only 30 days. Sorry, that's the best I could do.

HTH,
Romain
Comment 7 Romain 2010-05-03 15:46:33 UTC
Hi, 

did anyone manage to take a look at the profiling data ? I still have the same behavior in 4.0.0 (congratulations on releasing, by the way): it takes about 30 minutes for the task "Background parser" to finish...
Comment 8 Milian Wolff 2010-05-03 16:07:19 UTC
Just had a quick look, doesn't show anything interesting new that can be fixed easily. But KDev should properly remember that your project was parsed and be faster the next time, no?

If this is the case, this is a dupe of bug 215968.
Comment 9 Romain 2010-05-04 16:09:21 UTC
Could be a dupe, indeed. I ran some tests and duchainify (?) reparses an existing project in about 1:30 minutes (which is good enough for me), but KDevelop still takes perhaps half an hour to do it.

I couldn't make duchainify run as slow as KDevelop, even with the --force option (which takes about 9+ minutes).

What kind of experiments should I run ? Any additional "verbose" flags I can use to trace the problem ?

TIA,
Romain
Comment 10 Andrew Crouthamel 2018-09-20 22:12:37 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least 15 days. Please provide the requested information as soon as possible and set the bug status as REPORTED. Due to regular bug tracker maintenance, if the bug is still in NEEDSINFO status with no change in 30 days, the bug will be closed as RESOLVED > WORKSFORME due to lack of needed information.

For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please set the bug status as REPORTED so that the KDE team knows that the bug is ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 11 Andrew Crouthamel 2018-10-21 05:01:38 UTC
This bug has been in NEEDSINFO status with no change for at least
30 days. The bug is now closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

Thank you for helping us make KDE software even better for everyone!