Bug 146144 - kdevelop consumes all CPU resourses while building projects
Summary: kdevelop consumes all CPU resourses while building projects
Status: RESOLVED FIXED
Alias: None
Product: kdevelop
Classification: Applications
Component: general (show other bugs)
Version: 3.4.1
Platform: unspecified Linux
: NOR normal
Target Milestone: ---
Assignee: kdevelop-bugs-null
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-05-30 12:41 UTC by Vadym Krevs
Modified: 2008-11-13 00:32 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Requested callgrind output, part 1 (659.37 KB, application/octet-stream)
2007-05-31 13:35 UTC, Vadym Krevs
Details
Requested callgrind output, part 2 (617.23 KB, application/octet-stream)
2007-05-31 13:35 UTC, Vadym Krevs
Details
Requested callgrind output, part 3 (386.56 KB, application/octet-stream)
2007-05-31 13:36 UTC, Vadym Krevs
Details
Patch1 to eliminate high CPU usage during build (445 bytes, text/plain)
2007-07-14 12:37 UTC, Vadym Krevs
Details
Patch2 to eliminate high CPU usage during build (1.07 KB, text/plain)
2007-07-14 12:37 UTC, Vadym Krevs
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Vadym Krevs 2007-05-30 12:41:14 UTC
Version:           3.4.1 (using KDE 3.5.7 "release 10.1" , openSUSE )
Compiler:          Target: x86_64-suse-linux
OS:                Linux (x86_64) release 2.6.18.8-0.1-default

Just upgraded to the latest KDE 3.5.7 packages for openSUSE 10.2 for x86_64. And now when any project is being rebuilt, Kdevelop 3.4.1 consumes 100% of CPU - more than make and GCC/G++ compiler jobs taken together. Compiler output frequently freezes in the messages pane, etc. The same happens if I kickoff build from a separate konsole session while kdevelop is running. Looks as if kdevelop is doing some filesystem polling ...
Perhaps it is related to project size - mine contain ~14K files on average. 


# rpm -q kdevelop3 kdebase3 kdelibs3
kdevelop3-3.4.1-3.2
kdebase3-3.5.7-12.1
kdelibs3-3.5.7-10.1
Comment 1 Andreas Pakulat 2007-05-30 14:37:03 UTC
What type of project do you use? Yes some project manager do watch changes in the filesystem, but they do so only for the source dirs. Depending on your System different methods are used to do the watching. One such method is using a file-monitor called "fam" that is known to hog CPU, another is gamin that seemingly can do the same. If you have neither of those installed the watching uses the inotify kernel function, if available. Last resort is polling the filesystem for changes, but that shouldn't happen on any decend system nowadays. I suspect fam or gamin at the moment.
Comment 2 Vadym Krevs 2007-05-30 23:12:14 UTC
I use "Generic C++ Application" project type with an external makefile. The project itself was created via "Project->Import Existing Project". 

My system does have the "fam" package installed, and the kernel (`rpm -q kernel-default` -> kernel-default-2.6.18.8-0.3) has full inotify support.
 
The thing is Kdevelop 3.4.0 and all previous version earlier did not have this problem on the same system and Kdevelop 3.4.1 is nearly unusable because of it.
Comment 3 Andreas Pakulat 2007-05-30 23:31:23 UTC
Right, the watching of directories was added in KDevelop3.4 after the 3.4.0 release. As I said de-installing fam or replacing it with gamin might help. fam is known to hog the CPU. gamin is a fully compatible replacement for fam and often works better, even though I've also seen complains about gamin. If that works for you please let me know, as far as KDevelop is concerned there's nothing that can be done.
Comment 4 Vadym Krevs 2007-05-30 23:52:54 UTC
Alas, uninstalling fam is not an option as kdelibs rpm has a hard dependency on it. And there is no gamin package for openSUSE. 

As to what can or cannot be done, this new "enhancement" is a clear regression compared to 3.4.0. kdevelop is consuming all CPU even when make is kicked off from a separate console window when there are no targets to be rebuilt. I.e. Kdevelop appears to be watching not only for new/modified files only, but also for each file read in read-only mode by any process ...

Anyway, if you want peope to use kdevelop, then you'll fix this, otherwise ...
Comment 5 Andreas Pakulat 2007-05-31 00:53:28 UTC
As I tried to explain its not KDevelop that hogs your CPU even if top tells you that, its fam (probably). 

If you want more evidence for one or the other please provide valgrind output from such a session.

KDevelop watches all directories in the project dir (recursively) that are not excluded from the project. There's an option to tell KDevelop that certain files and/or directories should never be considered project files, even if they match the project filetypes. Also KDevelop doesn't watch any files, it only watches directories for adding/removing of files.

If openSUSE provides only a broken file alteration monitor and forces you to use that we can't really do anything about it.

Last but not least: Threatening that you won't use KDevelop anymore will not help you in any way. In fact there are quite a number of people who use KDevelop3.4 custom makefiles that seemingly don't have this problem.
Comment 6 Vadym Krevs 2007-05-31 10:56:05 UTC
Andreas, I am also a software developer just like you, and I would very much like to believe that my software is bug free and that all "bugs" in it are caused by buggy operating system/hardware/stupid users/etc. But real life is different. 

Anyhow, back to the problem at hand. What skin and arguments should I use for valgrind? I've only used memcheck and massif in the past, and and it is not clear to me how running kdevelop under valgrind with these skins would help us identify the problem.

Comment 7 Andreas Pakulat 2007-05-31 11:51:27 UTC
As I already tried to explain: FAM is widely known as a buggy implementation that often causes cpu hogging. gamin seems to be a good alternative and can be used as a drop-in replacement (Debian packages for gamin provide the same library names as fam does). 

Back to the problem indeed: Run valgrind --tool=callgrind to get a profile on which function is called most, this can give us a hint as to where the hogging occurs. Alternatively you can run kdevelop in gdb and when it hogs your CPU interrupt it and print a backtrace, that can also give some hints. Please add the information to this bugreport once you have it.
Comment 8 Andreas Pakulat 2007-05-31 11:55:57 UTC
PS: I am very well aware of the fact that KDevelop3.4 has some bugs even for C++ projects and  a few features that I'd like to see too. However from this bugreport all I can assume is that your problem is caused by fam, simply because KDevelop doesn't do any "active polling", in fact all it does when it gets notified of a directory change is checking a list of strings and eventually appending or removing from another list.
Comment 9 Vadym Krevs 2007-05-31 12:42:11 UTC
Stack trace from gdb is below. I've also strace'd kdevelop - it seems to access()/lstat() each h/c/cpp file in the project ....

$ gdb -pid 28901
GNU gdb 6.5
....
0x00002b9cc3b142dc in _int_malloc () from /lib64/libc.so.6
(gdb) bt
#0  0x00002b9cc3b142dc in _int_malloc () from /lib64/libc.so.6
#1  0x00002b9cc3b16386 in malloc () from /lib64/libc.so.6
#2  0x00002b9cc3862fcd in operator new () from /usr/lib64/libstdc++.so.6
#3  0x00002b9cc3278ec3 in QDomDocumentPrivate::createTextNode () from /usr/lib/qt3/lib64/libqt-mt.so.3
#4  0x00002b9cc3279a04 in QDomDocument::createTextNode () from /usr/lib/qt3/lib64/libqt-mt.so.3
#5  0x00002b9cc1872f3f in DomUtil::writeListEntry () from /opt/kde3/lib64/libkdevelop.so.1
#6  0x00002aaaaaad249a in CustomProjectPart::updateBlacklist () from /opt/kde3/lib64/kde3/libkdevcustomproject.so
#7  0x00002aaaaaad90a9 in CustomProjectPart::slotDirDirty () from /opt/kde3/lib64/kde3/libkdevcustomproject.so
#8  0x00002aaaaaae2eda in CustomProjectPart::qt_invoke () from /opt/kde3/lib64/kde3/libkdevcustomproject.so
#9  0x00002b9cc307559c in QObject::activate_signal () from /usr/lib/qt3/lib64/libqt-mt.so.3
#10 0x00002b9cc3075e86 in QObject::activate_signal () from /usr/lib/qt3/lib64/libqt-mt.so.3
#11 0x00002b9cc1ca97d5 in KDirWatch::dirty () from /opt/kde3/lib64/libkio.so.4
#12 0x00002b9cc1ca99b3 in KDirWatchPrivate::emitEvent () from /opt/kde3/lib64/libkio.so.4
#13 0x00002b9cc1cedfaf in KDirWatchPrivate::slotRescan () from /opt/kde3/lib64/libkio.so.4
#14 0x00002b9cc1cef29c in KDirWatchPrivate::qt_invoke () from /opt/kde3/lib64/libkio.so.4
#15 0x00002b9cc307559c in QObject::activate_signal () from /usr/lib/qt3/lib64/libqt-mt.so.3
#16 0x00002b9cc3076273 in QObject::activate_signal () from /usr/lib/qt3/lib64/libqt-mt.so.3
#17 0x00002b9cc3094905 in QTimer::event () from /usr/lib/qt3/lib64/libqt-mt.so.3
#18 0x00002b9cc301e425 in QApplication::internalNotify () from /usr/lib/qt3/lib64/libqt-mt.so.3
#19 0x00002b9cc301f057 in QApplication::notify () from /usr/lib/qt3/lib64/libqt-mt.so.3
#20 0x00002b9cc28234d8 in KApplication::notify () from /opt/kde3/lib64/libkdecore.so.4
#21 0x00002b9cc3014a6f in QEventLoop::activateTimers () from /usr/lib/qt3/lib64/libqt-mt.so.3
#22 0x00002b9cc2fd479d in QEventLoop::processEvents () from /usr/lib/qt3/lib64/libqt-mt.so.3
#23 0x00002b9cc3032fd3 in QEventLoop::enterLoop () from /usr/lib/qt3/lib64/libqt-mt.so.3
#24 0x00002b9cc3032e82 in QEventLoop::exec () from /usr/lib/qt3/lib64/libqt-mt.so.3
#25 0x0000000000407345 in main ()
(gdb) cont
Continuing.

Program received signal SIGINT, Interrupt.
[Switching to Thread 47952412159328 (LWP 28901)]
0x00002b9cc3b592c5 in _lxstat () from /lib64/libc.so.6
(gdb) bt
#0  0x00002b9cc3b592c5 in _lxstat () from /lib64/libc.so.6
#1  0x00002b9cc32cd0cf in QFileInfo::doStat () from /usr/lib/qt3/lib64/libqt-mt.so.3
#2  0x00002b9cc32cd450 in QFileInfo::isFile () from /usr/lib/qt3/lib64/libqt-mt.so.3
#3  0x00002aaaaaad9bb5 in CustomProjectPart::allFiles () from /opt/kde3/lib64/kde3/libkdevcustomproject.so
#4  0x00002b9cd092e03c in CppCodeCompletion::computeFileEntryList () from /opt/kde3/lib64/kde3/libkdevcppsupport.so
#5  0x00002b9cd094d958 in CppCodeCompletion::qt_invoke () from /opt/kde3/lib64/kde3/libkdevcppsupport.so
#6  0x00002b9cc3075500 in QObject::activate_signal () from /usr/lib/qt3/lib64/libqt-mt.so.3
#7  0x00002b9cc18310d5 in KDevProject::removedFilesFromProject () from /opt/kde3/lib64/libkdevelop.so.1
#8  0x00002aaaaaae2060 in CustomProjectPart::removeFiles () from /opt/kde3/lib64/kde3/libkdevcustomproject.so
#9  0x00002aaaaaad90c2 in CustomProjectPart::slotDirDirty () from /opt/kde3/lib64/kde3/libkdevcustomproject.so
#10 0x00002aaaaaae2eda in CustomProjectPart::qt_invoke () from /opt/kde3/lib64/kde3/libkdevcustomproject.so
#11 0x00002b9cc307559c in QObject::activate_signal () from /usr/lib/qt3/lib64/libqt-mt.so.3
#12 0x00002b9cc3075e86 in QObject::activate_signal () from /usr/lib/qt3/lib64/libqt-mt.so.3
#13 0x00002b9cc1ca97d5 in KDirWatch::dirty () from /opt/kde3/lib64/libkio.so.4
#14 0x00002b9cc1ca99b3 in KDirWatchPrivate::emitEvent () from /opt/kde3/lib64/libkio.so.4
#15 0x00002b9cc1cedfaf in KDirWatchPrivate::slotRescan () from /opt/kde3/lib64/libkio.so.4
#16 0x00002b9cc1cef29c in KDirWatchPrivate::qt_invoke () from /opt/kde3/lib64/libkio.so.4
#17 0x00002b9cc307559c in QObject::activate_signal () from /usr/lib/qt3/lib64/libqt-mt.so.3
#18 0x00002b9cc3076273 in QObject::activate_signal () from /usr/lib/qt3/lib64/libqt-mt.so.3
#19 0x00002b9cc3094905 in QTimer::event () from /usr/lib/qt3/lib64/libqt-mt.so.3
#20 0x00002b9cc301e425 in QApplication::internalNotify () from /usr/lib/qt3/lib64/libqt-mt.so.3
#21 0x00002b9cc301f057 in QApplication::notify () from /usr/lib/qt3/lib64/libqt-mt.so.3
#22 0x00002b9cc28234d8 in KApplication::notify () from /opt/kde3/lib64/libkdecore.so.4
#23 0x00002b9cc3014a6f in QEventLoop::activateTimers () from /usr/lib/qt3/lib64/libqt-mt.so.3
#24 0x00002b9cc2fd479d in QEventLoop::processEvents () from /usr/lib/qt3/lib64/libqt-mt.so.3
#25 0x00002b9cc3032fd3 in QEventLoop::enterLoop () from /usr/lib/qt3/lib64/libqt-mt.so.3
#26 0x00002b9cc3032e82 in QEventLoop::exec () from /usr/lib/qt3/lib64/libqt-mt.so.3
#27 0x0000000000407345 in main ()
Comment 10 Andreas Pakulat 2007-05-31 13:20:29 UTC
Looking at the two backtraces it seems that your build system rapidly creates and deletes files, is that correct? In that case I suggest you find a way to do out-of-source builds instead, as you can see each deletion or addition of a file causes some computation (the first backtrace looks wrong, there shouldn't be any writing to the project file at that point) and there's no way to add a way to turn off the watching of directories. If you already have special dirs in which these add/deletes happen (like a obj or build subdir in each source dir) then you should tell kdevelop to exclude these by adding them to the blacklist (right click the directory).

A callgrind output would, nevertheless, be helpful in finding out which parts take most of the time. I did that once a while back for the qmake manager and it helped a lot making it much faster.
Comment 11 Vadym Krevs 2007-05-31 13:32:25 UTC
Hmm, I don't think so. All I do is kick off make doing a "dry run" when nothing needs to be recompiled or relinked because all targets are up to date. 

Callgrind output is attached.
Comment 12 Vadym Krevs 2007-05-31 13:35:22 UTC
Created attachment 20742 [details]
Requested callgrind output, part 1

This callgrind output corresponds to the following activity: start kdevelop
under callgrind, wait until it loads, then execute gmake from a seperate
terminal window twice (no targets had to be remade) and exit kdevelop.
Comment 13 Vadym Krevs 2007-05-31 13:35:56 UTC
Created attachment 20743 [details]
Requested callgrind output, part 2
Comment 14 Vadym Krevs 2007-05-31 13:36:26 UTC
Created attachment 20744 [details]
Requested callgrind output, part 3
Comment 15 Vadym Krevs 2007-05-31 13:37:19 UTC
Hope this helps. Let me know if there are any other tests I can perform.
Comment 16 David Nolden 2007-06-01 10:48:41 UTC
I think I had this too once. As a workaround, does it help to close the file-tree view?
Comment 17 Vadym Krevs 2007-06-01 15:31:24 UTC
The file-tree plugin isn't even loaded. Closing and unloading file-selector and file-list has no effect either. 
Comment 18 Vadym Krevs 2007-06-02 19:04:34 UTC
Hmm, the last gdb stacktrace shows what is likely to be the problem. KDirWatch emits a signal for a single changed file, and CustomProjectPart::allFiles() goes through the list of all project's source files and stats every single one of them ... This is never going to scale well on any system regardless of which file monitoring backend is used. 
Comment 19 Andreas Pakulat 2007-06-02 19:40:56 UTC
So indeed when you build your project various project files are changed by that (else the KDirWatch wouldn't emit the fileChanged signal). 

The stat occurs because the list of files also contains directories in CustomProjectPart and IIRC some parts of KDevelop don't work when allFiles() returns also directories. I'm just rebuilding KDevelop3 and do some testing, if everything still works when it returns directories I'm going to commit that. If that introduces other problems somebody needs to sit down and write up some code that puts dirs and files into separate variables, I can't do that due to time constraints (its quite a bunch of work).

I can happily work with KDevelop3.4 on code bases as large as KDevelop4 or kdelibs/KDE4 without any problem, even though there are delays when adding/removing a file or directory.
Comment 20 Andreas Pakulat 2007-06-02 23:14:51 UTC
SVN commit 670887 by apaku:

Always return the full list of sources, this avoids having to stat the whole
project tree but the list of sources will contain directories and not just
files.
So far I haven't seen any problems in other plugins, thus I'm committing.
CCBUG: 146144


 M  +1 -7      customprojectpart.cpp  


--- branches/KDE/3.5/kdevelop/buildtools/custommakefiles/customprojectpart.cpp #670886:670887
@@ -665,13 +665,7 @@
 
 QStringList CustomProjectPart::allFiles() const
 {
-    QStringList sources;
-    for( QStringList::const_iterator it = m_sourceFiles.begin(); it != m_sourceFiles.end(); ++it )
-    {
-        if( QFileInfo( projectDirectory()+"/"+*it ).isFile() )
-            sources << *it;
-    }
-    return sources;
+    return m_sourceFiles;
 }
 
 
Comment 21 Vadym Krevs 2007-06-03 11:59:10 UTC
I've tried the same on my codebase and it does not resolve the issue. One thing that does help is to disable directory watching in CustomProjectPart::addDirWatches(). Was this directory watching added as part of a fix for http://bugs.kde.org/show_bug.cgi?id=69623?

$ diff -C3 customprojectpart.cpp customprojectpart.cpp.my
*** customprojectpart.cpp       2007-06-03 10:54:04.000000000 +0100
--- customprojectpart.cpp.my    2007-06-02 19:39:05.000000000 +0100
***************
*** 513,519 ****

  void CustomProjectPart::addDirWatches( const QString& absPath )
  {
!     dirwatch->addDir( absPath );
      QStringList entries = QDir( absPath ).entryList( QDir::Dirs );
      entries.remove(".");
      entries.remove("..");
--- 513,519 ----

  void CustomProjectPart::addDirWatches( const QString& absPath )
  {
!     //dirwatch->addDir( absPath );
      QStringList entries = QDir( absPath ).entryList( QDir::Dirs );
      entries.remove(".");
      entries.remove("..");
Comment 22 Vadym Krevs 2007-06-03 12:43:59 UTC
Regarding changed files during the build: I've straced make and all the sub-makes it creates. As expected, no project source files or object files or targets are created/modified/etc. However, make itself appears to create temporary files on the fly and these temporary files end up inside the build tree because neither TMP nor TEMP environment variables are set.

I've reinstalled the original kdevelop package - with directory watching enabled - and then started kdevelop, opened my project, and kicked off the build from a separate konsole window having set TMP to /tmp. This resulted in Kdevelop's CPU consumption dropping from 100% to about 30% during the build, which is much better than before, although it should not be consuming any CPU at all when a build is done externally.

However, all this is for a "dry-run" build when there are no targets to be remade. Any clean or incremental build still causes kdevelop to consume 100% of CPU.

IMHO, it would help to have a project option on the Custom Manager tab to turn off directory watching completely for those people who do not share their directory tree with others like the scenario described in bug 69623.
Comment 23 Andreas Pakulat 2007-06-03 13:01:27 UTC
Hmm, I have to admit that I didn't yet test with an in-source-build (i.e. sourcedir == builddir), I think the creation of all the object and lib files are causing kdevelop to do quite a bit of work.

As far as running an external build: Kdevelop consumes cpu in that case because files or dirs in the source tree change during that. This kicks off the KDirWatcher and thus KDevelop does a few things, which needs CPU. 

As far as the option to disable the dirwatching goes: I can't do that, KDE3 is feature frozen, which includes no new user-visible strings.
Comment 24 Vadym Krevs 2007-06-03 15:34:16 UTC
kdevelop could check for an environment variable and if it is set then disable directory watching ... Hacky but at least it's a workaround.
Comment 25 Vadym Krevs 2007-06-05 00:06:27 UTC
BTW, I've done another test. I've reenabled directory watching in CustomProjectPart::addDirWatches(), and commented out all code in CustomProjectPart::slotDirDirty(). Then I performed a clean build from konsole - and kdevelop did not consume any CPU during the build at all. Which proves that the original problem is not caused by the underlying filesystem monitoring backend be it FAM or inotify. It is whatever is done in response to filesystem events that's making kdevelop consume CPU.
Comment 26 Andreas Pakulat 2007-06-06 23:35:41 UTC
Hmm, I had another look at that function. Yes it is probably not the fastest thing on earth, but I currently don't see a way to significantly increase the performance. I'll do a valgrind run here on my own, as it seems even on saving the dirdirty slot is called (at least kdevelop "does something") and that shouldn't happen, except if KDirWatch is broken.

If you have an idea how to speed this code up (basically it checks which files/dirs where removed, removes these files/dirs from the blacklist, then removes the rest from the sourcelist - recursively - and adds all new files to a "to-be-added" list) I'd be glad to accept a patch. However I don't have the time to come up with one myself at the moment. In Qt4 I'd use QSet or QHashSet to get some speedup during lookups and such things, but in Qt3 all one gets is QStringList.
Comment 27 Rob L 2007-06-28 15:27:43 UTC
I have a similar problem with kdevelop using 100% cpu (according to top, g++ is running on another machine via rsh) towards the end of the build (linking) for ~30s.  This first appeared with kdevelop3-3.4.1-70.2  (kdevelop3-3.4.1-64 was ok) and is now fairly common.

VTune sampling suggests most of the clock ticks are in QRegExpEngine::matchHere(void) & QRegExpEngine::CharClass::in(QChar) const.
Comment 28 Rob L 2007-06-28 17:15:51 UTC
my build is 'out of source', the only files that should be changing are in blacklisted directories.
Comment 29 Vadym Krevs 2007-07-02 18:17:21 UTC
Just tested the new "most stable" Kdevelop shapshot which was released on 2007-June-14 according to http://www.kdevelop.org/index.html?filename=3.4/download.html, and the results are not good.

The following source RPM was downloaded and rebuilt: http://software.opensuse.org/download/home:/amilcarlucas/openSUSE_10.2/src/kdevelop3-3.4.1-77.1.src.rpm

Now even with the CustomProjectPart::slotDirDirty() commented out, Kdevelop consumes more CPU than the c++ compiler actually doing the build. Again, there are visible pauses in the compiler output display. 

I guess we ought to brace ourselves for Kdevelop4 ...
Comment 30 Andreas Pakulat 2007-07-02 19:32:58 UTC
As I said before, statements like "KDevelop uses 100% CPU" are totally useless unless you add profiling data. Especially as you say that even after basically disabling the directory watching (at least the code that runs when something changed) doesn't help.
Comment 31 Rob L 2007-07-03 16:28:46 UTC
using latest svn (Revision: 682763), and interrupting during 100% cpu load it appears the (my) problem is in CompileErrorFilter::processLine(), compileerrorfilter.cpp:94

This is being called on the link line 'g++ -m32 -Wl,-rpath ... -shared -o ...' which is pretty long in my project.

p line.length():
$1 = 13684
printqstring regExp.pattern():
([^:    ]+):([0-9]+):([^0-9]+)

Can this regex be made more efficient? Maybe you could ignore any line beginning 'g++', 'gcc', 'ld', 'ifort', 'cc', 'aCC' etc. as these won't contain the error messages.
Comment 32 Rob L 2007-07-03 18:47:10 UTC
On further inspection it looks like a whole load of regexs are being matched for errors from various compilers, change of directory etc.  It could really do with some mechanism to avoid matching echoed compiler commands (particularly link) with any of these.

This appears to be a know problem as there are various comments:
// avoid QRegExp if possible. This regex performs VERY badly with large inputs,
/// \FIXME This is very slow, possibly due to the regexr matching. It can take
//900-1000ms to execute on an Athlon XP 2000+, while the UI is completely blocked.
Comment 33 Rob L 2007-07-05 11:25:55 UTC
The following avoids the problem for me (svn Revision 683236):

diff parts/outputviews/compileerrorfilter.cpp{,.0}
80,82d79
<         static QRegExp cmdEcho("^\\s*(g\\+\\+|gcc|icc|ifort|ld)\\s+");
<         if (cmdEcho.search(line) >=0 ) //avoid searching echoed compiler commands
<           return;

It may be possible to make this regex more efficient, and I have only tested against g++ builds.
Comment 34 Rob L 2007-07-05 12:11:18 UTC
non-capturing group should be faster, ie:
static QRegExp cmdEcho("^\\s*(?:g\\+\\+|gcc|icc|ifort)\\s+");
Comment 35 Rob L 2007-07-06 10:44:33 UTC
missed ar from list:
static QRegExp cmdEcho("^\\s*(?:g\\+\\+|gcc|icc|ifort|ld|ar)\\s+"); 
Comment 36 Vadym Krevs 2007-07-14 12:37:14 UTC
Created attachment 21143 [details]
Patch1 to eliminate high CPU usage during build
Comment 37 Vadym Krevs 2007-07-14 12:37:44 UTC
Created attachment 21144 [details]
Patch2 to eliminate high CPU usage during build
Comment 38 Vadym Krevs 2007-07-14 12:42:44 UTC
Unfortunately Rob's patches did not work for me - Kdevelop's CPU usage remained at 100% matching regexps. The only thing that works is going back to the previous versions of parts/outputviews/compileerrorfilter.cpp and parts/outputviews/makeactionfilter.cpp (see attached diffs). From what I can tell, this effectively reverts changes done to address bugs 132601 and 145687. Don't know if it helps, but in my build compile lines all start with either "ccache /usr/bin/g++ ..." or "ccache /usr/bib/gcc ...".
Comment 39 Rob L 2007-07-16 10:58:11 UTC
adding ccache to my regexp may help:

static QRegExp cmdEcho("^\\s*(?:g\\+\\+|gcc|icc|ifort|ld|ar|ccache)\\s+");

However I don't think this is going to be a workable solution for everyone: there are always going to be tools that have been missed.  Maybe it would be better if a project was configured for the build tools that were actually being used?  That way it would have a much smaller list of regexs to match.  Or provide a user configurable list of commands to exclude, eg. 'g++,"ccache /usr/bin/"' that would be used to create the regex?
Comment 40 Andreas Pakulat 2007-07-30 03:01:55 UTC
SVN commit 694087 by apaku:

Remove the dirwatcher, it caused more trouble than its worth.
BUG:146144


 M  +21 -94    customprojectpart.cpp  
 M  +1 -6      customprojectpart.h  


--- branches/KDE/3.5/kdevelop/buildtools/custommakefiles/customprojectpart.cpp #694086:694087
@@ -27,7 +27,6 @@
 
 #include <kaction.h>
 #include <kdebug.h>
-#include <kdirwatch.h>
 #include <kdialogbase.h>
 #include <keditlistbox.h>
 #include <kdevgenericfactory.h>
@@ -67,7 +66,6 @@
 CustomProjectPart::CustomProjectPart( QObject *parent, const char *name, const QStringList & )
         : KDevBuildTool( &data, parent, name ? name : "CustomProjectPart" )
         , m_lastCompilationFailed( false ), m_recursive(false), m_first_recursive(false)
-        , dirwatch(new KDirWatch(this))
 {
     setInstance( CustomProjectFactory::instance() );
     setXMLFile( "kdevcustomproject.rc" );
@@ -80,10 +78,6 @@
     action->setToolTip( i18n( "Re-Populate Project" ) );
     action->setWhatsThis( i18n( "<b>Re-Populate Project</b><p>Re-Populate's the project, searches through the project directory and adds all files that match one of the wildcards set in the custom manager options to the project filelist." ) );
 
-    action = new KAction( i18n( "Add New Files To Project" ), 0, this, SLOT( addNewFilesToProject() ), actionCollection(), "addnewfiles_project" );
-    action->setToolTip( i18n( "Add New Files To Project" ) );
-    action->setWhatsThis( i18n( "<b>Add New Files To Project</b><p>Adds files created outside of kdevelop to the project." ) );
-
     action = new KAction( i18n( "&Build Project" ), "make_kdevelop", Key_F8,
                           this, SLOT( slotBuild() ),
                           actionCollection(), "build_build" );
@@ -187,8 +181,6 @@
              this, SLOT( slotCommandFinished( const QString& ) ) );
     connect( makeFrontend(), SIGNAL( commandFailed( const QString& ) ),
              this, SLOT( slotCommandFailed( const QString& ) ) );
-    connect( dirwatch, SIGNAL( dirty( const QString&) ),
-             this, SLOT( slotDirDirty( const QString& ) ) );
  }
 
 
@@ -237,6 +229,9 @@
 
     QString popupstr = fcontext->urls().first().fileName();
 
+    if( popupstr == QString::null )
+        popupstr = ".";
+
     if ( fcontext->urls().size() == 1 && URLUtil::isDirectory( fcontext->urls().first() ) && !isInBlacklist( fcontext->urls().first().path() ) )
     {
         popup->insertSeparator();
@@ -250,6 +245,7 @@
                                        "like the <i>New Class</i> wizard." ) );
     }
 
+    kdDebug(9025) << "context urls: " << fcontext->urls() << endl;
     if( fcontext->urls().size() == 1 && ( isProjectFileType( fcontext->urls().first().path() ) || URLUtil::isDirectory( fcontext->urls().first() ) ) )
     {
         popup->insertSeparator();
@@ -283,8 +279,10 @@
 
     for ( KURL::List::ConstIterator it = urls.begin(); it != urls.end(); ++it )
     {
+        kdDebug(9025) << "Checking URL: " << *it << endl;
         QString canPath( URLUtil::canonicalPath(( *it ).path() ) );
         QString relPath = relativeToProject( canPath );
+	kdDebug(9025) << "relpath: " << relPath << "|canpath: " << canPath << endl;
         if( isInBlacklist( relPath ) )
             continue;
         if ((( *it ).isLocalFile() && isProjectFileType(( *it ).fileName() ) ) )
@@ -393,14 +391,9 @@
 
 void CustomProjectPart::openProject( const QString &dirName, const QString &projectName )
 {
-    dirwatch->blockSignals( true );
-    dirwatch->stopScan();
-
     m_projectDirectory = dirName;
     m_projectName = projectName;
 
-    m_autoAddFiles.clear();
-
     QDomDocument &dom = *projectDom();
     // Set the default directory radio to "executable"
     if ( DomUtil::readEntry( dom, "/kdevcustomproject/run/directoryradio" ) == "" )
@@ -451,11 +444,9 @@
 
         if( newfiles.count() > 0 )
         {
-            m_autoAddFiles = newfiles;
-            addNewFilesToProject();
+            addNewFilesToProject(newfiles);
         }
 
-        addDirWatches( dirName );
     }
     else
     {
@@ -480,8 +471,6 @@
         envs.appendChild( el );
     }
     KDevProject::openProject( dirName, projectName );
-    dirwatch->startScan();
-    dirwatch->blockSignals( false );
 }
 
 void CustomProjectPart::findNewFiles( const QString& dir, QStringList& filelist ) const
@@ -511,19 +500,7 @@
     }
 }
 
-void CustomProjectPart::addDirWatches( const QString& absPath )
-{
-    dirwatch->addDir( absPath );
-    QStringList entries = QDir( absPath ).entryList( QDir::Dirs );
-    entries.remove(".");
-    entries.remove("..");
-    for( QStringList::const_iterator it = entries.begin(); it != entries.end(); ++it )
-    {
-        addDirWatches( absPath + "/" + *it );
-    }
-}
 
-
 void CustomProjectPart::populateProject()
 {
 
@@ -546,17 +523,14 @@
 
     findNewFiles( projectDirectory(), newlist );
 
-    m_autoAddFiles = newlist;
     QApplication::restoreOverrideCursor();
-    addNewFilesToProject();
-    addDirWatches( m_projectDirectory );
+    addNewFilesToProject(newlist);
 }
 
 
 void CustomProjectPart::closeProject()
 {
     saveProject();
-    m_autoAddFiles.clear();
 }
 
 void CustomProjectPart::saveProject()
@@ -691,11 +665,14 @@
         if ( isInBlacklist( *it ) )
             continue;
         QString relpath;
+	kdDebug(9025) << "Checking path: " << *it << endl;
         if( QDir::isRelativePath( *it ) )
         {
+            kdDebug(9025) << *it << " is relative" << endl;
             relpath = *it;
         }else
         {
+            kdDebug(9025) << *it << " is not relative" << endl;
             relpath = relativeToProject( *it );
         }
 
@@ -704,13 +681,18 @@
 
         if ( QFileInfo( projectDirectory() + "/" + relpath ).isDir() && ( m_recursive || m_first_recursive ) )
         {
+            kdDebug(9025) << "is a dir and " << m_recursive << "|" << m_first_recursive << endl;
             m_first_recursive = false;
             QStringList fileentries = QDir( projectDirectory() + "/" + relpath ).entryList( filetypes().join(";") );
             QStringList dirs = QDir( projectDirectory() + "/" + relpath ).entryList(QDir::Dirs);
             QStringList subentries = fileentries+dirs;
             for ( QStringList::iterator subit = subentries.begin(); subit != subentries.end(); ++subit )
+            {
                 if ( *subit != "." && *subit != ".." )
                     *subit = relpath + "/" + ( *subit );
+                if( (*subit).startsWith("/") )
+                    *subit = (*subit).mid(1,(*subit).length());
+	    }
             addFiles( subentries );
             addedFiles << relpath;
             m_sourceFiles.append( relpath );
@@ -1474,58 +1456,6 @@
     return false;
 }
 
-void CustomProjectPart::slotDirDirty( const QString& dir )
-{
-
-    QStringList remove;
-    QString reldir = relativeToProject( dir );
-    if( !reldir.isEmpty() )
-        reldir += "/";
-
-    QStringList blacklist = this->blacklist();
-    for( QStringList::const_iterator it = m_sourceFiles.begin(); it !=
-            m_sourceFiles.end(); ++it)
-    {
-        if( (*it).startsWith(reldir) )
-        {
-            QString lastpart = (*it).mid(reldir.length());
-            if( !QFileInfo(dir+"/"+lastpart).exists() )
-            {
-                remove << (*it);
-            }
-        }
-    }
-    for( QStringList::const_iterator it = remove.begin(); it != remove.end(); ++it )
-    {
-        if( isInBlacklist(*it) )
-        {
-            blacklist.remove(*it);
-        }
-    }
-    updateBlacklist( blacklist );
-    removeFiles(remove);
-
-    QStringList fileentries = QDir( dir ).entryList( filetypes().join(";") );
-    QStringList dirs = QDir( dir ).entryList(QDir::Dirs);
-    QStringList files = fileentries+dirs;
-    files.remove(".");
-    files.remove("..");
-    kdDebug(9025) << "Got dirty signal from " << dir << endl;
-    for( QStringList::const_iterator it = files.begin(); it != files.end(); ++it )
-    {
-        if( m_sourceFiles.find( reldir+*it ) == m_sourceFiles.end() && ( isProjectFileType( *it ) || QFileInfo( dir+"/"+*it ).isDir() ) && !isInBlacklist( reldir+*it ) )
-        {
-            kdDebug(9025) << "Adding " << reldir+*it << " to autolist" << endl;
-            m_autoAddFiles.append( reldir+*it );
-            if( QFileInfo( dir+"/"+*it ).isDir() )
-            {
-                addDirWatches( dir+"/"+*it );
-            }
-        }else
-            kdDebug(9025) << "Not Adding" << reldir+*it << endl;
-    }
-}
-
 void CustomProjectPart::switchBlacklistEntry( const QString& path)
 {
     QStringList blacklist = this->blacklist();
@@ -1546,8 +1476,11 @@
 QString CustomProjectPart::relativeToProject( const QString& abspath ) const
 {
     QString path = abspath.mid( projectDirectory().length()+1 );
+    kdDebug(9025) <<"abspath: " << "|project dir: " << projectDirectory() << "|path: " << path << endl;
     if( path.endsWith("/") )
         path = path.mid( 0, path.length()-1 );
+    if( path.startsWith("/") )
+        path = path.mid( 1, path.length() );
     return path;
 }
 
@@ -1581,10 +1514,10 @@
     return DomUtil::readListEntry( *projectDom(), "kdevcustomproject/blacklist", "path");
 }
 
-void CustomProjectPart::addNewFilesToProject()
+void CustomProjectPart::addNewFilesToProject( const QStringList& filelist )
 {
     QStringList addfiles;
-    for( QStringList::const_iterator it = m_autoAddFiles.begin(); it != m_autoAddFiles.end(); ++it )
+    for( QStringList::const_iterator it = filelist.begin(); it != filelist.end(); ++it )
     {
         if( m_sourceFiles.find( *it ) == m_sourceFiles.end() && ( isProjectFileType( *it ) || QFileInfo( projectDirectory()+"/"+*it ).isDir() ) && !isInBlacklist( *it ) )
         {
@@ -1598,7 +1531,6 @@
     AddFilesDialog *dlg = new AddFilesDialog( addfiles, mainWindow()->main() );
     if( dlg->exec() == KDialog::Accepted )
     {
-        m_autoAddFiles.clear();
         m_first_recursive = false;
         m_recursive = false;
         QStringList blacklist = this->blacklist();
@@ -1622,11 +1554,6 @@
             excludelist.remove(*it);
         }
         blacklist += excludelist;
-        for( QStringList::const_iterator it = excludelist.begin(); it != excludelist.end(); ++it )
-        {
-            if( QFileInfo( projectDirectory()+"/"+*it ).isDir() )
-                dirwatch->removeDir(projectDirectory()+"/"+*it);
-        }
         updateBlacklist( blacklist );
         addFiles( dlg->includedPaths() );
     }
--- branches/KDE/3.5/kdevelop/buildtools/custommakefiles/customprojectpart.h #694086:694087
@@ -26,7 +26,6 @@
 class QPopupMenu;
 class QStringList;
 class KDialogBase;
-class KDirWatch;
 class CustomProjectWidget;
 class Context;
 class KSelectAction;
@@ -66,14 +65,13 @@
 
 private slots:
     void populateProject();
-    void slotDirDirty( const QString& dir );
     void projectConfigWidget( KDialogBase *dlg );
     void contextMenu( QPopupMenu *popup, const Context *context );
     void slotAddToProject();
     void slotRemoveFromProject();
     void slotAddToProjectRecursive();
     void slotRemoveFromProjectRecursive();
-    void addNewFilesToProject();
+    void addNewFilesToProject( const QStringList& );
     void slotChangeBlacklist();
     void slotChooseActiveDirectory();
     void slotBuild();
@@ -102,7 +100,6 @@
     void cleanFileList();
     void setFiletypes( const QStringList& );
     QString relativeToProject( const QString& ) const;
-    void addDirWatches( const QString& );
     void findNewFiles( const QString& dir, QStringList& list) const;
 
     QStringList filetypes() const;
@@ -119,7 +116,6 @@
     QString m_projectName;
     QString m_filelistDir;
     QStringList m_sourceFiles;
-    QStringList m_autoAddFiles;
     QPopupMenu *m_targetMenu;
     QPopupMenu *m_targetObjectFilesMenu;
     QPopupMenu *m_targetOtherFilesMenu;
@@ -140,7 +136,6 @@
     QMap<QString, QString> m_makefileVars;
     bool m_recursive;
     bool m_first_recursive;
-    KDirWatch* dirwatch;
 };
 
 #endif
Comment 41 Rob L 2007-10-03 13:41:54 UTC
I am using svn://anonsvn.kde.org/home/kde/branches/KDE/3.5/kdevelop, Revision: 714381 (ie. after the above change), but without my workaround in CompileErrorFilter::processLine() (comment #33) kdevelop still uses too much cpu during program linkage.
Comment 42 Andreas Pakulat 2007-10-03 13:56:49 UTC
Seems like Kris didn't know there's a bugreport about this. Kris recently change the regexps that are used for compiler-output-matching to be much faster (he suffered from the same problem). That was done in branches/kdevelop/3.5 though, which will be merged back to KDE/3.5/kdevelop one last time at this weekend (after that kdevelop 3.5 development will happen only in that branch).
Comment 43 Rob L 2007-10-03 15:21:29 UTC
kdevelop3-3.4.90-24.1 3.5.x binary snapshot is fine (I hope it was made from branches/kdevelop/3.5!)
Comment 44 Andreas Pakulat 2007-10-03 16:36:37 UTC
Yes that snapshot includes the fixes. Good that this is now really fixed :)