Bug 78978 - Runs out of memory when importing generic C++ project
Summary: Runs out of memory when importing generic C++ project
Status: RESOLVED FIXED
Alias: None
Product: kdevelop
Classification: Applications
Component: Language Support: CPP (old) (show other bugs)
Version: unspecified
Platform: OpenSUSE Linux
: NOR normal with 40 votes (vote)
Target Milestone: ---
Assignee: kdevelop-bugs-null
URL:
Keywords:
: 83468 96909 103446 117420 122246 (view as bug list)
Depends on:
Blocks:
 
Reported: 2004-04-03 17:04 UTC by paul.leopardi
Modified: 2013-03-31 00:51 UTC (History)
7 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
sourcefile to reproduce the bug (194.76 KB, text/x-csrc)
2005-11-22 23:40 UTC, David Solbach
Details
Patch (2.16 KB, patch)
2006-04-27 22:15 UTC, Przemyslaw Bruski
Details

Note You need to log in before you can comment on or make changes to this bug.
Description paul.leopardi 2004-04-03 17:04:20 UTC
Version:            (using KDE KDE 3.2.1)
Installed from:    SuSE RPMs
OS:          Linux

I used glucat-0.1.4.tgz as per download via http://glucat.sf.net and unzipped this to a new directory. In KDevelop, I selected Import -> Existing Project, browsed to the directory, entered glucat as the name, and selected OK.
The KDevelop status line changed to Updating...
KDE System Guard confirmed that my 1GB main memory and 2GB sway was exhausted in less than 30s.
I am using Kernel 2.6.4 on SUSE LINUX Professional 9.0 AMD, with an AMD64 3200+ processor.
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 4
model name      : AMD Athlon(tm) 64 Processor 3200+
stepping        : 8
cpu MHz         : 2009.181
cache size      : 1024 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow
bogomips        : 3981.31
TLB size        : 1088 4K pages
clflush size    : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp
Comment 1 paul.leopardi 2004-04-03 17:05:12 UTC
2GB sway -> 2GB swap
Comment 2 Jens Dagerbo 2004-04-03 20:01:48 UTC
Can't reproduce. Using KDevelop CVS HEAD, I created two "Generic C++ Application" projects, I tried both "Custom Makefiles" and "Automake Based". Neither showed any problems and both were imported in seconds without any large increase in memory usage.

The only source package I could find was called "glucat-0.1.4-hash_map.tgz" though. Is this the same package?
Comment 3 Jens Dagerbo 2004-04-03 20:07:05 UTC
Ok, now I found the glucat-0.1.4.tgz package. Still doesn't give me any problems though.
Comment 4 paul.leopardi 2004-04-05 00:37:29 UTC
Jens,
Thanks for looking at this. 

Perhaps it is a problem peculiar to my architecture? I am running SUSE LINUX 
9.0, Kernel 2.6.4, KDE 3.2.1 on an AMD64 3200+ processor. 

Also, I'll try a few different GluCat downloads as projects and will see if I 
can reliably reproduce the problem. My guess is that my peculiar C++ coding 
style has bamboozled KDevelop's parser. Or it could be that KDevelop is 
interacting with a problem outside its control, in KDE libraries or the 
Kernel.
Best regards

On Sunday 04 April 2004 04:07, Jens Dagerbo wrote:
> ------- Ok, now I found the glucat-0.1.4.tgz package. Still doesn't give me
> any problems though.

Comment 5 paul.leopardi 2004-04-05 01:37:47 UTC
Hi,
I tried glucat-0.0.1 and glucat-0.1.0
They also reproduce the problem.

To reproduce:
1. Download and unzip eg. glucat-0.0.1 source directory.
2. Start KDevelop
3. Select Project->Import Existing...
4. Browse to source directory and fill in project name. Set project type to 
"Generic C++ Application (Custom Makefiles)". Click OK.
5. The import apparently succeeds.
6. Quit KDevelop.
7. Start KDevelop again.
8. The status bar shows "Updating..." and all memory fills up.
9. CTRL-ALT-BACKSPACE to kill X11.
10. Log in to KDE again.
11. Start KDevelop again.
12. This time KDevelop successfully opens the project.
13. Opening a previously imported project now fills up all memory.
Best regards

On Monday 05 April 2004 08:37, leopardi@bigpond.net.au wrote:
> ------- Additional Comments From leopardi bigpond net au  2004-04-05 00:37
> Also, I'll try a few different GluCat downloads as projects and will see if
> I can reliably reproduce the problem.

Comment 6 Daniel Franke 2004-04-05 01:54:24 UTC
Nope, can't confirm.
Using kdevelop-3.0.90-CVS (04/04/01), KDE-3.2.1, SuSE 9.0 prof, Intel PIII 1GH, 256 MB RAM
Comment 7 paul.leopardi 2004-04-05 02:25:30 UTC
Hi,
Also tried gaol-0.1.2 from http://sourceforge.net/projects/gaol/
and it reproduces the problem. So it is not a peculiarity of GluCat.

If you deselect "Load last project on startup", the problem appears every odd 
time that you "open recent project". In other words:
1. Import works. Quit KDevelop.
2. Open KDevelop. Open recent project -> runs out of memory. 
CTRL-ALT-BACKSPACE.
3. Open KDevelop. Open recent project -> works.
Best regards

On Monday 05 April 2004 09:37, leopardi@bigpond.net.au wrote:
> ------- Hi,
> I tried glucat-0.0.1 and glucat-0.1.0
> They also reproduce the problem.

Comment 8 Jens Dagerbo 2004-04-05 02:39:08 UTC
This sounds like it could be related to reading the project's PCS file.

Please try removing the file called <projectname>.kdevelop.pcs before you open the project. If this works, close the project/KDevelop, verify that this creates a the pcs again. Remove it again and load up the project again. 

If this works, your problems quite likely comes from reading the pcs file. 

(It's rather meaningless to speculate at this point, but if the algorithm reading from the pcs stream doesn't find the end of the file, I can see it filling up RAM pretty quickly..)

Exactly what KDevelop version are you running, btw?
Comment 9 paul.leopardi 2004-04-06 01:20:44 UTC
Jens, Answers below. Best regards
On Monday 05 April 2004 10:39, Jens Dagerbo wrote:
> ------- This sounds like it could be related to reading the project's PCS
> file.
>
> Please try removing the file called <projectname>.kdevelop.pcs before you
> open the project. If this works, close the project/KDevelop, verify that
> this creates a the pcs again. Remove it again and load up the project
> again.
>
> If this works, your problems quite likely comes from reading the pcs file.

This works, but for eg. gaol, the class browser is then unpopulated.

> Exactly what KDevelop version are you running, btw?

KDevelop 3.0.2 (Using KDE 3.2.1)
From SuSE RPM kdevelop3-3.0.2-8 for AMD64.
Last ChangeLog entries:
* Tue Mar 02 2004 - coolo@suse.de
 
- updating the tar ball again
 
* Mon Mar 01 2004 - coolo@suse.de
 
- update to version 3.0.2

Comment 10 paul.leopardi 2004-04-06 01:25:39 UTC
Jens, I also tried glucat-0.1.0, and your procedure works. Also, the class 
browser works. 
On Monday 05 April 2004 10:39, Jens Dagerbo wrote:
> Please try removing the file called <projectname>.kdevelop.pcs before you
> open the project. If this works, close the project/KDevelop, verify that
> this creates a the pcs again. Remove it again and load up the project
> again.

Comment 11 Jens Dagerbo 2004-04-06 02:55:54 UTC
Ok, the pcs file must be the cause of this. How large is it? My glucat.kdevelop.pcs file is only about 200 kb. Unless your is very large (which in itself would indicate a problem) could you mail it to me and so that I could have a look at it?

Are you running your system in the 32-bit "compatibility mode", btw? I don't know much about the AMD64, but I'm guessing it is related to the problem somehow.
Comment 12 paul.leopardi 2004-04-06 15:44:49 UTC
Jens, I sent the pcs file separately. Answers below. Best regards
On Tuesday 06 April 2004 10:55, Jens Dagerbo wrote:
> ------- Ok, the pcs file must be the cause of this. How large is it? 
For glucat-0.1.0, about 209 KB.

> My 
> glucat.kdevelop.pcs file is only about 200 kb. Unless your is very large
> (which in itself would indicate a problem) could you mail it to me and so
> that I could have a look at it?
Done (separate email)

> Are you running your system in the 32-bit "compatibility mode", btw? I
> don't know much about the AMD64, but I'm guessing it is related to the
> problem somehow.
No. Most SuSE binary RPMs are 64 bit.
leopardi@linfinit:~> which kdevelop
/opt/kde3/bin/kdevelop
leopardi@linfinit:~> file `which kdevelop`
/opt/kde3/bin/kdevelop: ELF 64-bit LSB executable, AMD x86-64, version 1 
(SYSV), for GNU/Linux 2.4.1, dynamically linked (uses shared libs), not 
stripped

Comment 13 Bernhard Übelacker 2004-04-15 20:50:01 UTC
Probably it's because of a too high grade of Macro usage like in file:
http://savannah.nongnu.org/cgi-bin/viewcvs/qemu/qemu/alpha-dis.c?rev=1.2&content-type=text/vnd.viewcvs-markup

The import goes fine until this file is reached. Then the memory is filled in
about 10 seconds and the machine stays unresponsive for about 4 minutes
(until the kernel kills some processes ;-) )

If the preprocessing of this file is turned off then the import finished successfully.
(Is the preprocessing needed at all?)

qemu/alpha-dis.c (1964 lines, 80,5KB)
165623 Tokens with preprocessing
 17853 Tokens without preprocessing
(counted without the "ADD_TOKEN( tk );" in lexer.cpp:488)

Are these numbers not in the normal range?
If the number of tokens for this file is normal
how are chances the Qt build has a bug (QPtrVector)?

SuSE 9.0 with KDE 3.2.0 (QT 3.3.0)
kdevelop-cvs 2004-04-11

Mit freundlichen Grüßen
Bernhard Übelacker


(disables the preprocessing on tokenizing)
--- kdevelop-cvs.original/kdevelop/lib/cppparser/driver.cpp	2003-12-18 20:06:57.000000000 +0100
+++ kdevelop-cvs/kdevelop/lib/cppparser/driver.cpp	2004-04-14 21:13:19.000000000 +0200
@@ -239,6 +239,7 @@
     Lexer lex( this );
     lexer = &lex;
     setupLexer( &lex );
+    lex.setPreprocessorEnabled(false);
 
     lex.setSource( sourceProvider()->contents(fileName) );



(logs also the filename of the file to be parsed)
--- kdevelop-cvs.original/kdevelop/languages/cpp/cppsupportpart.cpp	2004-04-10 20:53:52.000000000 +0200
+++ kdevelop-cvs/kdevelop/languages/cpp/cppsupportpart.cpp	2004-04-14 20:48:14.000000000 +0200
@@ -915,7 +915,7 @@
 					} 
 					else 
 					{
-						kdDebug(9007) << "newly parsing..." << endl;
+						kdDebug(9007) << "newly parsing... " << absFilePath << endl;
 						m_driver->parseFile( absFilePath );
 					}
 			

.
Comment 14 Jens Dagerbo 2004-04-15 23:11:58 UTC
This must be two different problems, if removing the .pcs file avoids the problem in the first case.

I looked at Paul Leopardi's PCS file, but nothing strikes me as obviously wrong with it. 
Comment 15 Nicolas anquetil 2004-04-18 17:25:21 UTC
Hi,

I am having a similar problem.

I use out of the box Mandrake 10.0

Kdevelop: 3.0.1
OS: Linux (i686) release 2.6.3-4mdk
Compiler: gcc version 3.3.2 (Mandrake Linux 10.0 3.3.2-6mdk)

I am trying to play around with gnucash 1.1.8.
I have had several crashes of Kdevelop running out of memory in different situations:
- Trying to Import Existing Project (GNOME C Application in this case)
- After sucessful import of the project, trying to reopen in (either starting up kdevelop when I had left without closing the project, or after kdevelop start-up, reopening manually the project)
- Playing around in the File Groups vie, deselecting and then reselecting the Show Non Project Files option.

I tried to remove gnucash.kdevelop.pcs but it did not help.

Until now, I succeded in re-opening the project by removing everything (project directory, and kdevelop configuration files in ~/.kde/share/apps/config). There could be a simpler way, but I did not find it yet.

nicolas
Comment 16 paul.leopardi 2004-04-22 17:36:43 UTC
Updated to KDE 2.2.2 and KDevelop 3.0.2. Bug still occurs.
Comment 17 Terry Barnaby 2004-06-16 18:09:18 UTC
I have had the same problem I believe.
When trying to import the XServer source tree KDevelop suddenly starts using
memory until the entire VM memory is exhausted (3G in my case).

I untar'ed XFree86 4.4.0
cd xc/programs/Xserver
kdevelop

And then tried to "Import an existing project".
Comment 18 Terry Barnaby 2004-06-16 18:12:22 UTC
Sorry forgot to put in system info:

KDevelop: 3.0.4
Linux: RedHat 9
Kde: 3.1-13
Arch: Dual Xeon SMP
Comment 19 Amilcar do Carmo Lucas 2004-07-19 22:30:57 UTC
I guess it's time to give KDevelop cvs HEAD a try.
Comment 20 James Gregory 2004-09-07 21:15:55 UTC
Normally import existing project works for me, but I have just found a project that doesn't work for me - the source version of Trigger found at the bottom of the page here:

http://www.positro.net/trigger/

All my memory is very rapidly eaten up and my computer freezes if I try to import it as "generic C++ application - custom Makefiles.

Various points:

1. I've just updated to 3.1, so maybe this might not have effected me using 3.0x? With an 800mHz processor I don't feel like compiling an alternate version of kdevelop right now to find out.

2. Importing other projects works fine for me. The main difference I can see between this project and other projects that I have successfully imported is that though I select "generic C++ custom Makefiles", in reality the project is generic C++ custom jamfiles. There are no Makefiles. Maybe it gets confused by jam?

3. KDevelop never get around to putting a pcs file in the project directory before the memory has all gone

4. I'm one of the people that is affected by the "reopen existing project crashes Kdevelop unless pcs file deleted first".

5. I have modified the source of my copy of kdevelop to include the fix (giving an argument of "true" to parseProject()) given in the bug report listing for the above bug.

I'm not sure if my OS etc is on record from when I registered on this database. I am using a Mandrake 9.2/Cooker hybrid. AMD 800mHz. KDevelop installed from source. KDE 3.2.0.
Comment 21 Paul Schneider 2004-11-16 14:29:08 UTC
Same for me on Gentoo x86_64 on EM64T Xeons. The problem is the same (including the trick with the pcs files), but occurs randomly
Comment 22 paragw 2005-02-25 04:20:27 UTC
I can always reproduce this - KDevelop gobbles up all system memory (in my case 768Mb of RAM and 1G of Swap) while importing linux kernel 2.6.10 sources as a custom makefile project. The import never completes - it freezes the machine when KDevelop is a 1898M VIRT and 634M RSS.

I am also on an AMD64 machine - Fedora Core 3 and have compiled recent KDevelop CVS.
Comment 23 paragw 2005-02-25 17:04:11 UTC
I see atleast three people have confirmed the problem. Why not change status from UNCONFIRMED to CONFIRMED?
Comment 24 Amilcar do Carmo Lucas 2005-02-25 17:43:32 UTC
Because It seams to be only reproducible in KDevelop 3.0.x systems.

So... AFAIK it is fixed in the next KDevelop 3.2 release.
BWT KDevelop 3.2 RC1 should be out this weekend.

Unless somebody with 3.2 RC1 reproduces this issue, it will stay unconfirmed.
Comment 25 paragw 2005-02-25 19:47:06 UTC
KDevelop-3.2.0 from Today's CVS (25 Feb 2005) - still eats up all the memory and the OOM killer has to step in to kill it. 

BTW, the way source code parsing is done is very wasteful - Why do we need to parse all 16841 files from Linux kernel source for example at one time? It has to be on demand - we need to parse a file and all it's dependencies only when it is opened. It is so slow and takes up above 90% CPU all of the time. (Inspite of that it doesn't complete but...)
Comment 26 Matt Rogers 2005-03-01 22:48:09 UTC
Re: Comment #25:

The parsing has to be done all at once because that's how the class view gets populated
Comment 27 kim Lux 2005-11-04 07:07:20 UTC
I got the same problem. 

Kdevelop 3.2.3

KDE 3.4.3-1.0

I am trying to load the gcc binutils project to do some development work on retargeting gcc.

Project->import existing project->select directory. 

If one choses an Automake based project then no files get imported.

If on choses a Custom Makefiles project then Kdevelop crashes. 

I reported and fought this same bug in a different version of KDevelop about a year ago.  I solved it by getting the current head from CVS and building from scratch.  Then I updated to FC4 and Kdevelop got updated and now the problem is back. 

What a pain.  Please fix this ASAP.  
Comment 28 kim Lux 2005-11-04 07:17:50 UTC
There is a gal at kdevelop.org (which wasn't up tonight,btw).  I think her name was Anne_Marie.  She hangs out on the kdevelop chat site.  She helped me fix it last time.  We ran kdevelop from source in a debugger and isolated the problem.  As her if she remembers what the problem was and how to fix it. 

BTW: How do I add my name to the email list on this ?  email me when it gets fixed. 
Comment 29 kim Lux 2005-11-04 07:19:24 UTC
There is a gal at kdevelop.org (which wasn't up tonight,btw).  I think her name was Anne_Marie.  She hangs out on the kdevelop chat site.  She helped me fix it last time.  We ran kdevelop from source in a debugger and isolated the problem.  As her if she remembers what the problem was and how to fix it. 

BTW: How do I add my name to the email list on this ?  email me when it gets fixed. 

"Can't reproduce. Using KDevelop CVS HEAD, I created two "Generic C++ Application" projects, I tried both "Custom Makefiles" and "Automake Based". Neither showed any problems and both were imported in seconds without any large increase in memory usage."

That is probably true.  That is how I fixed it last time, by getting CVS head.  What a pain.  Why not get the CVS head code and compare to the current KDE Kdevelop RPMs and find out what isn't included in the later and fix this thing.   Obviously something in the CVS code has fixed this problem but not in the KDE rpms.  
Comment 30 David Solbach 2005-11-22 23:40:39 UTC
Created attachment 13603 [details]
sourcefile to reproduce the bug

In the attachment you find a single source file which causes the bug. Please be
carefull since it will most likely crash your machine. It crashed my Athlon
3200+/1gb ram - system several times until I added enough debug statements in
lexer.cpp to slow the code down. I added a printf statement before each call of
nextToken() in lexxer.cpp. The Bug still appears, but those debug statements
seem to slow the code down enough to allow the kernel to kill that process in
time (at least on my machine)

The attached source file is taken from the mame project [1] (a part of the z80
emulator). Its a pretty complicated sourcefile with a lot of big functions
defined as macros for speed gains. (not that I understand it, but so far its
obvious) maybe that might be a cause for the problems?

From what I saw from the debug statements, I think I can thell, that it is not
a single recursion entry that called all over again because of a faulty
condition. It seems more that the file is simply too complicated to get parsed
by the current parser with a fair amount of memory. 

Steps to reproduce:

0. save all your work in progress
1. start kdevelop
2. click project->import existing project, use standard settings, select the
directory that contains only the attached source file (z80.c),
3. hit okay... pray

[1] http://www.mame.net/
Comment 31 David Solbach 2005-11-22 23:44:38 UTC
*** This bug has been confirmed by popular vote. ***
Comment 32 David Solbach 2005-11-22 23:48:28 UTC
and sorry about my terrible english in the last comment! If I can assist in any way please drop me a mail.
Comment 33 Tim Harper 2006-03-31 18:41:10 UTC
I can also reproduce this problem 100% of the time while trying to import the clucene project into kdevelop.  Its like the C++ parser gets in a recursive loop and allocates exponential amounts.  My kernel won't cap a processes memory usage, so once it blows the memory and the cache, it takes the system down for good.
Comment 34 Przemyslaw Bruski 2006-04-27 22:15:15 UTC
Created attachment 15803 [details]
Patch

I think it's a dupe of 96909. The problem lies in inefficient handling of
macros.
Comment 35 Przemyslaw Bruski 2006-04-27 22:19:01 UTC
I forgot to add that the attachment from comment #34 fixes the problem, if one decides to recompile kdevelop with it...
Comment 36 Jens Dagerbo 2006-05-01 22:29:48 UTC
Thanks A LOT for this patch, Przemyslaw! We've tested it for a few days and it doesn't seem to have any bad consequences, yet solves the RAM explosion for all cases I've been able to reproduce.

Marking as FIXED. :)
Comment 37 Jens Dagerbo 2006-05-01 22:30:56 UTC
*** Bug 83468 has been marked as a duplicate of this bug. ***
Comment 38 Jens Dagerbo 2006-05-01 22:31:13 UTC
*** Bug 96909 has been marked as a duplicate of this bug. ***
Comment 39 Jens Dagerbo 2006-05-01 22:31:25 UTC
*** Bug 103446 has been marked as a duplicate of this bug. ***
Comment 40 Jens Dagerbo 2006-05-01 22:31:39 UTC
*** Bug 117420 has been marked as a duplicate of this bug. ***
Comment 41 Jens Dagerbo 2006-05-01 22:33:05 UTC
*** Bug 122246 has been marked as a duplicate of this bug. ***
Comment 42 Johannes Ballé 2006-10-10 19:12:35 UTC
Sorry, my previous post went to bug #96909 by mistake.

I'm still observing this bug in kdevelop 3.3.4 on KDE 3.5.2. What baffles me is that Przemyslaw's patch applied without problems on that version. The bug was marked fixed in May, 3.3.4 came out in August, shouldn't it be included in that version already?

Anyway, it seems it doesn't fix the problem entirely. Even with the patch in place, kdevelop hogs my memory. The .kdev_ignore workaround does the trick.
Comment 43 Jens Dagerbo 2006-10-10 19:51:22 UTC
>Anyway, it seems it doesn't fix the problem entirely

What does this mean? Can you see an improvement at all, or does it still consume all memory and kill the system (unless interrupted)?
Comment 44 Johannes Ballé 2006-10-10 20:01:32 UTC
Subjectively I'd say there is a slight improvement in terms of time it takes to load the file. But it's not measurable. KDevelop still goes up to about 700 MB vsz, same as without the patch.
Comment 45 Johannes Ballé 2006-10-11 09:42:07 UTC
It doesn't kill the system, but it takes around 30-40 seconds to open a file of ~8000 lines code, and the system is unresponsive in that time because it's busy swapping.
Comment 46 Johannes Ballé 2006-10-11 12:12:21 UTC
Gee, I was mistaken. I had applied the patch with --dry-run, this couldn't work, of course. Apparently, I was working too much yesterday...

Now it does help. But the bug is definitely still there in kdevelop 3.3.4, I don't know if this is the way it should be. Anyway, I'll post a bug for my distribution. Thanks
Comment 47 Frederic Heem 2006-11-15 12:33:13 UTC
The problem still exists with kdevelop 3.3.5 while trying to import wireshark (www.wireshark.org)
Actually, the whole system became unresponsive, impossible to ssh from another pc, Ctrl-Alt-SysRq-B neither. Hardware reset was the only solution
The kernel 2.6.9-42.0.3.EL  was not able to kill kdevelop although it was eating all memory.
IMHO, kdevelop shall not parse C/C++ file by default, for the majority of developper, using a tag file is just enough, is very fast and reliable.

Comment 48 Amilcar do Carmo Lucas 2006-11-16 17:28:36 UTC
Yes it is probably still present in KDevelop 3.3.x
Please test with the new parser of KDevelop 3.4 RC1
Comment 49 Aleix Pol 2013-03-31 00:51:20 UTC
Moving all the bugs from the CPP Parser. It was not well defined the difference between it and C++ Language Support and people kept reporting in both places indistinctively