Application: kdevelop (4.2.60) KDE Platform Version: 4.6.5 (4.6.5) Qt Version: 4.7.4 Operating System: Linux 2.6.40.6-0.fc15.i686.PAE i686 Distribution: "Fedora release 15 (Lovelock)" -- Information about the crash: - What I was doing when the application crashed: nothing. I let the kdevelop parse my project (I set it to use 2 threads) - Unusual behavior I noticed: nothing special - Custom settings of the application: these are the last lines in the terminal where I ran the kdevelop kdevelop(15161)/kdevelop (cpp support) IncludePathComputer::computeBackground: Include-path was missing in list returned by build-manager, adding it now. file was: KUrl("file:///sandbox/vladimir/projectCyclades/build/configurations/unit_test_lib/a_qtr/qt_resources.qrc.cpp") missing path: "/usr/include/boost/" kdevelop(15161)/kdevelop (cpp support) IncludePathComputer::computeBackground: Include-path was missing in list returned by build-manager, adding it now. file was: KUrl("file:///sandbox/vladimir/projectCyclades/build/configurations/unit_test_lib/a_qtr/qt_resources.qrc.cpp") missing path: "/usr/include/SDL/" kdevelop(15161)/kdevelop (cpp support) IncludePathComputer::computeBackground: Include-path was missing in list returned by build-manager, adding it now. file was: KUrl("file:///sandbox/vladimir/projectCyclades/build/configurations/unit_test_lib/a_qtr/qt_resources.qrc.cpp") missing path: "/usr/include/phonon/" KCrash: Application 'kdevelop' crashing... KCrash: Attempting to start /usr/libexec/kde4/drkonqi from kdeinit sock_file=/home/vladimir/.kde/socket-juniper/kdeinit4__0 Warning: connect() failed: : No such file or directory KCrash: Attempting to start /usr/libexec/kde4/drkonqi directly QSocketNotifier: Invalid socket 39 and type 'Read', disabling... QSocketNotifier: Invalid socket 9 and type 'Read', disabling... QSocketNotifier: Invalid socket 12 and type 'Read', disabling... kdevelop: Fatal IO error: client killed FunctionDeclarationData::m_defaultParameters There were items left on destruction: 345 ClassFunctionDeclarationData::m_defaultParameters There were items left on destruction: 673 DUContextData::m_uses There were items left on destruction: 1652 ClassDeclarationData::baseClasses There were items left on destruction: 396 SpecialTemplateDeclarationData::m_specializations There were items left on destruction: 1384 The crash can be reproduced every time. -- Backtrace: Application: KDevelop (kdevelop), signal: Segmentation fault [Current thread is 1 (Thread 0xb78bb780 (LWP 15161))] Thread 10 (Thread 0xae0f8b70 (LWP 15162)): #0 0x001cd424 in __kernel_vsyscall () #1 0x450034f4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x41220f7f in QWaitCondition::wait(QMutex*, unsigned long) () from /usr/lib/libQtCore.so.4 #3 0x002c211a in KDevelop::DUChainPrivate::CleanupThread::run (this=0xa2b29d8) at /sandbox/vladimir/temp/kdevelop/src/kdevplatform/language/duchain/duchain.cpp:282 #4 0x41220ae4 in ?? () from /usr/lib/libQtCore.so.4 #5 0x44fffa2e in start_thread () from /lib/libpthread.so.0 #6 0x44f1234e in clone () from /lib/libc.so.6 Thread 9 (Thread 0xaccffb70 (LWP 15165)): #0 0x45037c40 in clock_gettime () from /lib/librt.so.1 #1 0x412786a6 in ?? () from /usr/lib/libQtCore.so.4 #2 0x4134ba77 in ?? () from /usr/lib/libQtCore.so.4 #3 0x4134bddb in ?? () from /usr/lib/libQtCore.so.4 #4 0x4134a663 in ?? () from /usr/lib/libQtCore.so.4 #5 0x4134a6fd in ?? () from /usr/lib/libQtCore.so.4 #6 0x45097b5c in g_main_context_prepare () from /lib/libglib-2.0.so.0 #7 0x450989d8 in ?? () from /lib/libglib-2.0.so.0 #8 0x4509906f in g_main_context_iteration () from /lib/libglib-2.0.so.0 #9 0x4134b167 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #10 0x4131be0e in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #11 0x4131c061 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #12 0x4121dbac in QThread::exec() () from /usr/lib/libQtCore.so.4 #13 0x003c4e16 in KDevelop::CompletionWorkerThread::run (this=0xa494688) at /sandbox/vladimir/temp/kdevelop/src/kdevplatform/language/codecompletion/codecompletionmodel.cpp:84 #14 0x41220ae4 in ?? () from /usr/lib/libQtCore.so.4 #15 0x44fffa2e in start_thread () from /lib/libpthread.so.0 #16 0x44f1234e in clone () from /lib/libc.so.6 Thread 8 (Thread 0xac4feb70 (LWP 15166)): #0 0x45037c40 in clock_gettime () from /lib/librt.so.1 #1 0x412786a6 in ?? () from /usr/lib/libQtCore.so.4 #2 0x4134ba77 in ?? () from /usr/lib/libQtCore.so.4 #3 0x4134bddb in ?? () from /usr/lib/libQtCore.so.4 #4 0x4134a663 in ?? () from /usr/lib/libQtCore.so.4 #5 0x4134a6fd in ?? () from /usr/lib/libQtCore.so.4 #6 0x45097b5c in g_main_context_prepare () from /lib/libglib-2.0.so.0 #7 0x450989d8 in ?? () from /lib/libglib-2.0.so.0 #8 0x4509906f in g_main_context_iteration () from /lib/libglib-2.0.so.0 #9 0x4134b167 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #10 0x4131be0e in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #11 0x4131c061 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #12 0x4121dbac in QThread::exec() () from /usr/lib/libQtCore.so.4 #13 0x003c4e16 in KDevelop::CompletionWorkerThread::run (this=0xa494dc8) at /sandbox/vladimir/temp/kdevelop/src/kdevplatform/language/codecompletion/codecompletionmodel.cpp:84 #14 0x41220ae4 in ?? () from /usr/lib/libQtCore.so.4 #15 0x44fffa2e in start_thread () from /lib/libpthread.so.0 #16 0x44f1234e in clone () from /lib/libc.so.6 Thread 7 (Thread 0xa86f0b70 (LWP 15171)): #0 0x001cd424 in __kernel_vsyscall () #1 0x4500314c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x43c78ed1 in ?? () from /usr/lib/libQtScript.so.4 #3 0x43c78f10 in ?? () from /usr/lib/libQtScript.so.4 #4 0x44fffa2e in start_thread () from /lib/libpthread.so.0 #5 0x44f1234e in clone () from /lib/libc.so.6 Thread 6 (Thread 0xa6713b70 (LWP 15187)): #0 0x45098bf0 in ?? () from /lib/libglib-2.0.so.0 #1 0x4509906f in g_main_context_iteration () from /lib/libglib-2.0.so.0 #2 0x4134b167 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #3 0x4131be0e in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #4 0x4131c061 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #5 0x4121dbac in QThread::exec() () from /usr/lib/libQtCore.so.4 #6 0x412fc40e in ?? () from /usr/lib/libQtCore.so.4 #7 0x41220ae4 in ?? () from /usr/lib/libQtCore.so.4 #8 0x44fffa2e in start_thread () from /lib/libpthread.so.0 #9 0x44f1234e in clone () from /lib/libc.so.6 Thread 5 (Thread 0xa5ec5b70 (LWP 15208)): #0 0x001cd424 in __kernel_vsyscall () #1 0x4500314c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x49b7a7e0 in ?? () from /usr/lib/libQtWebKit.so.4 #3 0x49b7a8c0 in ?? () from /usr/lib/libQtWebKit.so.4 #4 0x44fffa2e in start_thread () from /lib/libpthread.so.0 #5 0x44f1234e in clone () from /lib/libc.so.6 Thread 4 (Thread 0xa4bc5b70 (LWP 15209)): #0 0x45037c40 in clock_gettime () from /lib/librt.so.1 #1 0x412786a6 in ?? () from /usr/lib/libQtCore.so.4 #2 0x4134ba77 in ?? () from /usr/lib/libQtCore.so.4 #3 0x4134bddb in ?? () from /usr/lib/libQtCore.so.4 #4 0x4134a663 in ?? () from /usr/lib/libQtCore.so.4 #5 0x45097b5c in g_main_context_prepare () from /lib/libglib-2.0.so.0 #6 0x450989d8 in ?? () from /lib/libglib-2.0.so.0 #7 0x4509906f in g_main_context_iteration () from /lib/libglib-2.0.so.0 #8 0x4134b167 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #9 0x4131be0e in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #10 0x4131c061 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #11 0x4121dbac in QThread::exec() () from /usr/lib/libQtCore.so.4 #12 0x4121dc9c in QThread::run() () from /usr/lib/libQtCore.so.4 #13 0x41220ae4 in ?? () from /usr/lib/libQtCore.so.4 #14 0x44fffa2e in start_thread () from /lib/libpthread.so.0 #15 0x44f1234e in clone () from /lib/libc.so.6 Thread 3 (Thread 0xa3fb4b70 (LWP 15312)): [KCrash Handler] #5 QVector<Token>::realloc (this=0x984d4550, asize=17279119, aalloc=25918680) at /usr/include/QtCore/qvector.h:491 #6 0x05401241 in QVector<Token>::append (this=0x984d4550, t=...) at /usr/include/QtCore/qvector.h:549 #7 0x05400bda in Lexer::tokenize (this=0xa3fb3cf8, _session=0xe902968) at /sandbox/vladimir/temp/kdevelop/src/kdevelop/languages/cpp/parser/lexer.cpp:313 #8 0x05412973 in Parser::parse (this=0xa3fb3cf0, _session=0xe902968) at /sandbox/vladimir/temp/kdevelop/src/kdevelop/languages/cpp/parser/parser.cpp:202 #9 0x01453eb3 in CPPInternalParseJob::run (this=<optimized out>) at /sandbox/vladimir/temp/kdevelop/src/kdevelop/languages/cpp/cppparsejob.cpp:560 #10 0x01456eb0 in CPPInternalParseJob::run (this=0xe366af0) at /sandbox/vladimir/temp/kdevelop/src/kdevelop/languages/cpp/cppparsejob.cpp:423 #11 0x41f52b8e in ?? () from /usr/lib/libthreadweaver.so.4 #12 0x41f52cf5 in ThreadWeaver::Job::execute(ThreadWeaver::Thread*) () from /usr/lib/libthreadweaver.so.4 #13 0x41f542d6 in ?? () from /usr/lib/libthreadweaver.so.4 #14 0x41f521c2 in ?? () from /usr/lib/libthreadweaver.so.4 #15 0x41f522bb in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4 #16 0x41220ae4 in ?? () from /usr/lib/libQtCore.so.4 #17 0x44fffa2e in start_thread () from /lib/libpthread.so.0 #18 0x44f1234e in clone () from /lib/libc.so.6 Thread 2 (Thread 0xa35ffb70 (LWP 15313)): #0 0x001cd424 in __kernel_vsyscall () #1 0x4500314c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x41220ff8 in QWaitCondition::wait(QMutex*, unsigned long) () from /usr/lib/libQtCore.so.4 #3 0x41f50a77 in ?? () from /usr/lib/libthreadweaver.so.4 #4 0x41f5363c in ?? () from /usr/lib/libthreadweaver.so.4 #5 0x41f5069b in ?? () from /usr/lib/libthreadweaver.so.4 #6 0x41f5374c in ?? () from /usr/lib/libthreadweaver.so.4 #7 0x41f505f4 in ?? () from /usr/lib/libthreadweaver.so.4 #8 0x41f53768 in ?? () from /usr/lib/libthreadweaver.so.4 #9 0x41f505f4 in ?? () from /usr/lib/libthreadweaver.so.4 #10 0x41f521f4 in ?? () from /usr/lib/libthreadweaver.so.4 #11 0x41f522bb in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4 #12 0x41220ae4 in ?? () from /usr/lib/libQtCore.so.4 #13 0x44fffa2e in start_thread () from /lib/libpthread.so.0 #14 0x44f1234e in clone () from /lib/libc.so.6 Thread 1 (Thread 0xb78bb780 (LWP 15161)): #0 0x001cd424 in __kernel_vsyscall () #1 0x4500314c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x41220ff8 in QWaitCondition::wait(QMutex*, unsigned long) () from /usr/lib/libQtCore.so.4 #3 0x41220795 in QThread::wait(unsigned long) () from /usr/lib/libQtCore.so.4 #4 0x412eeaa6 in QFileSystemWatcher::~QFileSystemWatcher() () from /usr/lib/libQtCore.so.4 #5 0x412eeb73 in QFileSystemWatcher::~QFileSystemWatcher() () from /usr/lib/libQtCore.so.4 #6 0x4132ff82 in QObjectPrivate::deleteChildren() () from /usr/lib/libQtCore.so.4 #7 0x41335899 in QObject::~QObject() () from /usr/lib/libQtCore.so.4 #8 0x41a4453c in ?? () from /usr/lib/libsolid.so.4 #9 0x41a44573 in ?? () from /usr/lib/libsolid.so.4 #10 0x41a443d3 in ?? () from /usr/lib/libsolid.so.4 #11 0x419c77fa in ?? () from /usr/lib/libsolid.so.4 #12 0x44e698d1 in __run_exit_handlers () from /lib/libc.so.6 #13 0x44e6995d in exit () from /lib/libc.so.6 #14 0x421280d9 in ?? () from /usr/lib/libQtGui.so.4 #15 0x4304060a in KApplication::xioErrhandler(_XDisplay*) () from /usr/lib/libkdeui.so.5 #16 0x43040645 in ?? () from /usr/lib/libkdeui.so.5 #17 0x451f6075 in _XIOError () from /usr/lib/libX11.so.6 #18 0x451f37cf in _XEventsQueued () from /usr/lib/libX11.so.6 #19 0x451e40b8 in XEventsQueued () from /usr/lib/libX11.so.6 #20 0x42163a66 in ?? () from /usr/lib/libQtGui.so.4 #21 0x4509824c in g_main_context_check () from /lib/libglib-2.0.so.0 #22 0x45098c90 in ?? () from /lib/libglib-2.0.so.0 #23 0x4509906f in g_main_context_iteration () from /lib/libglib-2.0.so.0 #24 0x4134b108 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #25 0x42163c1b in ?? () from /usr/lib/libQtGui.so.4 #26 0x4131be0e in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #27 0x4131c061 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4 #28 0x413207bb in QCoreApplication::exec() () from /usr/lib/libQtCore.so.4 #29 0x420acf45 in QApplication::exec() () from /usr/lib/libQtGui.so.4 #30 0x080512d7 in main (argc=<optimized out>, argv=0xbf931634) at /sandbox/vladimir/temp/kdevelop/src/kdevelop/app/main.cpp:479 Reported using DrKonqi
*** Bug 294741 has been marked as a duplicate of this bug. ***
what project is this, can we investigate it on our own? it's clearly an out-of-memory issue - how much do you have available?
(In reply to comment #2) > what project is this, can we investigate it on our own? Unfortunately, it is an commercial product, and I can not provide source files. The size is about 250 kLOC, and it uses boost, glut, opengl, qt, sdl-audio, and several other 3rd party libraries. I got about 60-100 include paths. Could that cause an issue? > it's clearly an > out-of-memory issue - how much do you have available? I have 8 Gb ram, plus some virtual memory. Is there a way to provide the kdevelop's configuration? Maybe I set something wrong (I am using 4 threads in the parser).
I think I have identified the source of the problem. We are using Qt in our project. And we create a shared library containing all resources for all qt applications (we got only images in the resource files, but lots of them. They are mostly buttons icons, but there are some big image). Anyway, the kdevelop is crashing nearly every time it needs to parse the cpp file created from the qrc file (qt resource file).
heh that might be the problem indeed. is there maybe a way you could provide a qrc-generated file with some free images that exhibits this behavior? then I could try to look at the problem and maybe find a way to improve the memory consumption. bye
(In reply to comment #5) > heh that might be the problem indeed. is there maybe a way you could provide a > qrc-generated file with some free images that exhibits this behavior? then I > could try to look at the problem and maybe find a way to improve the memory > consumption. I tried attaching the file, but I couldn't because it is 50 Mb large (even compressed it is big), and bugzilla allows 1 mega files. Do you know of an alternatives?
you can try one of the gazillions of free upload sites, like zippyshare, dropbox, ...
Here is the problematic file : http://www10.zippyshare.com/v/48949300/file.html
Git commit 79edb4115f9950a98dd395a05a45f391e19bd0e3 by Milian Wolff. Committed on 01/03/2012 at 01:16. Pushed by mwolff into branch 'master'. optimize: reduce memory consumption of Token class by 50% on 64Bit By removing the ParseSession pointer from it, we get rid of 8 bytes and furthermore reduce the alignment size to 4. This way, we now only require 12 bytes per token compared to 24 bytes previously. This also allows us to define the Token class as a primitive type, potentially speeding up the TokenStream even further. The "cost" is a changed API, to get the string representation of a token, one must now ask the TokenStream. In practice this is very rarely a real pita, as before one often did stream->token(i)->symbol() now you just do stream->symbol(i) Furthermore I've consolidated the tons of custom "AST* node to QString" functions into one central ParseSession::stringForNode. Finally, I've replaced some costly token.symbol() == IndexedChar("somechar") with the much faster token.kind == Token_xyz comparisons. All in all, this should a) make our code faster and b) let it use much less memory while at it. For the big resource file in the bug below, the difference of 50% in the Token class results in ~250MB less memory consumption M +4 -4 languages/cpp/cppduchain/cppeditorintegrator.cpp M +2 -2 languages/cpp/cppduchain/declarationbuilder.cpp M +1 -7 languages/cpp/cppduchain/dumpchain.cpp M +7 -11 languages/cpp/cppduchain/expressionvisitor.cpp M +6 -23 languages/cpp/cppduchain/name_visitor.cpp M +3 -2 languages/cpp/parser/codegenerator.cpp M +3 -3 languages/cpp/parser/dumptree.cpp M +35 -38 languages/cpp/parser/lexer.cpp M +53 -21 languages/cpp/parser/lexer.h M +7 -22 languages/cpp/parser/name_compiler.cpp M +6 -11 languages/cpp/parser/parser.cpp M +15 -0 languages/cpp/parser/parsesession.cpp M +10 -0 languages/cpp/parser/parsesession.h M +15 -5 languages/cpp/parser/tests/test_generator.cpp M +2 -11 languages/cpp/parser/tests/test_parser.cpp M +0 -4 languages/cpp/parser/tests/test_parser.h M +2 -2 languages/cpp/parser/tests/test_parser_cpp2011.cpp M +3 -2 languages/cpp/tests/cpp-parser.cpp http://commits.kde.org/kdevelop/79edb4115f9950a98dd395a05a45f391e19bd0e3
Created attachment 69199 [details] massif file of duchainify run on the supplied test file parsing your files shows that the clear culprit is the AST design... your file has roughly 10mio initializer clauses, see e.g. http://www.nongnu.org/hcb/#dcl.init Our InitializerClauseAST has a sizeof(32) on 64 bit machines (8-byte aligned, two pointers, four integers) and every number is in turn a PrimaryExpressionAST with sizeof(64). So we end up at ~10mio * 96bytes = 960 MB. Massif (attached) says it's even more at ~1.1GB... so yeah we need to figure out how to optimize the memory consumption of our ast :-/
btw we could also think about never parsing files larger than X bytes, something like 10MB should do already. Furthermore an additional safeguard in our parser to "stop" after X tokens might be good, what do you think?
(In reply to comment #11) > btw we could also think about never parsing files larger than X bytes, > something like 10MB should do already. > 1 MB files are already way too big :) > Furthermore an additional safeguard in our parser to "stop" after X tokens > might be good, what do you think? If you ask me - sounds good.
the largest file I can find is /usr/include/ppl.h at 2.6MB so a limit of ~5mb could maybe work I'll see - David, do you have any objections to this?
Git commit c0225a06eae31b6ccd46d8b4924faf02caf29dcf by Milian Wolff. Committed on 01/03/2012 at 15:55. Pushed by mwolff into branch '1.3'. skip files that are larger than 5MiB during parsing our memory consumption is quite considerable for large files, in the bug report below e.g. it is roughly 30x that of the parsed files size. to prevent asserts / OOM exceptions we just skip files that are that large. most often enough, these files are uninmportant generated files anyways and don't need to be parsed to get useful developer features. Examples are e.g. Qt resource files. note: not translated to get it in for 1.3. Since this error is not very user visible / important, I think it's ok this way. I'll properly mark it as translatable for 1.4. M +14 -1 language/backgroundparser/parsejob.cpp http://commits.kde.org/kdevplatform/c0225a06eae31b6ccd46d8b4924faf02caf29dcf
Git commit 7310ea61043390d5e31f1d07d5bb35f0b06ad4e7 by Milian Wolff. Committed on 24/11/2012 at 21:16. Pushed by mwolff into branch 'master'. Use union in PrimaryExpressionAST to reduce memory footprint. On 64Bit machines, we had sizeof(PrimaryExpressionAST) == 64. By using a union and enum this can be decreased to 40. This can result in dramatically less memory consumption, esp. for large qrc files e.g. In the case of the file attached to bug 291248 the memory consumption dropped by 200MB. While the code handling is now a bit changed, I still think this change is worth it. While at it, I've also refactored ExpressionVisitor::visitPrimaryExpression. This gives us cleaner code and should also be faster since we can use the token type instead of doing string comparisons to find numbers. M +63 -78 languages/cpp/cppduchain/expressionvisitor.cpp M +2 -0 languages/cpp/cppduchain/expressionvisitor.h M +3 -1 languages/cpp/cppduchain/usedecoratorvisitor.cpp M +15 -5 languages/cpp/parser/ast.h M +12 -9 languages/cpp/parser/codegenerator.cpp M +17 -4 languages/cpp/parser/default_visitor.cpp M +5 -0 languages/cpp/parser/parser.cpp http://commits.kde.org/kdevelop/7310ea61043390d5e31f1d07d5bb35f0b06ad4e7