Created attachment 155239 [details] The slowly-parsed test file from the LLVM repository SUMMARY I have tried working on the LLVM project in KDevelop. Sometimes KDevelop crashes. Very often KDevelop freezes on exit. I traced the freeze to indefinite parsing of the file https://github.com/llvm/llvm-project/blob/main/clang/test/CodeGenCXX/mangle-ms-back-references-pr13207.cpp (attached). The line `using A8 = Fooob<A7, A7, A7, A7, A7, A7, A7, A7, A7, A7>;` , along with the previous similar nested aliases for A7, A6, ... A0, gives KDevelop too much work. If it gets to the next two aliases, A9 and A10, the parsing can probably take days or months. A likely culprit is Visitor::createClassTemplateSpecializationType(). There are several nested GDB frames at the line kdevelop/plugins/clang/duchain/builder.cpp:766: `currentType = makeType(argumentType, typeDecl);`. Either this implementation should be greatly optimized or KDevelop should detect such long parsing and give it up. The LLVM test's express purpose is to test Clang's performance, according to a comment below: // This should take milliseconds, not minutes. void f(A9 a) {} I guess KDevelop's attempt to parse the code more comprehensively backfires in this case. When I removed this test file from the llvm repository, KDevelop's parser froze again with a very similar backtrace. Unfortunately, the value of the tStr variable (clang_getTypeSpelling) cannot be found in the llvm repository, and so the other slowly-parsed file is not so easy to locate. The code must be generated somehow, maybe from variadic templates. A workaround that I hope will prevent background parser freezes is to exclude the "/clang/test/*/" pattern in the Project Filter. STEPS TO REPRODUCE 1. Download the attached file. 2. Open it in KDevelop. 3. Try to exit KDevelop. OBSERVED RESULT 2. After the second step KDevelop's Background Parser keeps a CPU core busy indefinitely. 3. After the third step KDevelop does not exit, but keeps parsing the vexing file. EXPECTED RESULT KDevelop finishes parsing the file in a reasonably short time (less than a minute perhaps). BACKTRACE #0 KDevHash::operator<< <unsigned int>() at kdevelop/kdevplatform/language/util/kdevhash.h:48 #1 KDevelop::AbstractType::hash() at kdevelop/kdevplatform/language/duchain/types/abstracttype.cpp:130 #2 KDevelop::IntegralType::hash() at kdevelop/kdevplatform/language/duchain/types/integraltype.cpp:145 #3 KDevelop::AbstractTypeDataRequest::hash() at kdevelop/kdevplatform/language/duchain/types/typerepository.cpp:36 #4 KDevelop::Bucket<KDevelop::AbstractTypeData, KDevelop::AbstractTypeDataRequest, true, 0u>::findIndex() at kdevelop/kdevplatform/serialization/itemrepository.h:266 #5 KDevelop::ItemRepository<KDevelop::AbstractTypeData, KDevelop::AbstractTypeDataRequest, true, QRecursiveMutex, 0u, 1048576u>::index(KDevelop::AbstractTypeDataRequest const&)::{lambda(unsigned short, KDevelop::Bucket<KDevelop::AbstractTypeData, KDevelop::AbstractTypeDataRequest, true, 0u> const*)#1}::operator()(unsigned short, KDevelop::Bucket<KDevelop::AbstractTypeData, KDevelop::AbstractTypeDataRequest, true, 0u> const*) const() at kdevelop/kdevplatform/serialization/itemrepository.h:1,153 #6 KDevelop::ItemRepository<KDevelop::AbstractTypeData, KDevelop::AbstractTypeDataRequest, true, QRecursiveMutex, 0u, 1048576u>::walkBucketChain<KDevelop::ItemRepository<KDevelop::AbstractTypeData, KDevelop::AbstractTypeDataRequest, true, QRecursiveMutex, 0u, 1048576u>::index(KDevelop::AbstractTypeDataRequest const&)::{lambda(unsigned short, KDevelop::Bucket<KDevelop::AbstractTypeData, KDevelop::AbstractTypeDataRequest, true, 0u> const*)#1}>(unsigned int, KDevelop::ItemRepository<KDevelop::AbstractTypeData, KDevelop::AbstractTypeDataRequest, true, QRecursiveMutex, 0u, 1048576u>::index(KDevelop::AbstractTypeDataRequest const&)::{lambda(unsigned short, KDevelop::Bucket<KDevelop::AbstractTypeData, KDevelop::AbstractTypeDataRequest, true, 0u> const*)#1} const&) const() at kdevelop/kdevplatform/serialization/itemrepository.h:1,957 #7 KDevelop::ItemRepository<KDevelop::AbstractTypeData, KDevelop::AbstractTypeDataRequest, true, QRecursiveMutex, 0u, 1048576u>::index() at kdevelop/kdevplatform/serialization/itemrepository.h:1,150 #8 operator()() at kdevelop/kdevplatform/language/duchain/types/typerepository.cpp:120 #9 KDevelop::LockedItemRepository::write<KDevelop::AbstractType, KDevelop::TypeRepository::indexForType(const KDevelop::AbstractType::Ptr&)::<lambda(KDevelop::TypeItemRepository&)> >(struct {...} &&)() at kdevelop/kdevplatform/serialization/itemrepository.h:2,312 #10 KDevelop::TypeRepository::indexForType() at kdevelop/kdevplatform/language/duchain/types/typerepository.cpp:118 #11 KDevelop::IndexedType::IndexedType() at kdevelop/kdevplatform/language/duchain/types/indexedtype.cpp:14 #12 KDevelop::AbstractType::indexed() at kdevelop/kdevplatform/language/duchain/types/abstracttype.cpp:117 #13 (anonymous namespace)::Visitor::createClassTemplateSpecializationType() at kdevelop/plugins/clang/duchain/builder.cpp:770 #14 (anonymous namespace)::Visitor::createType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:645 #15 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #16 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,444 #17 (anonymous namespace)::Visitor::makeAbsType() at kdevelop/plugins/clang/duchain/builder.cpp:337 #18 (anonymous namespace)::Visitor::createType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:619 #19 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #20 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,439 #21 (anonymous namespace)::Visitor::createClassTemplateSpecializationType() at kdevelop/plugins/clang/duchain/builder.cpp:766 #22 (anonymous namespace)::Visitor::createType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:645 #23 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #24 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,444 #25 (anonymous namespace)::Visitor::makeAbsType() at kdevelop/plugins/clang/duchain/builder.cpp:337 #26 (anonymous namespace)::Visitor::createType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:619 #27 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #28 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,439 #29 (anonymous namespace)::Visitor::createClassTemplateSpecializationType() at kdevelop/plugins/clang/duchain/builder.cpp:766 #30 (anonymous namespace)::Visitor::createType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:645 #31 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #32 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,444 #33 (anonymous namespace)::Visitor::makeAbsType() at kdevelop/plugins/clang/duchain/builder.cpp:337 #34 (anonymous namespace)::Visitor::createType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:619 #35 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #36 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,439 #37 (anonymous namespace)::Visitor::createClassTemplateSpecializationType() at kdevelop/plugins/clang/duchain/builder.cpp:766 #38 (anonymous namespace)::Visitor::createType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:645 #39 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #40 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,444 #41 (anonymous namespace)::Visitor::makeAbsType() at kdevelop/plugins/clang/duchain/builder.cpp:337 #42 (anonymous namespace)::Visitor::createType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:619 #43 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #44 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,439 #45 (anonymous namespace)::Visitor::createClassTemplateSpecializationType() at kdevelop/plugins/clang/duchain/builder.cpp:766 #46 (anonymous namespace)::Visitor::createType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:645 #47 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #48 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,444 #49 (anonymous namespace)::Visitor::makeAbsType() at kdevelop/plugins/clang/duchain/builder.cpp:337 #50 (anonymous namespace)::Visitor::createType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:619 #51 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #52 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,439 #53 (anonymous namespace)::Visitor::createClassTemplateSpecializationType() at kdevelop/plugins/clang/duchain/builder.cpp:766 #54 (anonymous namespace)::Visitor::createType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:645 #55 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #56 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,444 #57 (anonymous namespace)::Visitor::makeAbsType() at kdevelop/plugins/clang/duchain/builder.cpp:337 #58 (anonymous namespace)::Visitor::createType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:619 #59 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #60 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,439 #61 (anonymous namespace)::Visitor::createClassTemplateSpecializationType() at kdevelop/plugins/clang/duchain/builder.cpp:766 #62 (anonymous namespace)::Visitor::createType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:645 #63 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #64 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,444 #65 (anonymous namespace)::Visitor::makeAbsType() at kdevelop/plugins/clang/duchain/builder.cpp:337 #66 (anonymous namespace)::Visitor::createType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:619 #67 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)107>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #68 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,439 #69 (anonymous namespace)::Visitor::createClassTemplateSpecializationType() at kdevelop/plugins/clang/duchain/builder.cpp:766 #70 (anonymous namespace)::Visitor::createType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:645 #71 (anonymous namespace)::Visitor::dispatchType<(CXTypeKind)1>() at kdevelop/plugins/clang/duchain/builder.cpp:369 #72 (anonymous namespace)::Visitor::makeType() at kdevelop/plugins/clang/duchain/builder.cpp:1,444 #73 (anonymous namespace)::Visitor::makeAbsType() at kdevelop/plugins/clang/duchain/builder.cpp:337 #74 (anonymous namespace)::Visitor::createType<(CXCursorKind)36>() at kdevelop/plugins/clang/duchain/builder.cpp:675 #75 (anonymous namespace)::Visitor::createDeclaration<(CXCursorKind)36, KDevelop::Declaration>() at kdevelop/plugins/clang/duchain/builder.cpp:457 #76 (anonymous namespace)::Visitor::buildDeclaration<(CXCursorKind)36, KDevelop::Declaration, false>() at kdevelop/plugins/clang/duchain/builder.cpp:1,291 #77 (anonymous namespace)::Visitor::dispatchCursor<(CXCursorKind)36, (Decision)1, (Decision)1>() at kdevelop/plugins/clang/duchain/builder.cpp:1,007 #78 (anonymous namespace)::Visitor::dispatchCursor<(CXCursorKind)36>() at kdevelop/plugins/clang/duchain/builder.cpp:971 #79 (anonymous namespace)::visitCursor() at kdevelop/plugins/clang/duchain/builder.cpp:1,677 #80 ??() at /usr/lib/libclang.so.13 #81 ??() at /usr/lib/libclang.so.13 #82 ??() at /usr/lib/libclang.so.13 #83 ??() at /usr/lib/libclang.so.13 #84 clang_visitChildren() at /usr/lib/libclang.so.13 #85 (anonymous namespace)::Visitor::Visitor() at kdevelop/plugins/clang/duchain/builder.cpp:1,590 #86 Builder::visit() at kdevelop/plugins/clang/duchain/builder.cpp:1,737 #87 ClangHelpers::buildDUChain(void*, QMultiHash<void*, Import> const&, ParseSession const&, QFlags<KDevelop::TopDUContext::Feature>, QHash<void*, KDevelop::ReferencedTopDUContext>&, QHash<KDevelop::IndexedString, KDevelop::ModificationRevision> const&, KDevelop::IndexedString const&, ClangIndex*, std::function<bool ()> const&)() at kdevelop/plugins/clang/duchain/clanghelpers.cpp:209 #88 ClangParseJob::run() at kdevelop/plugins/clang/clangparsejob.cpp:322 #89 ThreadWeaver::IdDecorator::run(QSharedPointer<ThreadWeaver::JobInterface>, ThreadWeaver::Thread*)() at /usr/lib/libKF5ThreadWeaver.so.5 #90 ThreadWeaver::Executor::run(QSharedPointer<ThreadWeaver::JobInterface> const&, ThreadWeaver::Thread*)() at /usr/lib/libKF5ThreadWeaver.so.5 #91 ThreadWeaver::Job::execute(QSharedPointer<ThreadWeaver::JobInterface> const&, ThreadWeaver::Thread*)() at /usr/lib/libKF5ThreadWeaver.so.5 #92 ThreadWeaver::Thread::run()() at /usr/lib/libKF5ThreadWeaver.so.5 #93 ??() at /usr/lib/libQt5Core.so.5 #94 ??() at /usr/lib/libc.so.6 #95 ??() at /usr/lib/libc.so.6
do you have an idea on how we could possibly detect such cases to prevent them from slowing things down extremely?
(In reply to Milian Wolff from comment #1) > do you have an idea on how we could possibly detect such cases to prevent > them from slowing things down extremely? A separate (dedicated or main) thread could time all running parse jobs and abort those that run longer than a minute. A parse-aborting mechanism already exists: ParseJob::abortRequested(). The atomic abortRequested variable can be passed to the Visitor and checked in slow functions where the checks wouldn't harm performance much, such as createClassTemplateSpecializationType().
(In reply to Igor Kushnir from comment #2) > abort those that run longer than a minute. The maximum allotted time can be configurable of course. On the Configure-KDevelop=>Language Support=>Background Parser tab, I think.
Created attachment 155263 [details] Not yet working and probably unneeded workaround attempt Just tried to implement aborting very-long-running parse jobs. Doesn't work properly. I suspect a separate custom aborting mechanism should be used instead of the existing ParseJob::requestAbort() one to avoid reparsing and RAM exhaustion. That is, a separate atomic variable should be used for Visitor::createClassTemplateSpecializationType(), possibly set up in this function too. However, now I think the code should be optimized instead of working the slowness around. I suspect that the LLVM test exposes gross inefficiency in KDevelop: instead of remembering and reusing already created class template specialization types/aliases, each new occurrence is created from the start. Apparently this inefficiency has been eliminated in Clang years ago. KDevelop should follow suit. Perhaps such an optimization can noticeably improve background parsing performance for normal non-test files too. For example, it could fix Bug 399783.
before we do such hackery, let's first try to optimize the code. I'm looking into that now
https://invent.kde.org/kdevelop/kdevelop/-/merge_requests/424 has a fix for the file here, if there are other's like it that are still slow afterwards please tell me :)
(In reply to Igor Kushnir from comment #0) > A workaround that I hope will prevent background parser freezes is to exclude > the "/clang/test/*/" pattern in the Project Filter. This workaround does work: with the test dir pattern exluded the LLVM project has been parsed successfully and KDevelop exited within seconds.