still a serious issue in master branch, I get this deadlock at least once per week.

(gdb) thread apply all bt

Thread 10 (Thread 0x7fefcfd7f710 (LWP 1747)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fefe3ae10eb in QWaitCondition::wait(QMutex*, unsigned long) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#2  0x00007fefe3adb5f4 in QReadWriteLock::lockForWrite() () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#3  0x00007fefe02ff65f in KDevelop::DUChainPrivate::doMoreCleanup (this=0x1403bd0, retries=1, needLockRepository=true)
    at /home/milian/projects/kde4/kdevplatform/language/duchain/duchain.cpp:720
#4  0x00007fefe02fcaa4 in KDevelop::DUChainPrivate::CleanupThread::run (this=0x1401170)
    at /home/milian/projects/kde4/kdevplatform/language/duchain/duchain.cpp:292
#5  0x00007fefe3ae0515 in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#6  0x00007fefe383e9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#7  0x00007fefe1e9070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8  0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fefce40d710 (LWP 1749)):
#0  0x00007fefe1e88ff3 in select () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fefe3baf30d in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#2  0x00007fefe3ae0515 in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#3  0x00007fefe383e9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007fefe1e9070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fefc0332710 (LWP 1782)):
#0  0x00007fefe1e83f93 in *__GI___poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007fefdc6924a9 in ?? () from /lib/libglib-2.0.so.0
#2  0x00007fefdc6928fc in g_main_context_iteration () from /lib/libglib-2.0.so.0
#3  0x00007fefe3bff213 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#4  0x00007fefe3bd1db2 in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#5  0x00007fefe3bd219c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#6  0x00007fefe3add51d in QThread::exec() () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#7  0x00007fefe0427f03 in KDevelop::CompletionWorkerThread::run (this=0x1f01a10)
    at /home/milian/projects/kde4/kdevplatform/language/codecompletion/codecompletionmodel.cpp:80
#8  0x00007fefe3ae0515 in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#9  0x00007fefe383e9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#10 0x00007fefe1e9070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#11 0x0000000000000000 in ?? ()

---Type <return> to continue, or q <return> to quit---
Thread 7 (Thread 0x7fefbfb31710 (LWP 1790)):
#0  0x00007fefe1e83f93 in *__GI___poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007fefdc6924a9 in ?? () from /lib/libglib-2.0.so.0
#2  0x00007fefdc6928fc in g_main_context_iteration () from /lib/libglib-2.0.so.0
#3  0x00007fefe3bff213 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#4  0x00007fefe3bd1db2 in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#5  0x00007fefe3bd219c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#6  0x00007fefe3add51d in QThread::exec() () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#7  0x00007fefe0427f03 in KDevelop::CompletionWorkerThread::run (this=0x22ae710)
    at /home/milian/projects/kde4/kdevplatform/language/codecompletion/codecompletionmodel.cpp:80
#8  0x00007fefe3ae0515 in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#9  0x00007fefe383e9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#10 0x00007fefe1e9070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#11 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fefb7d9d710 (LWP 1791)):
#0  0x00007fefe1e83f93 in *__GI___poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007fefdc6924a9 in ?? () from /lib/libglib-2.0.so.0
#2  0x00007fefdc6928fc in g_main_context_iteration () from /lib/libglib-2.0.so.0
#3  0x00007fefe3bff213 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#4  0x00007fefe3bd1db2 in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#5  0x00007fefe3bd219c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#6  0x00007fefe3add51d in QThread::exec() () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#7  0x00007fefe0427f03 in KDevelop::CompletionWorkerThread::run (this=0x22ae560)
    at /home/milian/projects/kde4/kdevplatform/language/codecompletion/codecompletionmodel.cpp:80
#8  0x00007fefe3ae0515 in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#9  0x00007fefe383e9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#10 0x00007fefe1e9070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#11 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fefb759c710 (LWP 1804)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fefe100bdb4 in ?? () from /home/milian/projects/compiled/qt/lib/libQtScript.so.4
#2  0x00007fefe100bde9 in ?? () from /home/milian/projects/compiled/qt/lib/libQtScript.so.4
#3  0x00007fefe383e9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007fefe1e9070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
---Type <return> to continue, or q <return> to quit---
#5  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fefb67ff710 (LWP 1825)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fefe3ae10eb in QWaitCondition::wait(QMutex*, unsigned long) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#2  0x00007fefe5b36d84 in ?? () from /usr/lib/libthreadweaver.so.4
#3  0x00007fefe5b39f8b in ?? () from /usr/lib/libthreadweaver.so.4
#4  0x00007fefe5b384ff in ?? () from /usr/lib/libthreadweaver.so.4
#5  0x00007fefe5b38a58 in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4
#6  0x00007fefe3ae0515 in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#7  0x00007fefe383e9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#8  0x00007fefe1e9070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fefb5ffe710 (LWP 1826)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:211
#1  0x00007fefe3ae1052 in QWaitCondition::wait(QMutex*, unsigned long) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#2  0x00007fefe6e72002 in KDevelop::ForegroundLock::relock (this=0x7fefb5ffa71f) at /home/milian/projects/kde4/kdevplatform/interfaces/foregroundlock.cpp:113
#3  0x00007fefe6e71b84 in ForegroundLock (this=0x7fefb5ffa71f, lock=true) at /home/milian/projects/kde4/kdevplatform/interfaces/foregroundlock.cpp:71
#4  0x00007fefe02dbfe0 in KDevelop::ParseJob::readContents (this=0x7fefca5b3e10)
    at /home/milian/projects/kde4/kdevplatform/language/backgroundparser/parsejob.cpp:268
#5  0x00007fefbc6a1f0b in PreprocessJob::readContents (this=0x7fefac04ba20) at /home/milian/projects/kde4/kdevelop/languages/cpp/preprocessjob.cpp:665
#6  0x00007fefbc69e85c in PreprocessJob::run (this=0x7fefac04ba20) at /home/milian/projects/kde4/kdevelop/languages/cpp/preprocessjob.cpp:231
#7  0x00007fefbc6918cf in CPPParseJob::parseForeground (this=0x7fefca5b3e10) at /home/milian/projects/kde4/kdevelop/languages/cpp/cppparsejob.cpp:173
#8  0x00007fefbc6a12e7 in PreprocessJob::sourceNeeded (this=0x7fefaae86660, _fileName="projectexplorer/projectexplorerconstants.h", 
    type=rpp::Preprocessor::IncludeGlobal, sourceLine=51, skipCurrentPath=false) at /home/milian/projects/kde4/kdevelop/languages/cpp/preprocessjob.cpp:593
#9  0x00007fefbc43e748 in rpp::pp::handle_include (this=0x7fefb5ffc8e0, skip_current_path=false, input=..., output=...)
    at /home/milian/projects/kde4/kdevelop/languages/cpp/parser/rpp/pp-engine.cpp:209
#10 0x00007fefbc43dea7 in rpp::pp::handle_directive (this=0x7fefb5ffc8e0, directive=100740, input=..., output=...)
    at /home/milian/projects/kde4/kdevelop/languages/cpp/parser/rpp/pp-engine.cpp:131
#11 0x00007fefbc43ea4b in rpp::pp::operator() (this=0x7fefb5ffc8e0, input=..., output=...)
    at /home/milian/projects/kde4/kdevelop/languages/cpp/parser/rpp/pp-engine.cpp:262
#12 0x00007fefbc43da01 in rpp::pp::processFileInternal (this=0x7fefb5ffc8e0, 
    fileName="/home/milian/projects/kde4/qtci/src/plugins/debugger/debuggerrunner.cpp", 
    fileContents="/", '*' <repeats 74 times>, "\n**\n** This file is part of Qt Creator\n**\n** Copyright (c) 2010 Nokia Corporation and/or its subsidiary(-ies).\n**\n** Contact: Nokia Corporation (qt-info@nokia.com)\n**\n** Commercial Usage\n**"... = {...}, result=...)
    at /home/milian/projects/kde4/kdevelop/languages/cpp/parser/rpp/pp-engine.cpp:98
#13 0x00007fefbc43d8b2 in rpp::pp::processFile (this=0x7fefb5ffc8e0, fileName="/home/milian/projects/kde4/qtci/src/plugins/debugger/debuggerrunner.cpp", 
---Type <return> to continue, or q <return> to quit---
    data="/", '*' <repeats 74 times>, "\n**\n** This file is part of Qt Creator\n**\n** Copyright (c) 2010 Nokia Corporation and/or its subsidiary(-ies).\n**\n** Contact: Nokia Corporation (qt-info@nokia.com)\n**\n** Commercial Usage\n**"... = {...})
    at /home/milian/projects/kde4/kdevelop/languages/cpp/parser/rpp/pp-engine.cpp:85
#14 0x00007fefbc69ec10 in PreprocessJob::run (this=0x7fefaae86660) at /home/milian/projects/kde4/kdevelop/languages/cpp/preprocessjob.cpp:259
#15 0x00007fefe5b391e1 in ?? () from /usr/lib/libthreadweaver.so.4
#16 0x00007fefe5b394de in ThreadWeaver::Job::execute(ThreadWeaver::Thread*) () from /usr/lib/libthreadweaver.so.4
#17 0x00007fefe5b3a8e3 in ?? () from /usr/lib/libthreadweaver.so.4
#18 0x00007fefe5b384cf in ?? () from /usr/lib/libthreadweaver.so.4
#19 0x00007fefe5b38a58 in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4
#20 0x00007fefe3ae0515 in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#21 0x00007fefe383e9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#22 0x00007fefe1e9070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#23 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7fefb57fd710 (LWP 1827)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fefe3ae10eb in QWaitCondition::wait(QMutex*, unsigned long) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#2  0x00007fefe5b36d84 in ?? () from /usr/lib/libthreadweaver.so.4
#3  0x00007fefe5b39f8b in ?? () from /usr/lib/libthreadweaver.so.4
#4  0x00007fefe5b39fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#5  0x00007fefe5b39fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#6  0x00007fefe5b39fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#7  0x00007fefe5b39fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#8  0x00007fefe5b39fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#9  0x00007fefe5b39fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#10 0x00007fefe5b384ff in ?? () from /usr/lib/libthreadweaver.so.4
#11 0x00007fefe5b38a58 in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4
#12 0x00007fefe3ae0515 in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#13 0x00007fefe383e9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#14 0x00007fefe1e9070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#15 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fefe7737760 (LWP 1550)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fefe3adf7c3 in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#2  0x00007fefe3adb105 in QMutex::lock() () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#3  0x00007fefe6e719cc in lockForegroundMutexInternal () at /home/milian/projects/kde4/kdevplatform/interfaces/foregroundlock.cpp:38
#4  0x00007fefe6e71d87 in KDevelop::ForegroundLock::relock (this=0x7fff5b69192f) at /home/milian/projects/kde4/kdevplatform/interfaces/foregroundlock.cpp:80
---Type <return> to continue, or q <return> to quit---
#5  0x00007fefe6e71b84 in ForegroundLock (this=0x7fff5b69192f, lock=true) at /home/milian/projects/kde4/kdevplatform/interfaces/foregroundlock.cpp:71
#6  0x0000000000405335 in KDevelopApplication::notify (this=0x7fff5b691f40, receiver=0x1255560, event=0x7fefac3b7d20)
    at /home/milian/projects/kde4/kdevelop/app/main.cpp:89
#7  0x00007fefe3bd308c in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#8  0x00007fefe3bd6033 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#9  0x00007fefe3bff6d3 in ?? () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#10 0x00007fefdc68e8c2 in g_main_context_dispatch () from /lib/libglib-2.0.so.0
#11 0x00007fefdc692748 in ?? () from /lib/libglib-2.0.so.0
#12 0x00007fefdc6928fc in g_main_context_iteration () from /lib/libglib-2.0.so.0
#13 0x00007fefe3bff213 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#14 0x00007fefe2debfbe in ?? () from /home/milian/projects/compiled/qt/lib/libQtGui.so.4
#15 0x00007fefe3bd1db2 in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#16 0x00007fefe3bd219c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#17 0x00007fefe3bd626b in QCoreApplication::exec() () from /home/milian/projects/compiled/qt/lib/libQtCore.so.4
#18 0x000000000040b8b8 in main (argc=3, argv=0x7fff5b693bc8) at /home/milian/projects/kde4/kdevelop/app/main.cpp:468
(gdb)

Next time you encounter this, please try to find out which thread holds the foreground mutex, eg. for which of the threads "holderThread == QThread::currentThread()" is true, and also the value of "recursion".

I remember I've encountered a similar problem at some point, where "recursion" and "holderThread" was 0, but since those values are only changed through lockForegroundMutexInternal and unlockForegroundMutexInternal, this seems impossible, so I was stuck..

I had this deadlock once again, and I've got the feeling now that it's a bug somewhere in the QMutex implementation for recursive mutexes.

During debugging, I've seen that QMutex::d->owner is zero, while the mutex cannot be locked. Looking at the implementation though, QMutex::d->owner should never be zero when a recursive mutex is locked, as the locker-thread is stored there.

I might have worked around this bug, by not using a recursive mutex any more in the foreground lock. We'll see.

Lets reopen if it happens again after the patch.

Just got it again with todays git master:

(gdb) print holderThread 
$1 = (class QThread *) 0x0
(gdb) print recursion
$6 = 0
(gdb) print internalMutex.tryLock()
$9 = false
(gdb) print tryLockMutex.tryLock()
$10 = false
(gdb) print waitMutex.tryLock()
$11 = true
(gdb) print finishMutex.tryLock()
$12 = false
(gdb) 


Shouldn't recursion and holderThread be QAtomicInt's? Otherwise we might want to add some debug map or similar that keeps track of which thread has locked the mutex, we could also save the output of kBacktrace() in there, maybe it contains useful things (or is it all done through the eventloop hence impossible to debug?).

Anyway, this should show you that your workaround does not help, can you remove it again?

(gdb) thread apply all bt 

Thread 14 (Thread 0x7f42b87f8710 (LWP 31334)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:211
#1  0x00007f42ccf1e1e9 in QWaitConditionPrivate::wait (this=0x15ea8a0, time=1000) at thread/qwaitcondition_unix.cpp:86
#2  0x00007f42ccf1dfad in QWaitCondition::wait (this=0x15ea548, mutex=0x15ea550, time=1000) at thread/qwaitcondition_unix.cpp:160
#3  0x00007f42c92bdf70 in KDevelop::DUChainPrivate::CleanupThread::run (this=0x15ea530)
    at /home/milian/projects/kde4/kdevplatform/language/duchain/duchain.cpp:283
#4  0x00007f42ccf1caed in QThreadPrivate::start (arg=0x15ea530) at thread/qthread_unix.cpp:315
#5  0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#6  0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7f42b6e83710 (LWP 31335)):
#0  0x00007f42caf42ff3 in select () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f42cd022ead in QProcessManager::run (this=0x7f42cd3f28e0) at io/qprocess_unix.cpp:245
#2  0x00007f42ccf1caed in QThreadPrivate::start (arg=0x7f42cd3f28e0) at thread/qthread_unix.cpp:315
#3  0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f42a8c0b710 (LWP 31360)):
#0  0x00007f42caf3df93 in *__GI___poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007f42c56514a9 in ?? () from /lib/libglib-2.0.so.0
#2  0x00007f42c56518fc in g_main_context_iteration () from /lib/libglib-2.0.so.0
#3  0x00007f42cd090019 in QEventDispatcherGlib::processEvents (this=0x217b1d0, flags=...) at kernel/qeventdispatcher_glib.cpp:422
#4  0x00007f42cd04dac4 in QEventLoop::processEvents (this=0x7f42a8c0ad70, flags=...) at kernel/qeventloop.cpp:149
#5  0x00007f42cd04dc1a in QEventLoop::exec (this=0x7f42a8c0ad70, flags=...) at kernel/qeventloop.cpp:201
#6  0x00007f42ccf19edf in QThread::exec (this=0x2190490) at thread/qthread.cpp:490
#7  0x00007f42c93e9417 in KDevelop::CompletionWorkerThread::run (this=0x2190490)
    at /home/milian/projects/kde4/kdevplatform/language/codecompletion/codecompletionmodel.cpp:80
#8  0x00007f42ccf1caed in QThreadPrivate::start (arg=0x2190490) at thread/qthread_unix.cpp:315
#9  0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#10 0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#11 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7f42a529d710 (LWP 31361)):
#0  0x00007f42caf3df93 in *__GI___poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007f42c56514a9 in ?? () from /lib/libglib-2.0.so.0
---Type <return> to continue, or q <return> to quit---
#2  0x00007f42c56518fc in g_main_context_iteration () from /lib/libglib-2.0.so.0
#3  0x00007f42cd090019 in QEventDispatcherGlib::processEvents (this=0x2192cd0, flags=...) at kernel/qeventdispatcher_glib.cpp:422
#4  0x00007f42cd04dac4 in QEventLoop::processEvents (this=0x7f42a529cd70, flags=...) at kernel/qeventloop.cpp:149
#5  0x00007f42cd04dc1a in QEventLoop::exec (this=0x7f42a529cd70, flags=...) at kernel/qeventloop.cpp:201
#6  0x00007f42ccf19edf in QThread::exec (this=0x217e3b0) at thread/qthread.cpp:490
#7  0x00007f42c93e9417 in KDevelop::CompletionWorkerThread::run (this=0x217e3b0)
    at /home/milian/projects/kde4/kdevplatform/language/codecompletion/codecompletionmodel.cpp:80
#8  0x00007f42ccf1caed in QThreadPrivate::start (arg=0x217e3b0) at thread/qthread_unix.cpp:315
#9  0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#10 0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#11 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7f42a4a9c710 (LWP 31365)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f42ca038baf in QTWTF::TCMalloc_PageHeap::scavengerThread (this=0x7f42ca3d1540) at ../3rdparty/javascriptcore/JavaScriptCore/wtf/FastMalloc.cpp:2359
#2  0x00007f42ca0381ea in QTWTF::TCMalloc_PageHeap::runScavengerThread (context=0x7f42ca3d1540)
    at ../3rdparty/javascriptcore/JavaScriptCore/wtf/FastMalloc.cpp:1464
#3  0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f42a840a710 (LWP 31370)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:211
#1  0x00007f42ccf1e1e9 in QWaitConditionPrivate::wait (this=0xe7e1a0, time=30) at thread/qwaitcondition_unix.cpp:86
#2  0x00007f42ccf1dfad in QWaitCondition::wait (this=0x7f42d0775680, mutex=0x7f42d0775670, time=30) at thread/qwaitcondition_unix.cpp:160
#3  0x00007f42d05550be in KDevelop::ForegroundLock::relock (this=0x7f42a840881f) at /home/milian/projects/kde4/kdevplatform/interfaces/foregroundlock.cpp:126
#4  0x00007f42d0554c40 in ForegroundLock (this=0x7f42a840881f, lock=true) at /home/milian/projects/kde4/kdevplatform/interfaces/foregroundlock.cpp:84
#5  0x00007f42c929d4bc in KDevelop::ParseJob::readContents (this=0x6306080)
    at /home/milian/projects/kde4/kdevplatform/language/backgroundparser/parsejob.cpp:268
#6  0x00007f42a5af21e7 in PreprocessJob::readContents (this=0x6582470) at /home/milian/projects/kde4/kdevelop/languages/cpp/preprocessjob.cpp:665
#7  0x00007f42a5aeeb38 in PreprocessJob::run (this=0x6582470) at /home/milian/projects/kde4/kdevelop/languages/cpp/preprocessjob.cpp:231
#8  0x00007f42cf2161e1 in ?? () from /usr/lib/libthreadweaver.so.4
#9  0x00007f42cf2164de in ThreadWeaver::Job::execute(ThreadWeaver::Thread*) () from /usr/lib/libthreadweaver.so.4
#10 0x00007f42cf2178e3 in ?? () from /usr/lib/libthreadweaver.so.4
#11 0x00007f42cf2154cf in ?? () from /usr/lib/libthreadweaver.so.4
#12 0x00007f42cf215a58 in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4
#13 0x00007f42ccf1caed in QThreadPrivate::start (arg=0x2fc7ea0) at thread/qthread_unix.cpp:315
#14 0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
---Type <return> to continue, or q <return> to quit---
#15 0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#16 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f429fb7f710 (LWP 31371)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f42ccf1e205 in QWaitConditionPrivate::wait (this=0x1438130, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:88
#2  0x00007f42ccf1dfad in QWaitCondition::wait (this=0x1438588, mutex=0x11cde70, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:160
#3  0x00007f42cf213d84 in ?? () from /usr/lib/libthreadweaver.so.4
#4  0x00007f42cf216f8b in ?? () from /usr/lib/libthreadweaver.so.4
#5  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#6  0x00007f42cf2154ff in ?? () from /usr/lib/libthreadweaver.so.4
#7  0x00007f42cf215a58 in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4
#8  0x00007f42ccf1caed in QThreadPrivate::start (arg=0x350f260) at thread/qthread_unix.cpp:315
#9  0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#10 0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#11 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f429f37e710 (LWP 31372)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f42ccf1e205 in QWaitConditionPrivate::wait (this=0x1438130, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:88
#2  0x00007f42ccf1dfad in QWaitCondition::wait (this=0x1438588, mutex=0x11cde70, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:160
#3  0x00007f42cf213d84 in ?? () from /usr/lib/libthreadweaver.so.4
#4  0x00007f42cf216f8b in ?? () from /usr/lib/libthreadweaver.so.4
#5  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#6  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#7  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#8  0x00007f42cf2154ff in ?? () from /usr/lib/libthreadweaver.so.4
#9  0x00007f42cf215a58 in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4
#10 0x00007f42ccf1caed in QThreadPrivate::start (arg=0x7f42a003be30) at thread/qthread_unix.cpp:315
#11 0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#12 0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#13 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f429eb7d710 (LWP 31374)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f42c44c7439 in WTF::TCMalloc_PageHeap::scavengerThread() () from /home/milian/projects/compiled/qt/lib/libQtWebKit.so.4
#2  0x00007f42c44c6a0a in WTF::TCMalloc_PageHeap::runScavengerThread(void*) () from /home/milian/projects/compiled/qt/lib/libQtWebKit.so.4
#3  0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
---Type <return> to continue, or q <return> to quit---
#4  0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f429b17f710 (LWP 2723)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f42ccf1e205 in QWaitConditionPrivate::wait (this=0x7a48740, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:88
#2  0x00007f42ccf1dfad in QWaitCondition::wait (this=0x850e5b8, mutex=0x6b7d470, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:160
#3  0x00007f42cf213d84 in ?? () from /usr/lib/libthreadweaver.so.4
#4  0x00007f42cf216f8b in ?? () from /usr/lib/libthreadweaver.so.4
#5  0x00007f42cf2154ff in ?? () from /usr/lib/libthreadweaver.so.4
#6  0x00007f42cf215a58 in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4
#7  0x00007f42ccf1caed in QThreadPrivate::start (arg=0x87f1920) at thread/qthread_unix.cpp:315
#8  0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#9  0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#10 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f429a4fb710 (LWP 2724)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f42ccf1e205 in QWaitConditionPrivate::wait (this=0x7a48740, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:88
#2  0x00007f42ccf1dfad in QWaitCondition::wait (this=0x850e5b8, mutex=0x6b7d470, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:160
#3  0x00007f42cf213d84 in ?? () from /usr/lib/libthreadweaver.so.4
#4  0x00007f42cf216f8b in ?? () from /usr/lib/libthreadweaver.so.4
#5  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#6  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#7  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#8  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#9  0x00007f42cf2154ff in ?? () from /usr/lib/libthreadweaver.so.4
#10 0x00007f42cf215a58 in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4
#11 0x00007f42ccf1caed in QThreadPrivate::start (arg=0x6938900) at thread/qthread_unix.cpp:315
#12 0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#13 0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f4299cfa710 (LWP 3169)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f42ccf1e205 in QWaitConditionPrivate::wait (this=0x7a48740, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:88
#2  0x00007f42ccf1dfad in QWaitCondition::wait (this=0x850e5b8, mutex=0x6b7d470, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:160
#3  0x00007f42cf213d84 in ?? () from /usr/lib/libthreadweaver.so.4
---Type <return> to continue, or q <return> to quit---
#4  0x00007f42cf216f8b in ?? () from /usr/lib/libthreadweaver.so.4
#5  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#6  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#7  0x00007f42cf2154ff in ?? () from /usr/lib/libthreadweaver.so.4
#8  0x00007f42cf215a58 in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4
#9  0x00007f42ccf1caed in QThreadPrivate::start (arg=0x736ffd0) at thread/qthread_unix.cpp:315
#10 0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#11 0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#12 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f42994f9710 (LWP 3170)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f42ccf1e205 in QWaitConditionPrivate::wait (this=0x7a48740, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:88
#2  0x00007f42ccf1dfad in QWaitCondition::wait (this=0x850e5b8, mutex=0x6b7d470, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:160
#3  0x00007f42cf213d84 in ?? () from /usr/lib/libthreadweaver.so.4
#4  0x00007f42cf216f8b in ?? () from /usr/lib/libthreadweaver.so.4
#5  0x00007f42cf216fa4 in ?? () from /usr/lib/libthreadweaver.so.4
#6  0x00007f42cf2154ff in ?? () from /usr/lib/libthreadweaver.so.4
#7  0x00007f42cf215a58 in ThreadWeaver::Thread::run() () from /usr/lib/libthreadweaver.so.4
#8  0x00007f42ccf1caed in QThreadPrivate::start (arg=0x6612080) at thread/qthread_unix.cpp:315
#9  0x00007f42ccc669ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#10 0x00007f42caf4a70d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#11 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f42d0e1c760 (LWP 31333)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f42ccf1c36a in QMutexPrivate::wait (this=0xe7dfa0, timeout=-1) at thread/qmutex_unix.cpp:84
#2  0x00007f42ccf157cd in QMutex::lock (this=0x7f42d0775660) at thread/qmutex.cpp:204
#3  0x00007f42d0554a5e in lockForegroundMutexInternal () at /home/milian/projects/kde4/kdevplatform/interfaces/foregroundlock.cpp:43
#4  0x00007f42d0554e43 in KDevelop::ForegroundLock::relock (this=0x7fff8dfcfc6f) at /home/milian/projects/kde4/kdevplatform/interfaces/foregroundlock.cpp:93
#5  0x00007f42d0554c40 in ForegroundLock (this=0x7fff8dfcfc6f, lock=true) at /home/milian/projects/kde4/kdevplatform/interfaces/foregroundlock.cpp:84
#6  0x0000000000405365 in KDevelopApplication::notify (this=0x7fff8dfd0420, receiver=0x7f42d07756a0, event=0x7f42a1af0990)
    at /home/milian/projects/kde4/kdevelop/app/main.cpp:89
#7  0x00007f42cd050ac4 in QCoreApplication::notifyInternal (this=0x7fff8dfd0420, receiver=0x7f42d07756a0, event=0x7f42a1af0990)
    at kernel/qcoreapplication.cpp:731
#8  0x00007f42ceac3449 in QCoreApplication::sendEvent (receiver=0x7f42d07756a0, event=0x7f42a1af0990)
    at ../../../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:215
#9  0x00007f42cd051d24 in QCoreApplicationPrivate::sendPostedEvents (receiver=0x0, event_type=0, data=0xe7c910) at kernel/qcoreapplication.cpp:1372
---Type <return> to continue, or q <return> to quit---
#10 0x00007f42cd05193f in QCoreApplication::sendPostedEvents (receiver=0x0, event_type=0) at kernel/qcoreapplication.cpp:1265
#11 0x00007f42cd0907af in QCoreApplication::sendPostedEvents () at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:220
#12 0x00007f42cd08eccd in postEventSourceDispatch (s=0xea3fb0) at kernel/qeventdispatcher_glib.cpp:277
#13 0x00007f42c564d8c2 in g_main_context_dispatch () from /lib/libglib-2.0.so.0
#14 0x00007f42c5651748 in ?? () from /lib/libglib-2.0.so.0
#15 0x00007f42c56518fc in g_main_context_iteration () from /lib/libglib-2.0.so.0
#16 0x00007f42cd090019 in QEventDispatcherGlib::processEvents (this=0xe7c060, flags=...) at kernel/qeventdispatcher_glib.cpp:422
#17 0x00007f42cbf42c40 in QGuiEventDispatcherGlib::processEvents (this=0xe7c060, flags=...) at kernel/qguieventdispatcher_glib.cpp:204
#18 0x00007f42cd04dac4 in QEventLoop::processEvents (this=0x7fff8dfd0130, flags=...) at kernel/qeventloop.cpp:149
#19 0x00007f42cd04dc1a in QEventLoop::exec (this=0x7fff8dfd0130, flags=...) at kernel/qeventloop.cpp:201
#20 0x00007f42cd0511c0 in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1008
#21 0x00007f42cbe50110 in QApplication::exec () at kernel/qapplication.cpp:3736
#22 0x000000000040b8e8 in main (argc=3, argv=0x7fff8dfd20a8) at /home/milian/projects/kde4/kdevelop/app/main.cpp:468
(gdb)

I think the code "with workaround" is better than without, so it can stay. We track the recursion anyway, so the QMutex doesn't need to do it.

Regarding QAtomic: "holderThread" and "recursion" are protected by the internalMutex, so they don't need to be atomic. They can only be written by one thread at a time.

Please look at the code of lockMutexInternal, tryLockMutexInternal, and unlockMutexInternal to find some mistakes, maybe I've missed something.

I've encountered this deadlock again, and did some debugging within QMutex. I've found out that QMutex ends up waiting for a wait-condition, while there is no other thread holding the mutex (d->owner == 0), so there is noone to wake the thread up from the wait-condition.

I've changed the code to use a pthread mutex directly now. If it happens again with the pthread mutex, then it should be a clear sign that the problem is on our side. If it doesn't happen again with the pthread mutex, then it's a QMutex bug.

commit bae38ceb983f7c7b0b66ca475dbfa689ace8313e
branch master
Author: David Nolden <david.nolden.kde@art-master.de>
Date:   Wed Dec 1 17:53:52 2010 +0100

    Use pthread directly for the foreground mutex, potentially trying to workaround a problem in the QMutex implementation which leads to a rare deadlock.
    BUG: 252659

diff --git a/interfaces/CMakeLists.txt b/interfaces/CMakeLists.txt
index 4fd1ec4..d832c85 100644
--- a/interfaces/CMakeLists.txt
+++ b/interfaces/CMakeLists.txt
@@ -49,7 +49,8 @@ target_link_libraries(kdevplatforminterfaces
         ${KDE4_KTEXTEDITOR_LIBS} 
         ${KDE4_THREADWEAVER_LIBRARIES} 
         ${QT_QTDESIGNER_LIBRARY} 
-        ${KDE4_KROSSCORE_LIBS})
+        ${KDE4_KROSSCORE_LIBS}
+        rt)
 target_link_libraries(kdevplatforminterfaces LINK_INTERFACE_LIBRARIES 
         ${KDE4_KPARTS_LIBS} 
         ${KDE4_KTEXTEDITOR_LIBS} 
diff --git a/interfaces/foregroundlock.cpp b/interfaces/foregroundlock.cpp
index ada7db9..4e3a73d 100644
--- a/interfaces/foregroundlock.cpp
+++ b/interfaces/foregroundlock.cpp
@@ -24,8 +24,65 @@
 
 using namespace KDevelop;
 
+#define USE_PTHREAD_MUTEX
+
+#ifdef USE_PTHREAD_MUTEX
+
+#include <pthread.h>
+#include <time.h>
+
+
+class SimplePThreadMutex {
+public:
+    SimplePThreadMutex() {
+        m_mutex = PTHREAD_MUTEX_INITIALIZER;
+        int result = pthread_mutex_init(&m_mutex, 0);
+        Q_ASSERT(result == 0);
+    }
+    ~SimplePThreadMutex() {
+        pthread_mutex_destroy(&m_mutex);
+    }
+    void lock() {
+        int result = pthread_mutex_lock(&m_mutex);
+        Q_ASSERT(result == 0);
+    }
+    void unlock() {
+        int result = pthread_mutex_unlock(&m_mutex);
+        Q_ASSERT(result == 0);
+    }
+    bool tryLock(int interval) {
+        if(interval == 0)
+        {
+            int result = pthread_mutex_trylock(&m_mutex);
+            return result == 0;
+        }else{
+            timespec abs_time;
+            clock_gettime(CLOCK_REALTIME, &abs_time);
+            abs_time.tv_nsec += interval * 1000000;
+            if(abs_time.tv_nsec >= 1000000000)
+            {
+                abs_time.tv_nsec -= 1000000000;
+                abs_time.tv_sec += 1;
+            }
+            
+            int result = pthread_mutex_timedlock(&m_mutex, &abs_time);
+            return result == 0;
+        }
+    }
+    
+private:
+    pthread_mutex_t m_mutex;
+};
+
 namespace {
+
+SimplePThreadMutex internalMutex;
+#else
+
+namespace {
+
 QMutex internalMutex;
+#endif
 QMutex tryLockMutex;
 QMutex waitMutex;
 QMutex finishMutex;

Good luck trying to reproduce something that happens once in a week. You don't know in what exact order the mutex was accessed before the problem appeared, and this might be a very tiny race condition.

In turn, it doesn't make any sense to report a bug which you cannot reproduce, and also, it isn't 100% sure that it's a bug in QMutex. The only way to answer that question is by using a working replacement, and seeing whether the replacement works better.

After having seen the internal code of QMutex, it's really not that unprobable that it's buggy. It's a huge pile of spin-locks, wait-conditions, etc., instead of being a simple wrapper around pthread mutex. Furthermore, the state I've found the internals of QMutex in when debugging the deadlock seemed invalid (no registered holder, only one thread accessing the mutex but waiting on the wait-condition -> impossible). The code I've put around the mutex in KDevelop indicates exactly the same thing. It's just 42 lines of code(lock..., tryLock..., and unlock...), read them finally and convince yourself.

Anyway the primary sense of using the pthread mutex is bisecting the problem between KDevelop and Qt. If  the outcome is that QMutex is buggy, it means we have to report the problem, and/or switch our uses of QMutex in KDevelop to something else.

*** Bug 259513 has been marked as a duplicate of this bug. ***

apparently it's reliably reproduced on mac osx (your workaround isn't used there yet, even though pthread is available there) - the foreground lock should be changed in a way to let cmake detect whether pthread is available and use the workaround in that case.

I've asked Till to test it locally whether your workaround is applicable on Mac OS X as well. Since he says the initial deadlock is 100% reproducible, this should give us a good indicator of the quality of your workaround.

Bye

@Milian: If it's 100% reproducible, then it's probably another issue (KDevelop-internal). QMutex cannot be that broken on mac.

	A	 interfaces/tests/test_foregroundlock.h	 [License: UNKNOWN]


	A	 interfaces/tests/test_foregroundlock.cpp	 [License: UNKNOWN]


	A	 interfaces/tests/CMakeLists.txt	 [License: Trivialfile.]

commit c53cd44796c12304ccf71693e414c6a1c99be54c
Author: Milian Wolff <mail@milianw.de>
Date:   Sun Dec 12 19:58:31 2010 +0100

    add testcase for deadlock in ForegroundLock with QMutex
    
    comment out #define USE_PTHREAD_MUTEX in foregroundlock.cpp , then run the testcase. Locks up reliably for me.
    With the alternative mutex it seems to work fine though.
    
    CCBUG: 252659

diff --git a/interfaces/CMakeLists.txt b/interfaces/CMakeLists.txt
index 599c7fc..b7d9a0e 100644
--- a/interfaces/CMakeLists.txt
+++ b/interfaces/CMakeLists.txt
@@ -1,5 +1,7 @@
 find_package(Threads)
 
+add_subdirectory(tests)
+
 add_definitions( -DKDE_DEFAULT_DEBUG_AREA=9502 )
 set(kdevplatforminterfaces_LIB_SRCS
     foregroundlock.cpp
diff --git a/interfaces/tests/CMakeLists.txt b/interfaces/tests/CMakeLists.txt
new file mode 100644
index 0000000..ba5ebbc
--- /dev/null
+++ b/interfaces/tests/CMakeLists.txt
@@ -0,0 +1,5 @@
+set( EXECUTABLE_OUTPUT_PATH ${CMAKE_CURRENT_BINARY_DIR} )
+
+set(foregroundlock_SRC test_foregroundlock.cpp ../foregroundlock.cpp)
+kde4_add_unit_test(foregroundlocktest ${foregroundlock_SRC})
+target_link_libraries(foregroundlocktest ${KDE4_KDECORE_LIBS} ${QT_QTTEST_LIBRARY})
diff --git a/interfaces/tests/test_foregroundlock.cpp b/interfaces/tests/test_foregroundlock.cpp
new file mode 100644
index 0000000..d47811b
--- /dev/null
+++ b/interfaces/tests/test_foregroundlock.cpp
@@ -0,0 +1,72 @@
+/* This file is part of KDevelop
+    Copyright 2010 Milian Wolff <mail@milianw.de>
+
+   This library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Library General Public
+   License version 2 as published by the Free Software Foundation.
+
+   This library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Library General Public License for more details.
+
+   You should have received a copy of the GNU Library General Public License
+   along with this library; see the file COPYING.LIB.  If not, write to
+   the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
+   Boston, MA 02110-1301, USA.
+*/
+
+#include "test_foregroundlock.h"
+
+#include <QtTest/QtTest>
+
+#include <QDebug>
+#include <QThread>
+
+#include <unistd.h>
+
+#include "../foregroundlock.h"
+
+QTEST_MAIN(KDevelop::TestForegroundLock)
+
+using namespace KDevelop;
+
+//BEGIN Helper Threads
+
+class TryLockThread : public QThread {
+    Q_OBJECT
+public:
+    virtual void run() {
+        ForegroundLock lock(false);
+        for(int i = 0; i < 1000; ++i) {
+            if (lock.tryLock()) {
+                lock.unlock();
+            }
+            usleep(qrand() % 20);
+        }
+    }
+};
+
+void TestForegroundLock::testTryLock()
+{
+    TryLockThread t1;
+    TryLockThread t2;
+    TryLockThread t3;
+
+    ForegroundLock lock(true);
+
+    t1.start();
+    t2.start();
+    t3.start();
+
+    lock.unlock();
+
+    while(t1.isRunning() || t2.isRunning() || t3.isRunning()) {
+        lock.relock();
+        usleep(10);
+        lock.unlock();
+    }
+}
+
+#include "moc_test_foregroundlock.cpp"
+#include "test_foregroundlock.moc"
diff --git a/interfaces/tests/test_foregroundlock.h b/interfaces/tests/test_foregroundlock.h
new file mode 100644
index 0000000..9ce6b17
--- /dev/null
+++ b/interfaces/tests/test_foregroundlock.h
@@ -0,0 +1,34 @@
+/* This file is part of KDevelop
+    Copyright 2010 Milian Wolff <mail@milianw.de>
+
+   This library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Library General Public
+   License version 2 as published by the Free Software Foundation.
+
+   This library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Library General Public License for more details.
+
+   You should have received a copy of the GNU Library General Public License
+   along with this library; see the file COPYING.LIB.  If not, write to
+   the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
+   Boston, MA 02110-1301, USA.
+*/
+
+#ifndef TEST_FOREGROUNDLOCK_H
+#define TEST_FOREGROUNDLOCK_H
+
+#include <QObject>
+
+namespace KDevelop {
+class TestForegroundLock : public QObject {
+    Q_OBJECT
+
+private slots:
+    void testTryLock();
+};
+}
+
+
+#endif // TEST_FOREGROUNDLOCK_H

The test doesn't deadlock here (Core2 Duo 2.6GHZ, 32-bit Linux, Qt 4.6), although I was also affected by the occassional deadlock.

Anyway if you can reproduce it, you should try using a plain QMutex instead of the ForegroundLock, and then you can send this directly as a test-case to Nokia.

Oh sorry, actually I was doing the test with the pthread mutex enabled. :-)

Till Adam also hinted that it's probably QMutex::tryLock at fault in BUG 259513.

Ok, using a simple QMutex instead of ForegroundLock doesn't expose the deadlock. Some of the more complex behavior of ForegroundLock plays a role too it seems, maybe because it changes the timing.

Btw. the test is not 100% realistic, because there is no proper event-loop running on the foreground. Anyway, the stuff shouldn't deadlock anyway, so I gues s it's enough.

*** Bug 259513 has been marked as a duplicate of this bug. ***

Created attachment 54555 [details]
irc log from 14th of december 2010 with till, thiago and others about the foreground lock

some more information from a IRC discussion. It showed a few things:

a) the test case was not minimal and did not use the custom QApplication with the notify overload, I've since reduced it quite much, fixed the issues, and put the code up on git: git@git.kde.org:scratch/mwolff/foregroundlock-test
this is a standalone and minimal version exhibiting the deadlock.

b) it still looks like a Qt internal issue, as described by Till

c) the code for nested eventloops in relock is very bad according to Thiago, see attachement

d) we should in the future spent time to use a proper design, e.g. the memento pattern, to make this foreground lock obsolete. It is a hack according to Thiago, Till and others. People like Roberto Raggi, Christoph Cullman and others also urged us to use this.

e) While being a hack, it is technically "correct" according to Till and should work

f) maybe more that I forgot to mention...

bug reported upstream: http://bugreports.qt.nokia.com/browse/QTBUG-16115

It's a Qt bug after all, and according to Thiago the workaround to use PThread directly where possible is also ok. I'm impressed David, I wouldn't have thought this is really an issue in Qt!

This is going to be fixed for Qt 4.7.2 hopefully. In an unrelated note: QMutex is going to see a considerable speedup on Qt 4.8, hopefully we can get rid of our custom locks in KDevelop then :)

Maybe ForegroundLock is a hack, but it is simple and _works_. We had enough of over-engineered over-complicated design before, and overall such a simple locking-mechanisms makes the application more stable due to its simplicity.

Git commit 3b172ac59ffdcd1e8e534b6ba80cb03036a884e6 by Aleix Pol, on behalf of David Nolden.
Committed on 01/12/2010 at 17:53.
Pushed by apol into branch '1.1wreck'.

Use pthread directly for the foreground mutex, potentially trying to workaround a problem in the QMutex implementation which leads to a rare deadlock.

M  +2    -1    interfaces/CMakeLists.txt
M  +57   -0    interfaces/foregroundlock.cpp

http://commits.kde.org/kdevplatform/3b172ac59ffdcd1e8e534b6ba80cb03036a884e6