Bug 219687

This happens also on short text format messages without attachments.

This is probably because of the distribution list expanding code, which uses a ContactSearchJob that uses Nepomuk to do the search.

No idea why that is slow, adding Sebastian to the CC list, maybe he knows.

I'm also seeing this, but it doesn't take half a minute for me, but some seconds.

*** Bug 217367 has been marked as a duplicate of this bug. ***

i see similar behavior,

but the composer window is "greyed out" immediately at click-to-send, and the message *never* sends.  just sits there with 'top' showing,

 6017 bendj   20   0 1582m 1.4g  14m S   87 17.8  14:22.66 nepomukservices
 5863 bendj   20   0  294m  18m 9432 S    7  0.2   1:42.90 nepomukserver
 5891 bendj   20   0  337m  24m  17m S    6  0.3   3:42.65 nepomukservices

@ the hang point, a backtrace of kmail reports,

Program received signal SIGINT, Interrupt.
0x00007ffff5110033 in poll () from /lib64/libc.so.6
(gdb) bt                                           
#0  0x00007ffff5110033 in poll () from /lib64/libc.so.6
#1  0x00007fffedcd758c in ?? () from /usr/lib64/libglib-2.0.so.0
#2  0x00007fffedcd78d0 in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#3  0x00007ffff64b00f3 in QEventDispatcherGlib::processEvents (this=0x610970,         
    flags=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:412
#4  0x00007ffff591973e in ?? () from /usr/lib64/libQtGui.so.4
#5  0x00007ffff6485482 in QEventLoop::processEvents (this=<value optimized out>, flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
)
    at kernel/qeventloop.cpp:149
#6  0x00007ffff648585c in QEventLoop::exec (this=0x7fffffffc1c0, flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
)
    at kernel/qeventloop.cpp:201
#7  0x00007ffff6892f26 in KJob::exec() () from /usr/lib64/libkdecore.so.5
#8  0x00007fffdffb4eff in Kleo::KeyResolver::lookupContactPreferences (this=0x29cb820, address=...)
    at /usr/src/debug/kdepim-4.3.85/kmail/keyresolver.cpp:1741
#9  0x00007fffdffb6b1e in Kleo::KeyResolver::getEncryptionItems (this=<value optimized out>,
    addresses=<value optimized out>) at /usr/src/debug/kdepim-4.3.85/kmail/keyresolver.cpp:935
#10 0x00007fffdffb7007 in Kleo::KeyResolver::setPrimaryRecipients (this=0x29cb820, addresses=...)
    at /usr/src/debug/kdepim-4.3.85/kmail/keyresolver.cpp:923
#11 0x00007fffdffa68eb in MessageComposer::adjustCryptFlags (this=0x29bc430)
    at /usr/src/debug/kdepim-4.3.85/kmail/messagecomposer.cpp:736
#12 0x00007fffdff9b508 in MessageComposer::slotDoNextJob (this=0x29bc430)
---Type <return> to continue, or q <return> to quit---

if at this point i STOP akonadi server, then START akonadi server, I get this Nepomuk error dialog,

 http://img213.imageshack.us/img213/8498/nepomukdialog.png

at which point, even before dismissin the dialog, kmail sends the hung message with no error.

subsequent mail sends -- in the same kontact/kmail session -- are fine.  if i quit/kill everything kontact/kmail, making sure all slave procs are gone, then relaunch kontact, the whole problem repeats itself.

fyi,

kmail --version
Qt: 4.6.1
KDE Development Platform: 4.3.85 (KDE 4.3.85 (KDE 4.4 Beta2)) "release 203"
KMail: 1.13.0

lsb_release -a
LSB Version:    core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch
Distributor ID: SUSE LINUX
Description:    openSUSE 11.2 (x86_64)
Release:        11.2
Codename:       n/a

@ IRC, commenting that in my case, it *never* sends (until Akonadi stop/start), feedback was:

"well then you need to debug why it never stops. it is either because akonadi never gets a response back from nepomuk, or because akonadi doesn't respond to kmail."

Happy to provide that/any/all info, with some specific guidance as to how to get it.

Here's the Akonadi Console debugger-output from initial launch of kontact, through the hung attempt to kmail-send, then stop-start the server, with the resultant immediate/successful send of the hung mail:

0x773eb0 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x773eb0 0 LOGIN "kontact-2104006769" 
0x773eb0 0 OK User logged in 
0x773eb0 1 SEARCH "prefix nco:<http://www.semanticdesktop.org/ontologies/2007/03/22/nco#>SELECT ?person WHERE { ?person nco:hasEmailAddress ?email . ?email nco:emailAddress \"news@recovery.org\"^^<http://www.w3.org/2001/XMLSchema#string> . }" FULLPAYLOAD EXTERNALPAYLOAD (UID REMOTEID COLLECTIONID FLAGS SIZE DATETIME) 
0x773eb0 1 OK SEARCH completed 
0x773eb0 2 LSUB 0 INF () () 
0x773eb0 * 1 0 (NAME "Search" MIMETYPE () REMOTEID "" RESOURCE "akonadi_search_resource" CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT -1 SYNCONDEMAND false LOCALPARTS (ALL)) ) 
0x773eb0 2 OK List completed 
0x773eb0 3 SEARCH "prefix nco:<http://www.semanticdesktop.org/ontologies/2007/03/22/nco#>SELECT ?r WHERE { ?r a nco:PersonContact }" FULLPAYLOAD EXTERNALPAYLOAD (UID REMOTEID COLLECTIONID FLAGS SIZE DATETIME) 
0x773eb0 3 OK SEARCH completed 
0x773eb0 4 SEARCH "prefix nco:<http://www.semanticdesktop.org/ontologies/2007/03/22/nco#>SELECT ?person WHERE { ?person nco:hasEmailAddress ?email . ?email nco:emailAddress \"bendj@mail.local\"^^<http://www.w3.org/2001/XMLSchema#string> . }" FULLPAYLOAD EXTERNALPAYLOAD (UID REMOTEID COLLECTIONID FLAGS SIZE DATETIME) 
0x773eb0 4 OK SEARCH completed 
0x6a4260 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x660420 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x6a4260 0 LOGIN "akonadi_nepomuk_calendar_feeder" 
0x6a4260 0 OK User logged in 
0x6a4260 1 LSUB 0 INF (MIMETYPE (application/x-vnd.akonadi.calendar.event application/x-vnd.akonadi.calendar.todo application/x-vnd.akonadi.calendar.journal application/x-vnd.akonadi.calendar.freebusy)) () 
0x6a4260 1 OK List completed 
0x660420 0 LOGIN "akonadi_kabc_resource_0" 
0x660420 0 OK User logged in 
0x660420 1 RESSELECT "akonadi_kabc_resource_0" 
0x660420 1 NO akonadi_kabc_resource_0 is not a valid resource identifier 
0x660420 2 BEGIN 
0x660420 2 OK Begin completed 
0x660420 3 LSUB 0 INF (RESOURCE akonadi_kabc_resource_0) (ANCESTORS 1) 
0x660420 3 NO Unknown resource 
0x660420 4 ROLLBACK 
0x660420 4 OK Rollback completed 
0x64bb60 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x659820 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x64fbb0 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x6b3620 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x64bb60 0 LOGIN "akonadi_birthdays_resource_0" 
0x64bb60 0 OK User logged in 
0x64bb60 1 RESSELECT "akonadi_birthdays_resource_0" 
0x64bb60 1 NO akonadi_birthdays_resource_0 is not a valid resource identifier 
0x6b3620 0 LOGIN "akonadi_maildir_resource_3" 
0x6b3620 0 OK User logged in 
0x6b3620 1 RESSELECT "akonadi_maildir_resource_3" 
0x6b3620 1 NO akonadi_maildir_resource_3 is not a valid resource identifier 
0x6ca7f0 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x6c35a0 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x6c35a0 0 LOGIN "akonadi_nepomuk_contact_feeder" 
0x6c35a0 0 OK User logged in 
0x6c35a0 1 LSUB 0 INF (MIMETYPE (text/directory application/x-vnd.kde.contactgroup)) () 
0x6c35a0 1 OK List completed 
0x64fbb0 0 LOGIN "akonadi_maildispatcher_agent" 
0x64fbb0 0 OK User logged in 
0x64fbb0 1 BEGIN 
0x64fbb0 1 OK Begin completed 
0x64fbb0 2 BEGIN 
0x64fbb0 2 OK Begin completed 
0x64fbb0 3 LSUB 0 INF (RESOURCE akonadi_maildir_resource_3) () 
0x64fbb0 3 NO Unknown resource 
0x64fbb0 4 ROLLBACK 
0x64fbb0 4 OK Rollback completed 
0x64fbb0 5 ROLLBACK 
0x64fbb0 5 OK Rollback completed 
0x659820 0 LOGIN "akonadi_nepomuk_email_feeder" 
0x659820 0 OK User logged in 
0x6f49e0 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x722100 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x6f49e0 0 LOGIN "krunner-1202724756" 
0x6f49e0 0 OK User logged in 
0x722100 0 LOGIN "konversation-1762071561" 
0x722100 0 OK User logged in 
0x724ca0 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x721860 * OK Akonadi Almost IMAP Server [PROTOCOL 23] 
0x724ca0 0 LOGIN "kontact-2104006769" 
0x724ca0 0 OK User logged in 
0x724ca0 5 SEARCH "prefix nco:<http://www.semanticdesktop.org/ontologies/2007/03/22/nco#>SELECT ?r WHERE { ?r a nco:PersonContact }" FULLPAYLOAD EXTERNALPAYLOAD (UID REMOTEID COLLECTIONID FLAGS SIZE DATETIME) 
0x721860 0 LOGIN "AkonadiConsole Browser Widget" 
0x721860 0 OK User logged in 
0x6ca7f0 0 LOGIN "akonadi_kcal_resource_0" 
0x6ca7f0 0 OK User logged in 
0x6ca7f0 1 RESSELECT "akonadi_kcal_resource_0" 
0x6ca7f0 1 NO akonadi_kcal_resource_0 is not a valid resource identifier 
0x6ca7f0 2 BEGIN 
0x6ca7f0 2 OK Begin completed 
0x6ca7f0 3 LSUB 0 INF (RESOURCE akonadi_kcal_resource_0) (ANCESTORS 1) 
0x6ca7f0 3 NO Unknown resource 
0x6ca7f0 4 ROLLBACK 
0x6ca7f0 4 OK Rollback completed 
0x724ca0 5 OK SEARCH completed 
0x724ca0 6 SEARCH "prefix nco:<http://www.semanticdesktop.org/ontologies/2007/03/22/nco#>SELECT ?r WHERE { ?r a nco:PersonContact }" FULLPAYLOAD EXTERNALPAYLOAD (UID REMOTEID COLLECTIONID FLAGS SIZE DATETIME) 
0x724ca0 6 OK SEARCH completed 

AgentBase(akonadi_kabc_resource_0): Unknown error. (Unknown resource)
AgentBase(akonadi_maildispatcher_agent): Could not access the outbox folder (Unknown error. (Unknown resource)).
AgentBase(akonadi_kcal_resource_0): Unknown error. (Unknown resource)

I managed to find a (the?) issue that, upon fixing, cures the problem above -- in my case.

On both boxes where I was able to reproduce these results, the akonadi calendar & address_book resource were carryovers from KDE 4.3.4 installs.

I.e., I'd simply upgraded KDE to 4.3.85.

Although all contacts and calendar events were available, and, once the hoops mentioned above were jumped through, I could send, receive, schedule, etc. w/o further problem.

On a whim, I quit Kontact completely, and, using the "Akonadi Resources Configuration" control panel, deleted the existing resources ('traditional' KDE calendar & address_book in folders), and simply re-created them, pointing to the same, previously assigned folders.

Shut everything down, rebooted the box, and now I can't get it to fail anymore.  Akonadi start/stop, or restart for that matter, no longer complains about Nepomuk/Strigi, send-mail works instantly, etc.

Seems something @upgrade didn't like the old resources?  Not sure, but the problem, for me, is gone.

*** Bug 217367 has been marked as a duplicate of this bug. ***

It hangs randomly, not just when trying to send the letter.
For example, now I was typing fast and once I weren't able to...

Interestingly, in the background the main program still checked my inbox because I could see the progress bar and the popup notification about the new unread letter.

And the solution for me now (in this case I haven't clicked the 'Send' button):
`killall kontact`. Everything works fine as nothing were happened. Really :) Nothing shut down, I could continue writing my letter.

Here is the backtrace from the bad state of the program:
#0  0x00007f20b860a033 in poll () from /lib64/libc.so.6
#1  0x00007f20b11a558c in g_main_context_poll (n_fds=<value optimized out>, fds=<value optimized out>, priority=<value optimized out>,
    timeout=<value optimized out>, context=<value optimized out>) at gmain.c:2904
#2  g_main_context_iterate (n_fds=<value optimized out>, fds=<value optimized out>, priority=<value optimized out>, timeout=<value optimized out>,
    context=<value optimized out>) at gmain.c:2586
#3  0x00007f20b11a58d0 in IA__g_main_context_iteration (context=0x647120, may_block=1) at gmain.c:2654
#4  0x00007f20b99aa0f3 in QEventDispatcherGlib::processEvents (this=0x610970, flags=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:412
#5  0x00007f20b8e1373e in QGuiEventDispatcherGlib::processEvents (this=0x1dd20a0, flags=<value optimized out>) at kernel/qguieventdispatcher_glib.cpp:204
#6  0x00007f20b997f482 in QEventLoop::processEvents (this=<value optimized out>, flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at kernel/qeventloop.cpp:149
#7  0x00007f20b997f85c in QEventLoop::exec (this=0x7fff9aac21e0, flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at kernel/qeventloop.cpp:201
#8  0x00007f20b9d8cfa6 in KJob::exec (this=0x2079730) at /usr/src/debug/kdelibs-4.3.86svn1068163/kdecore/jobs/kjob.cpp:204
#9  0x00007f20a259110f in Kleo::KeyResolver::lookupContactPreferences (this=0x107e220, address=...)
    at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/keyresolver.cpp:1741
#10 0x00007f20a2592d2e in Kleo::KeyResolver::getEncryptionItems (this=<value optimized out>, addresses=<value optimized out>)
    at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/keyresolver.cpp:935
#11 0x00007f20a2593217 in Kleo::KeyResolver::setPrimaryRecipients (this=0x107e220, addresses=...)
    at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/keyresolver.cpp:923
#12 0x00007f20a2582abb in MessageComposer::adjustCryptFlags (this=0x1f11610) at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/messagecomposer.cpp:736
#13 0x00007f20a2577558 in MessageComposer::slotDoNextJob (this=0x1f11610) at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/messagecomposer.cpp:435
#14 0x00007f20a2577631 in MessageComposer::qt_metacall (this=0x1f11610, _c=InvokeMetaMethod, _id=<value optimized out>, _a=0x7fff9aac2c00)
    at /usr/src/debug/kdepim-4.3.86svn1068163/build/kmail/messagecomposer.moc:76
#15 0x00007f20b999394f in QMetaObject::activate (sender=0x2581930, m=<value optimized out>, local_signal_index=<value optimized out>, argv=0xffffffffffffffff)
    at kernel/qobject.cpp:3267
#16 0x00007f20b999b12f in QSingleShotTimer::timerEvent (this=0x2581930) at kernel/qtimer.cpp:308
#17 0x00007f20b99905a3 in QObject::event (this=0x2581930, e=0x7fff9aac33a0) at kernel/qobject.cpp:1204
#18 0x00007f20b8d6577c in QApplicationPrivate::notify_helper (this=0x644540, receiver=0x2581930, e=0x7fff9aac33a0) at kernel/qapplication.cpp:4297
#19 0x00007f20b8d6bd5b in QApplication::notify (this=0x7fff9aac37b0, receiver=0x2581930, e=0x7fff9aac33a0) at kernel/qapplication.cpp:4180
#20 0x00007f20ba341b16 in KApplication::notify (this=0x7fff9aac37b0, receiver=0x2581930, event=0x7fff9aac33a0)
    at /usr/src/debug/kdelibs-4.3.86svn1068163/kdeui/kernel/kapplication.cpp:302
#21 0x00007f20b9980b6c in QCoreApplication::notifyInternal (this=0x7fff9aac37b0, receiver=0x2581930, event=0x7fff9aac33a0) at kernel/qcoreapplication.cpp:704
#22 0x00007f20b99ad895 in sendEvent (event=<value optimized out>, receiver=<value optimized out>) at kernel/qcoreapplication.h:215
#23 QTimerInfoList::activateTimers (event=<value optimized out>, receiver=<value optimized out>) at kernel/qeventdispatcher_unix.cpp:617
#24 0x00007f20b99aa428 in timerSourceDispatch (source=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:184
#25 idleTimerSourceDispatch (source=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:231
#26 0x00007f20b11a1dde in g_main_dispatch (context=<value optimized out>) at gmain.c:1960
#27 IA__g_main_context_dispatch (context=<value optimized out>) at gmain.c:2513
#28 0x00007f20b11a57a8 in g_main_context_iterate (context=0x647120, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
    at gmain.c:2591
#29 0x00007f20b11a58d0 in IA__g_main_context_iteration (context=0x647120, may_block=1) at gmain.c:2654
#30 0x00007f20b99aa0f3 in QEventDispatcherGlib::processEvents (this=0x610970, flags=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:412
#31 0x00007f20b8e1373e in QGuiEventDispatcherGlib::processEvents (this=0x1dd20a0, flags=<value optimized out>) at kernel/qguieventdispatcher_glib.cpp:204
#32 0x00007f20b997f482 in QEventLoop::processEvents (this=<value optimized out>, flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at kernel/qeventloop.cpp:149
#33 0x00007f20b997f85c in QEventLoop::exec (this=0x7fff9aac36f0, flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at kernel/qeventloop.cpp:201
#34 0x00007f20b99835ab in QCoreApplication::exec () at kernel/qcoreapplication.cpp:981
#35 0x0000000000403ed7 in main (argc=<value optimized out>, argv=<value optimized out>) at /usr/src/debug/kdepim-4.3.86svn1068163/kontact/src/main.cpp:221

And now, from the old-new working state:
#0  0x00007f20b8605dfb in read () from /lib64/libc.so.6
#1  0x00007f20ad6e9470 in ?? () from /usr/lib64/libxcb.so.1
#2  0x00007f20ad6e9998 in xcb_poll_for_event () from /usr/lib64/libxcb.so.1
#3  0x00007f20b2d62cad in ?? () from /usr/lib64/libX11.so.6
#4  0x00007f20b2d63547 in _XEventsQueued () from /usr/lib64/libX11.so.6
#5  0x00007f20b2d4c24b in XEventsQueued () from /usr/lib64/libX11.so.6
#6  0x00007f20b8e13967 in x11EventSourcePrepare (s=0x648450, timeout=<value optimized out>) at kernel/qguieventdispatcher_glib.cpp:79
#7  0x00007f20b11a4fca in IA__g_main_context_prepare (context=0x647120, priority=<value optimized out>) at gmain.c:2280
#8  0x00007f20b11a53a1 in g_main_context_iterate (context=0x647120, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
    at gmain.c:2571
#9  0x00007f20b11a58d0 in IA__g_main_context_iteration (context=0x647120, may_block=0) at gmain.c:2654
#10 0x00007f20b99aa0f3 in QEventDispatcherGlib::processEvents (this=0x610970, flags=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:412
#11 0x00007f20b8e1373e in QGuiEventDispatcherGlib::processEvents (this=0x8, flags=<value optimized out>) at kernel/qguieventdispatcher_glib.cpp:204
#12 0x00007f20b998370f in QCoreApplication::processEvents (flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at kernel/qcoreapplication.cpp:896
#13 0x00007f20a2438428 in KMKernel::dumpDeadLetters (this=<value optimized out>) at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/kmkernel.cpp:1915
#14 0x00007f20a24fb250 in kmsignalHandler (sigId=15) at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/kmstartup.cpp:58
#15 <signal handler called>
#16 0x00007f20b860a033 in poll () from /lib64/libc.so.6
#17 0x00007f20b11a558c in g_main_context_poll (n_fds=<value optimized out>, fds=<value optimized out>, priority=<value optimized out>,
    timeout=<value optimized out>, context=<value optimized out>) at gmain.c:2904
#18 g_main_context_iterate (n_fds=<value optimized out>, fds=<value optimized out>, priority=<value optimized out>, timeout=<value optimized out>,
    context=<value optimized out>) at gmain.c:2586
#19 0x00007f20b11a58d0 in IA__g_main_context_iteration (context=0x647120, may_block=1) at gmain.c:2654
#20 0x00007f20b99aa0f3 in QEventDispatcherGlib::processEvents (this=0x610970, flags=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:412
#21 0x00007f20b8e1373e in QGuiEventDispatcherGlib::processEvents (this=0x1dd20a0, flags=<value optimized out>) at kernel/qguieventdispatcher_glib.cpp:204
#22 0x00007f20b997f482 in QEventLoop::processEvents (this=<value optimized out>, flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at kernel/qeventloop.cpp:149
#23 0x00007f20b997f85c in QEventLoop::exec (this=0x7fff9aac21e0, flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at kernel/qeventloop.cpp:201
#24 0x00007f20b9d8cfa6 in KJob::exec (this=0x2079730) at /usr/src/debug/kdelibs-4.3.86svn1068163/kdecore/jobs/kjob.cpp:204
#25 0x00007f20a259110f in Kleo::KeyResolver::lookupContactPreferences (this=0x107e220, address=...)
    at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/keyresolver.cpp:1741
#26 0x00007f20a2592d2e in Kleo::KeyResolver::getEncryptionItems (this=<value optimized out>, addresses=<value optimized out>)
    at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/keyresolver.cpp:935
#27 0x00007f20a2593217 in Kleo::KeyResolver::setPrimaryRecipients (this=0x107e220, addresses=...)
    at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/keyresolver.cpp:923
#28 0x00007f20a2582abb in MessageComposer::adjustCryptFlags (this=0x1f11610) at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/messagecomposer.cpp:736
#29 0x00007f20a2577558 in MessageComposer::slotDoNextJob (this=0x1f11610) at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/messagecomposer.cpp:435
#30 0x00007f20a2577631 in MessageComposer::qt_metacall (this=0x1f11610, _c=InvokeMetaMethod, _id=<value optimized out>, _a=0x7fff9aac2c00)
    at /usr/src/debug/kdepim-4.3.86svn1068163/build/kmail/messagecomposer.moc:76
#31 0x00007f20b999394f in QMetaObject::activate (sender=0x2581930, m=<value optimized out>, local_signal_index=<value optimized out>, argv=0xffffffffffffffff)
    at kernel/qobject.cpp:3267
#32 0x00007f20b999b12f in QSingleShotTimer::timerEvent (this=0x2581930) at kernel/qtimer.cpp:308
#33 0x00007f20b99905a3 in QObject::event (this=0x2581930, e=0x7fff9aac33a0) at kernel/qobject.cpp:1204
#34 0x00007f20b8d6577c in QApplicationPrivate::notify_helper (this=0x644540, receiver=0x2581930, e=0x7fff9aac33a0) at kernel/qapplication.cpp:4297
#35 0x00007f20b8d6bd5b in QApplication::notify (this=0x7fff9aac37b0, receiver=0x2581930, e=0x7fff9aac33a0) at kernel/qapplication.cpp:4180
#36 0x00007f20ba341b16 in KApplication::notify (this=0x7fff9aac37b0, receiver=0x2581930, event=0x7fff9aac33a0)
    at /usr/src/debug/kdelibs-4.3.86svn1068163/kdeui/kernel/kapplication.cpp:302
#37 0x00007f20b9980b6c in QCoreApplication::notifyInternal (this=0x7fff9aac37b0, receiver=0x2581930, event=0x7fff9aac33a0) at kernel/qcoreapplication.cpp:704
#38 0x00007f20b99ad895 in sendEvent (event=<value optimized out>, receiver=<value optimized out>) at kernel/qcoreapplication.h:215
#39 QTimerInfoList::activateTimers (event=<value optimized out>, receiver=<value optimized out>) at kernel/qeventdispatcher_unix.cpp:617
#40 0x00007f20b99aa428 in timerSourceDispatch (source=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:184
#41 idleTimerSourceDispatch (source=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:231
#42 0x00007f20b11a1dde in g_main_dispatch (context=<value optimized out>) at gmain.c:1960
#43 IA__g_main_context_dispatch (context=<value optimized out>) at gmain.c:2513
#44 0x00007f20b11a57a8 in g_main_context_iterate (context=0x647120, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
    at gmain.c:2591
#45 0x00007f20b11a58d0 in IA__g_main_context_iteration (context=0x647120, may_block=1) at gmain.c:2654
#46 0x00007f20b99aa0f3 in QEventDispatcherGlib::processEvents (this=0x610970, flags=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:412
#47 0x00007f20b8e1373e in QGuiEventDispatcherGlib::processEvents (this=0x1dd20a0, flags=<value optimized out>) at kernel/qguieventdispatcher_glib.cpp:204
#48 0x00007f20b997f482 in QEventLoop::processEvents (this=<value optimized out>, flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at kernel/qeventloop.cpp:149
#49 0x00007f20b997f85c in QEventLoop::exec (this=0x7fff9aac36f0, flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at kernel/qeventloop.cpp:201
#50 0x00007f20b99835ab in QCoreApplication::exec () at kernel/qcoreapplication.cpp:981
#51 0x0000000000403ed7 in main (argc=<value optimized out>, argv=<value optimized out>) at /usr/src/debug/kdepim-4.3.86svn1068163/kontact/src/main.cpp:221

The only difference between them is the head of the latter:
#0  0x00007f20b8605dfb in read () from /lib64/libc.so.6
#1  0x00007f20ad6e9470 in ?? () from /usr/lib64/libxcb.so.1
#2  0x00007f20ad6e9998 in xcb_poll_for_event () from /usr/lib64/libxcb.so.1
#3  0x00007f20b2d62cad in ?? () from /usr/lib64/libX11.so.6
#4  0x00007f20b2d63547 in _XEventsQueued () from /usr/lib64/libX11.so.6
#5  0x00007f20b2d4c24b in XEventsQueued () from /usr/lib64/libX11.so.6
#6  0x00007f20b8e13967 in x11EventSourcePrepare (s=0x648450, timeout=<value optimized out>) at kernel/qguieventdispatcher_glib.cpp:79
#7  0x00007f20b11a4fca in IA__g_main_context_prepare (context=0x647120, priority=<value optimized out>) at gmain.c:2280
#8  0x00007f20b11a53a1 in g_main_context_iterate (context=0x647120, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
    at gmain.c:2571
#9  0x00007f20b11a58d0 in IA__g_main_context_iteration (context=0x647120, may_block=0) at gmain.c:2654
#10 0x00007f20b99aa0f3 in QEventDispatcherGlib::processEvents (this=0x610970, flags=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:412
#11 0x00007f20b8e1373e in QGuiEventDispatcherGlib::processEvents (this=0x8, flags=<value optimized out>) at kernel/qguieventdispatcher_glib.cpp:204
#12 0x00007f20b998370f in QCoreApplication::processEvents (flags=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at kernel/qcoreapplication.cpp:896
#13 0x00007f20a2438428 in KMKernel::dumpDeadLetters (this=<value optimized out>) at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/kmkernel.cpp:1915
#14 0x00007f20a24fb250 in kmsignalHandler (sigId=15) at /usr/src/debug/kdepim-4.3.86svn1068163/kmail/kmstartup.cpp:58
#15 <signal handler called>

[the rest are equal]

*** Bug 222170 has been marked as a duplicate of this bug. ***

from Bug 222170:

> However, in your case, I suspect a misconfigured Akoandi/Nepomuk, check that
> "akonadictl status" says it is using Virtuoso as backend.

Well, it did run:
Akonadi Control: running
Akonadi Server: running
Akonadi Server Search Support: available (backend: Virtuoso)

After some testing, suddenly there was no lag in kmail. And well, Akonadi was not using Virtuoso anymore:
Akonadi Control: running
Akonadi Server: running
Akonadi Server Search Support: not available

So for my trouble it seems that when not using Akonadi+Virtuoso, everything is fast, but when Akonadi is using Virtuoso, it is slow, eats lots of memory (permanently) and cpu (temporarily).

About freezing while sending mail:
Usually there is only a short lag for me (5-6 seconds), but I did experience a long freeze (about 2 minutes) when trying a few combinations of sending/receiving address pairs.

akonadictl status shows also for me:
Akonadi Control: running
Akonadi Server: running
Akonadi Server Search Support: available (backend: Virtuoso)

I'm at KDE 4.4 RC1 now. Sending a single small mail takes minimum 20 - 30
seconds.

My hardware:
Processor (CPU): Intel(R) Pentium(R) D CPU 3.00GHz
Speed:  3,000.00 MHz
Cores:  2
Total memory (RAM): 1.8 GiB
This is not a hardware I would expect such a delay from.

In KDE 4.3.X this problem didn't exist.
I have done a straight upgrade on them using the OpenSUSE 11.2 KDE4 Desktop
Factory packages.

Since the applications hang in exec(), I have to kill them, which causes index corruption, so this bug even leads to data loss.

I don't think we should release KMail with this bug, adding blocker keyword.

Backtrace of the Akonadi server during one of the hangs.
There seem to be multiple different hangs, though.

#0  0x00007f32c5590396 in poll () from /lib64/libc.so.6
#1  0x00007f32c438b0a0 in ?? () from /lib64/libdbus-1.so.3
#2  0x00007f32c4389464 in ?? () from /lib64/libdbus-1.so.3
#3  0x00007f32c437610e in ?? () from /lib64/libdbus-1.so.3
#4  0x00007f32c437830a in ?? () from /lib64/libdbus-1.so.3
#5  0x00007f32c437746c in dbus_connection_send_with_reply_and_block () from /lib64/libdbus-1.so.3
#6  0x00007f32c6584e6d in q_dbus_connection_send_with_reply_and_block (connection=0x63ade0, message=0x7f32bc052a40, timeout_milliseconds=-1, error=0x7f32b97f88a0) at qdbus_symbols
_p.h:133
#7  0x00007f32c6595182 in QDBusConnectionPrivate::sendWithReply (this=0x639c00, message=..., sendMode=1, timeout=-1) at qdbusintegrator.cpp:1811
#8  0x00007f32c657c89f in QDBusConnection::call (this=0x641510, message=..., mode=Block, timeout=-1) at qdbusconnection.cpp:516
#9  0x00007f32c65a6b80 in QDBusAbstractInterface::callWithArgumentList (this=0x641360, mode=Block, method=..., args=...) at qdbusabstractinterface.cpp:440
#10 0x00007f32c65a6dd3 in QDBusAbstractInterface::internalConstCall (this=0x641360, mode=AutoDetect, method=..., args=...) at qdbusabstractinterface.cpp:763
#11 0x00007f32c658211d in QDBusConnectionInterface::isServiceRegistered (this=0x641360, serviceName=...) at qdbusconnectioninterface.cpp:207
#12 0x00007f32c7a99556 in Nepomuk::Search::QueryServiceClient::serviceAvailable() () from /media/kdedev/trunk/kde/lib/libakonadiprivate.so.1
#13 0x00007f32c7b24b78 in Akonadi::NepomukSearch::NepomukSearch(QObject*) () from /media/kdedev/trunk/kde/lib/libakonadiprivate.so.1
#14 0x00007f32c7a8a804 in Akonadi::Search::parseStream() () from /media/kdedev/trunk/kde/lib/libakonadiprivate.so.1
#15 0x00007f32c7a6170e in Akonadi::AkonadiConnection::slotNewData() () from /media/kdedev/trunk/kde/lib/libakonadiprivate.so.1
#16 0x00007f32c7a6215e in Akonadi::AkonadiConnection::qt_metacall(QMetaObject::Call, int, void**) () from /media/kdedev/trunk/kde/lib/libakonadiprivate.so.1
#17 0x00007f32c764199b in QMetaObject::metacall (object=0x72b670, cl=InvokeMetaMethod, idx=12, argv=0x7f32b97f97b0) at kernel/qmetaobject.cpp:237
#18 0x00007f32c7656c9a in QMetaObject::activate (sender=0x72e230, m=0x7f32c79af9a0, local_signal_index=0, argv=0x0) at kernel/qobject.cpp:3275
#19 0x00007f32c76bd55a in QIODevice::readyRead (this=0x72e230) at .moc/debug-shared/moc_qiodevice.cpp:91
#20 0x00007f32c76bd5cd in QIODevice::qt_metacall (this=0x72e230, _c=InvokeMetaMethod, _id=0, _a=0x7f32b97f99c0) at .moc/debug-shared/moc_qiodevice.cpp:77
#21 0x00007f32c6fb9c02 in QLocalSocket::qt_metacall (this=0x72e230, _c=InvokeMetaMethod, _id=4, _a=0x7f32b97f99c0) at .moc/debug-shared/moc_qlocalsocket.cpp:81
#22 0x00007f32c764199b in QMetaObject::metacall (object=0x72e230, cl=InvokeMetaMethod, idx=4, argv=0x7f32b97f99c0) at kernel/qmetaobject.cpp:237
#23 0x00007f32c7656c9a in QMetaObject::activate (sender=0x72dc18, m=0x7f32c79af9a0, local_signal_index=0, argv=0x0) at kernel/qobject.cpp:3275
#24 0x00007f32c76bd55a in QIODevice::readyRead (this=0x72dc18) at .moc/debug-
shared/moc_qiodevice.cpp:91
#25 0x00007f32c6fb2597 in QAbstractSocketPrivate::canReadNotification (this=0x733200) at socket/qabstractsocket.cpp:626
#26 0x00007f32c6fb5ec5 in QAbstractSocketPrivate::readNotification (this=0x733200) at socket/qabstractsocket_p.h:77
#27 0x00007f32c6f9b54d in QAbstractSocketEngine::readNotification (this=0x712020) at socket/qabstractsocketengine.cpp:154
#28 0x00007f32c6f9cecf in QReadNotifier::event (this=0x72d8d0, e=0x7f32b97f9c80) at socket/qnativesocketengine.cpp:1089
#29 0x00007f32c763779a in QCoreApplicationPrivate::notify_helper (this=0x61dcd0, receiver=0x72d8d0, event=0x7f32b97f9c80) at kernel/qcoreapplication.cpp:839
#30 0x00007f32c763b031 in QCoreApplication::notify (this=0x7fff77ee9860, receiver=0x72d8d0, event=0x7f32b97f9c80) at kernel/qcoreapplication.cpp:785
#31 0x00007f32c7639b81 in QCoreApplication::notifyInternal (this=0x7fff77ee9860, receiver=0x72d8d0, event=0x7f32b97f9c80) at kernel/qcoreapplication.cpp:704
#32 0x00007f32c763eed7 in QCoreApplication::sendEvent (receiver=0x72d8d0, event=0x7f32b97f9c80) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:215
#33 0x00007f32c7676bfe in socketNotifierSourceDispatch (source=0x732aa0) at kernel/qeventdispatcher_glib.cpp:110
#34 0x00007f32c4a130fb in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#35 0x00007f32c4a168cd in ?? () from /usr/lib64/libglib-2.0.so.0
#36 0x00007f32c4a16a8b in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#37 0x00007f32c7675579 in QEventDispatcherGlib::processEvents (this=0x7287a0, flags=...) at kernel/qeventdispatcher_glib.cpp:412
#38 0x00007f32c7635f7b in QEventLoop::processEvents (this=0x7f32b97f9fa0, flags=...) at kernel/qeventloop.cpp:149
#39 0x00007f32c763619f in QEventLoop::exec (this=0x7f32b97f9fa0, flags=...) at kernel/qeventloop.cpp:201
#40 0x00007f32c7508dbf in QThread::exec (this=0x72b670) at thread/qthread.cpp:487
#41 0x00007f32c7a62592 in Akonadi::AkonadiConnection::run() () from /media/kdedev/trunk/kde/lib/libakonadiprivate.so.1
#42 0x00007f32c750d962 in QThreadPrivate::start (arg=0x72b670) at thread/qthread_unix.cpp:248
#43 0x00007f32c725c070 in start_thread () from /lib64/libpthread.so.0
#44 0x00007f32c559911d in clone () from /lib64/libc.so.6

Created attachment 40045 [details]
Patch that makes reproducing the problem easier

Small test that runs contact search jobs in an infinite loop, to make it easier to reproduce the problem.

I was too lazy to write a standalone app for this, so i integrated it in KMail 2: Simply click "Message->Feeze me" and watch the debug output. The bug is probably gone if it manages >5000 jobs without problems.

SVN commit 1077216 by tmcguire:

Work around a DBus bug that using a connection from within multiple threads can cause freezes.
In this case, that connection was the session bus, on which we called isServiceRegistered.

The workaround: Just don't check if the search service is available, just call the methods. They'll
fail properly if the service is not there.

Volker, can you backport this fix to older Akonadi releases and prepare tarballs, so that we can
notify the release team about this?

Should fix
BUG: 219687


 M  +7 -8      nepomuksearch.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1077216

FYI, the DBus bug report for this is at https://bugs.freedesktop.org/show_bug.cgi?id=857.

And the Qt bug report is at http://bugreports.qt.nokia.com/browse/QTBUG-7475

Can someone with the necessary permissions please reopen this bug? This is not fixed for me yet, symtoms are still exactly the same (nepomuk still leaking memory and with high cpu load, kmail still freezing). Or should I reopen my "duplicate" bug instead?

Software stack used right now:

KDE SC 4.4rc2
QT 4.6.1
Akonadi-Server 1.3.0

Leaking memory? Which process exactly is leaking memory?

As described in Bug 222170 (resolved duplicate), each time I click on a mail that has one of my mail addresses in the from-field or has the non-domain part of my gmx address (smorg) in the subject-field, nepomuk uses a lot of cpu and increases its memory use gradually. This happens at a rate of approximately 2,5MB/s and does not stop until it consumes at least 300MB more than before (it used to be at least 500MB, sometimes 1GB with rc1). Once that point is reached, the cpu usage drops back to normal levels and the gui of kmail again responds to user actions. The memory usage of nepomuk stays at the high level though.

If I click at another mail later (with my address in the from-field...), this game continues for additional 300MB/500MB/1GB on top of the previous amount. Killing the nepomuk process after it released the cpu seems to have no negative side effects though, so I have not yet let nepomuk use more than 3GB of memory.

when using htop, the process that accumulates all the memory shows up as "/usr/bin/nepomukservicestub nepomukqueryservice"

Tagging date of KDE 4.4 is 3rd February. You mustn't release KMail with this severe bug in it. Please, reopen as soon as possible.

My daily Kontact opening procedure is (on every startup):
- Write password into KWallet
- Send a mail to random address, KMail freezes
- killall -9 kontact
- Alt + F2 > Kontact
- A warning dialog pops up
- Delete the folder that Kontact cannot (Bug 217369) after closing the dialog
- _Now_ it works as it should from the first step

Seriously, you can't expect every user has the ability (and patience) to do this for months...

i think you can also add https://bugs.kde.org/show_bug.cgi?id=222332 to this
see Seb's comment 10: https://bugs.kde.org/show_bug.cgi?id=222332#c10 regarding how to debug the memory usage issue.

(In reply to comment #24)

> see Seb's comment 10: https://bugs.kde.org/show_bug.cgi?id=222332#c10 regarding
> how to debug the memory usage issue.

Unfortunately valgrind does not cope well with code optimized for newer AMD cpus (-march=amd10fam). In many cases it cannot analyze the code properly yet. In the case of nepomuk (and most other kde programs), I cannot use valgrind for debugging until at least bug 180217 is fixed.

Skander, there seems to be a patch to the bug #180217 comment #11

Interesting, but sometimes KMail is able to send a message when there is no internet connection (NetworkManager is connecting to the wireless, but it isn't finished). In this case it reports an error and stores the message in the Outgoing folder. And is also can send more messages after the net is established.

I would be interested in the output of
qdbus org.kde.nepomuk.services.nepomukqueryservice
once the mem usage of the query service is way up again.

(In reply to comment #28)
> I would be interested in the output of
> qdbus org.kde.nepomuk.services.nepomukqueryservice
> once the mem usage of the query service is way up again.

skander@balar ~ $ qdbus org.kde.nepomuk.services.nepomukqueryservice
/
/nepomukqueryservice
/nepomukqueryservice/query16
/servicecontrol

The number of the query is increased when the next "problematic" case is triggered. Once the search job is done, the line containing queryN is omitted from the output.

(In reply to comment #26)
Yes, with that patch valgrind is usable

I will reopen my "duplicate" bug 222170 now and would like to continue there. One reason is that no one seems to be willing or able to reopen this one and at least my problem is not resolved. Furthermore it seems to me there are some different cases mixed up in this bug report.

Created attachment 40323 [details]
Fix for memory leak in libnepomuk

Please apply the attached patch in kdelibs/nepomuk/core and see if that solves the memory issue.

(In reply to comment #30)
> Please apply the attached patch in kdelibs/nepomuk/core and see if that
> solves the memory issue.

It does, thanks! Memory usage stays low now.

While investigating the problem further, I noticed the console output of "nepomukservicestub nepomukqueryservice"

Reading that output told me that each time another mail message is displayed, a nepomuk search for the address in the from-field of the mail is started. I have no clue if this is useful, from a user point of view it does not seem necessary to the expected task (display that mail). Unfortunately those searches can take a long time to complete (up to more than 1 hour!). And even worse, the gui is frozen until the search job is completed.

I noticed that at the end of the output from "nepomukservicestub nepomukqueryservice" there is a line containing the following text:

nepomukqueryservice(3385)/nepomuk (query service) Nepomuk::Query::SearchThread::run: SOMENUMBER

The number displayed there seems to be the number of hits of the query. The search job is fast for addresses that do not produce many hits (e.g. 10 or 25), but my three topmost used addresses produced 122136, 246050 and 5206017 hits (the last one took more than 1 hour!)

So, a few questions arise (at least for me):
1) Is it really necessary (or useful) to start a search each time an other message is displayed? I did not notice any disadvantage when the akonadi search support was disabled.
2) If it is needed/wanted, can the nepomuk search job be sped to be fast enough? Even if it would be sped up by a factor of 1000, I would still notice a lag of a few seconds if I click on a mail that has my most used address in the from-header.
3) Is it necessary that the gui is frozen until the search job is finnished?

After applying the patch I am getting crashes by just opening and closing dolphin, I have added it as a new bug report: https://bugs.kde.org/show_bug.cgi?id=224717 with bt. It does look like the patch needs another patch :)

should I reopen this report?

the memory usage seems to be under control now :)

damn, memory is out of hand (nepomukfilewatch) and increasing :( even after the patch

kde-devel@dummy:~/Desktop$ ps guax | grep -i nepomuk
kde-devel     5627  0.0  0.3 330080 21032 ?        Sl   14:38   0:01 kdeinit4: nepomukserver [kdeinit]
kde-devel     5637  3.5  0.3 349220 23516 ?        Sl   14:38   5:58 /opt/kde/bin/nepomukservicestub nepomukstorage
kde-devel     5649  0.8  0.2 185744 18216 ?        S    14:38   1:29 /opt/kde/bin/nepomukservicestub nepomukqueryservice
kde-devel     5650  0.8  0.3 224620 22808 ?        S    14:38   1:29 /opt/kde/bin/nepomukservicestub nepomukontologyloader
kde-devel     5651  1.0  0.3 175408 18472 ?        S    14:38   1:44 /opt/kde/bin/nepomukservicestub nepomukremovablestorageservice
kde-devel     5652  0.9 13.4 992028 824308 ?       Sl   14:38   1:31 /opt/kde/bin/nepomukservicestub nepomukfilewatch
kde-devel     5653  4.5  0.9 371668 59052 ?        SNl  14:38   7:35 /opt/kde/bin/nepomukservicestub nepomukstrigiservice
kde-devel    12863  0.0  0.0   7532  1004 pts/3    S+   17:26   0:00 grep -i nepomuk

@SlashDevDsp: stupid question just to make sure: you restarted the filwatch service after applying the patch, right?

yes I have

Are you moving around a lot of files while this is happening? Could you maybe give me the output of the service from its start up to the high mem usage?

This report mixing together two independent bugs, so I reopen my original one, Bug 217367.

@Szots: but your memory leaking problem is definitely fixed by the patch?

I haven't tried your patch yet, but I don't know how to cause the memory leak either.
I'm currently working now but after I go home I download the source and try to patch it and somehow generate the leakage.

@Szots: sorry, I actually meant to ask Skander about the memleak since he wrote "It does, thanks! Memory usage stays low now.".
@Skander: so you are positive about the fix then?

Ok, the problem about the slow searches seems to be that Strigi file indexer indexes KMail's maildir folder, and creates one Contact object for the receivers and senders of each mail. That of course is a lot for big maildir folders.

Now, when KMail searches for contacts that match the email address of the sender of a mail (it does that to e.g. display a contact photo), the resource usage of Nepomuk skyrockets since there are so many results.

The solutions for the problem are:
1) Limit the search to only return 1 result. We ignore the rest anyway
2) Don't let the Strigi file indexer unconditionally create Contact objects for 
   each mail it indexes(!)
3) When searching, only search for the contacts created by the Nepomuk Contact 
   Feeder, not for other contacts
4) For KMail 2, let the maildir resource and the mbox resource disable Strigi 
   indexing of their folders and files

SVN commit 1081954 by tmcguire:

Add the ability to limit the number of search results.

CCBUG: 219687


 M  +12 -1     contactsearchjob.cpp  
 M  +11 -0     contactsearchjob.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1081954

Just to clarify something about the bug report: It basically mixed two issues:

1) Search jobs hanging totally and therefore freezing the KMail GUI forever
2) Search jobs being slow and therefore freezing KMail GUI for a period of time

1) is fixed by the Akonadi server fix in r1077216.
2) not yet fixed, we're working on that. These are basically the 4 solutions 
   outlined in comment 42

SVN commit 1081964 by tmcguire:

Set a limit of 1 result for search jobs where we don't need more than 1 result anyway.
Should help with bug 219687.

CCBUG: 219687


 M  +2 -0      kmail/keyresolver.cpp  
 M  +2 -1      kmail/kmmainwidget.cpp  
 M  +1 -0      kmail/kmreadermainwin.cpp  
 M  +2 -0      kmail/kmsearchpattern.cpp  
 M  +2 -1      kmail/xfaceconfigurator.cpp  
 M  +3 -0      libkdepim/kaddrbookexternal.cpp  
 M  +3 -0      messageviewer/headerstyle.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1081964

(In reply to comment #41)
> @Szots: sorry, I actually meant to ask Skander about the memleak since he wrote
> "It does, thanks! Memory usage stays low now.".
> @Skander: so you are positive about the fix then?

Yes, so far it works great. I did not notic any further memory problems.

(In reply to comment #42)
Ah, thanks for the explanation. And for the patches in comment #43 and #45, I
will definitely test it.

SVN commit 1081970 by tmcguire:

Backport r1081954 and r1081959 by tmcguire from trunk to the 4.4 branch:

- Add the ability to limit the number of search results.
- Actually implement setLimit()

CCBUG: 219687



 M  +17 -1     contactsearchjob.cpp  
 M  +11 -0     contactsearchjob.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1081970

SVN commit 1081978 by tmcguire:

Backport r1081964 by tmcguire from trunk to the 4.4 branch:

Set a limit of 1 result for search jobs where we don't need more than 1 result anyway.
Should help with bug 219687.

CCBUG: 219687



 M  +1 -0      kmail/headerstyle.cpp  
 M  +2 -0      kmail/keyresolver.cpp  
 M  +1 -0      kmail/kmmainwidget.cpp  
 M  +1 -0      kmail/kmreadermainwin.cpp  
 M  +2 -0      kmail/kmsearchpattern.cpp  
 M  +1 -0      kmail/xfaceconfigurator.cpp  
 M  +3 -0      libkdepim/kaddrbookexternal.cpp  
 M  +3 -0      messageviewer/headerstyle.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1081978

Regarding comment #42: The limit for search results is now implemented in _some_ cases, like for reading or sending mail. In other cases it is not implemented, as we actually need to iterate over all results. To fix that, 3) needs to be implemented.

Nevertheless, normal mail reading and sending should not cause such a high Nepomuk resource usage anymore.
This has been backported to the 4.4 branch.

Reopening and lowering priority. The main issue is fixed, but still, all points in comment#42 should be implemented.

(In reply to comment #49)
> Nevertheless, normal mail reading and sending should not cause such a high
> Nepomuk resource usage anymore.
> This has been backported to the 4.4 branch.

Yes, thank you! Works great (unsing 4.4rc3). Now I can finally enjoy kmail again :)

*** Bug 224134 has been marked as a duplicate of this bug. ***

*** Bug 227447 has been marked as a duplicate of this bug. ***

*** Bug 227649 has been marked as a duplicate of this bug. ***

*** Bug 200453 has been marked as a duplicate of this bug. ***

I still see temporary UI hangs during mail send. 

looking at akonadiconsole I see 3 calls per send to  KMail::StringUtil::expandAliases that each start a nested event loop and return all details of all contacts(!), perhaps this is slow?

First call - immediately on clicking Send:
#8  0xb5e46972 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) ()    
   from /usr/lib/libQtCore.so.4                                                 
#9  0xb71a6319 in KJob::exec() () from /usr/lib/libkdecore.so.5                 
#10 0xb6bb154b in KabcBridge::expandNickName (nickName=@0xbfeb182c)             
    at /usr/src/debug/kdepim-4.4.0/kmail/kmaddrbook.cpp:117                     
#11 0xb6e6abe3 in KMail::StringUtil::expandAliases (recipients=@0xbfeb1afc,     
    distributionListEmpty=@0xbfeb18f0)                                          
    at /usr/src/debug/kdepim-4.4.0/kmail/stringutil.cpp:944                     
#12 0xb6e230bd in KMail::Util::validateAddresses (parent=0x8b12428,             
    addresses=@0xbfeb1afc) at /usr/src/debug/kdepim-4.4.0/kmail/util.cpp:141    
#13 0xb6adf303 in KMComposeWin::doSend (this=0x8b12428,                         
    method=KMail::MessageSender::SendImmediate, saveIn=KMComposeWin::None)      


second call - after the widgets of the send window are all disabled:
#8  0xb5e46972 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) ()    
   from /usr/lib/libQtCore.so.4                                                 
#9  0xb71a6319 in KJob::exec() () from /usr/lib/libkdecore.so.5                 
#10 0xb6bb154b in KabcBridge::expandNickName (nickName=@0xbfeb1f4c)             
    at /usr/src/debug/kdepim-4.4.0/kmail/kmaddrbook.cpp:117                     
#11 0xb6e6abe3 in KMail::StringUtil::expandAliases (recipients=@0xbfeb2008,     
    distributionListEmpty=@0xbfeb2014)                                          
    at /usr/src/debug/kdepim-4.4.0/kmail/stringutil.cpp:944                     
#12 0xb6ad62b6 in KMComposeWin::slotContinueDoSend (this=0x8b12428,             
    sentOk=true) at /usr/src/debug/kdepim-4.4.0/kmail/kmcomposewin.cpp:3703

I couldn't catch the third (or zero'th) hang yet.

I can see processEvents is getting called each time so one of these events is causing the hang.

@Will: Hmm, the expandNickName case should be fixed, I remember adding a SPARQL query that searches only for the nickname, and limits the results. Probably I forgot to backport that, I'll take a look.

Ok, I backported the expandNickName() fix to the 4.4 branch in r1092906,1092907,1092908 and 1092911.

Sending now is faster again (to the point where it is no longer annoying, for me)

*** Bug 228020 has been marked as a duplicate of this bug. ***

*** Bug 228022 has been marked as a duplicate of this bug. ***

SVN commit 1094466 by tmcguire:

Improve search performance for searches that don't have a LIMIT set.
- Only search in PersonContacts. The strigi file indexer creates ordinary Contact object, we
  don't want to search them.
- use DISTINCT
- use "graph ?g"

Thanks to Yury G. Kudryashov for his help on IRC.

The search speed was improved from 220 seconds to 20 milliseconds (!) here, when searching for
my own mail address.

CCBUG: 219687


 M  +10 -9     contactsearchjob.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1094466

SVN commit 1094467 by tmcguire:

Backport r1094466 by tmcguire from trunk to the 4.4 branch:

Improve search performance for searches that don't have a LIMIT set.
- Only search in PersonContacts. The strigi file indexer creates ordinary Contact object, we
  don't want to search them.
- use DISTINCT
- use "graph ?g"

Thanks to Yury G. Kudryashov for his help on IRC.

The search speed was improved from 220 seconds to 20 milliseconds (!) here, when searching for
my own mail address.

CCBUG: 219687



 M  +10 -9     contactsearchjob.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1094467

*** Bug 228282 has been marked as a duplicate of this bug. ***

*** Bug 228448 has been marked as a duplicate of this bug. ***

For all those having problems with sending taking a long time, there is a workaround until 4.4.1 is released:
- Disable Strigi file indexing. It will index your mails, which in Nepomuk will create many Contact objects, which will massively slow down searching for contacts (which is done when sending)
- Delete your Nepomuk database to remove those Contact objects Strigi has created. Warning: this will also remove all your tags, ratings and annotations and other Nepomuk data. The Nepomuk database is in $KDEHOME/share/apps/nepomuk/.

Before you start deleting all your data please have a look at the available documentation: http://techbase.kde.org/Development/Tutorials/Metadata/Nepomuk/TipsAndTricks#Remove_all_Strigi-indexed_data

Disabling strigi and removing the database did initially seem to have cured the problem, in that the first message I composed left quickly.  Then I had to leave the keyboard, and KMail was open.  On my return I composed another message.  I hit send several minutes ago.  According to System Activity KMail is still running, but using minimal CPU, and kio_imap4 plus four kio)files are listed.  KMail main window and composer window are just blank boxes, and top says I have a zombie.  I've no way of knowing whether this is related or not.

Strike that comment.  It's probably unrelated.  Whatever had caused it, something does seem to have changed. Not only are message being sent at a decent rate, but Kontact has actually started up.  Perhaps bug # 227636 us related to this after all

I followed the hints to remove all strigi entries (now my virtuoso database is only ~100mb where it used to be over 600mb), but there's still a problem when sending mails to many recipients. It has taken about 10 minutes to send one email so far (still not finished), nepomukqueryservice takes about 30% percent of my cpu, and akonadiconsole debugger shows the same massive input as before.

*** Bug 223763 has been marked as a duplicate of this bug. ***

When I hit SEND for a simple test message, I get the following CPU usage:
virtuoso-t   45%
nepomukservices    30%
nepomukservices    15%

I don't know why there are two nepomuk services.
My nepomuk widget says I have 3037 files indexed for a size of 720 MByte.

Delete file index using script referenced in an above link. What's interesting is that the script takes the file count to zero, but the file size is still > 700 MByte. So I deleted the db file by hand. Logged-out, then logged-in. nepomuk immediately kicked-in to start indexing; I stopped it at about 20 MBytes.

Started kmail. Was able to send a test message "normally" (meaning hit SEND and it's off immediately). Closed kmail.

Re-started indexer. Got to about 300 MByte in size. Suspended indexing.

Started kmail. Now a test message was taking a long time as before -- roughly 45 seconds.

> Delete file index using script referenced in an above link.
> What's interesting is that the script takes the file count to zero, but
> the file size is still > 700 MByte. So I deleted the db file by hand.

Interesting. Sebastian, any ideas about that? Maybe the contents is deleted, 
but the space is not reclaimed.

> Logged-out, then logged-in. nepomuk immediately kicked-in to start
> indexing; I stopped it at about 20 MBytes.
> 
> Started kmail. Was able to send a test message "normally" (meaning hit SEND
> and it's off immediately). Closed kmail.
> 
> Re-started indexer. Got to about 300 MByte in size. Suspended indexing.
> 
> Started kmail. Now a test message was taking a long time as before --
> roughly 45 seconds.

Yep, that's the expected behavior of this bug. With KDE SC 4.4.1, this will be 
fixed: The Strigi index size should not matter anymore for sending speed.

>> Delete file index using script referenced in an above link.
>> What's interesting is that the script takes the file count to zero, but
>> the file size is still > 700 MByte. So I deleted the db file by hand.
>
> Interesting. Sebastian, any ideas about that? Maybe the contents is deleted, 
> but the space is not reclaimed.

There is more in the db than just file metadata: ontologies and contact data indexed by akonadi.

*** Bug 229397 has been marked as a duplicate of this bug. ***

Working OK now in KDE 4.4.1 

Thanks.

*** Bug 231979 has been marked as a duplicate of this bug. ***

*** Bug 225493 has been marked as a duplicate of this bug. ***

*** Bug 233589 has been marked as a duplicate of this bug. ***

*** Bug 233589 has been marked as a duplicate of this bug. ***

SVN commit 1114023 by tmcguire:

Use a ContactPhotoMemento to make the photo loading fully async.

BUG: 219687

 M  +1 -0      CMakeLists.txt  
 A             contactphotomemento.cpp   [License: GPL (v2/3)]
 A             contactphotomemento.h   [License: GPL (v2/3)]
 M  +43 -41    headerstyle.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1114023

Ups, didn't want to close this one yet.

*** Bug 233589 has been marked as a duplicate of this bug. ***

*** Bug 235162 has been marked as a duplicate of this bug. ***

*** Bug 217367 has been marked as a duplicate of this bug. ***

I'm seeing this behaviour on kde 4.4.5 / kmail 1.13.5 
kmail freezes when sending an email exactly as described. Interesnting enough, this behavour didn't happened with 4.4.4 (on my system).
Some info:
gcc (Gentoo 4.4.4-r1 p1.0, pie-0.4.5) 4.4.4
x86-64
Intel(R) Core(TM) i5 CPU 
CFLAGS="-march=native -mtune=native -O2 -pipe"

The following output seems weird to me, but I have to admit that I can't understad exactly ho I can further debug this, so any help or hint will be really appreciated :)

Akonadi Control: running
Akonadi Server: running
Akonadi Server Search Support: available (backend: Unknown)

FYI, we introduced a regression in kdepimlibs of KDE SC 4.5.0 that made sending slower/block the GUI because we had a slow search job in the composer. That is now fixed again with http://websvn.kde.org/?revision=1156791&view=revision, packagers were informed about the patch, so it will hopefully not be a problem. People who have used the development/beta versions/RC of KDE SC 4.5 might have noticed the problem though.

> Akonadi Server Search Support: available (backend: Unknown)

That seems definitely like a problem, although I don't know enough about it to help.

I can add a symptom (kde 4.4.5 kmail 1.13.5)
when kmail freezes in mail sending, it's enough to close firefox and the mail is automagically sent.
Does this rings a bell to someone?

commit 05809931c8f777206be882a495b477154bac5f6c
branch refs/tags/v1.3.0
Author: Volker Krause <vkrause@kde.org>
Date:   Wed Jan 20 08:23:35 2010 +0000

    Backport from trunk.
    
    r1077216 | tmcguire | 2010-01-19 18:57:13 +0100 (Tue, 19 Jan 2010) | 12 lines
    
    Work around a DBus bug that using a connection from within multiple threads can cause freezes.
    In this case, that connection was the session bus, on which we called isServiceRegistered.
    
    The workaround: Just don't check if the search service is available, just call the methods. They'll
    fail properly if the service is not there.
    
    Volker, can you backport this fix to older Akonadi releases and prepare tarballs, so that we can
    notify the release team about this?
    
    Should fix
    CCBUG: 219687
    
    svn path=/branches/akonadi/1.3/; revision=1077436

diff --git a/server/src/nepomuksearch.cpp b/server/src/nepomuksearch.cpp
index 757f96b..8b6f3ed 100644
--- a/server/src/nepomuksearch.cpp
+++ b/server/src/nepomuksearch.cpp
@@ -42,13 +42,9 @@ static qint64 uriToItemId( const QUrl &url )
 NepomukSearch::NepomukSearch( QObject* parent )
   : QObject( parent ), mSearchService( 0 )
 {
-  if ( !Nepomuk::Search::QueryServiceClient::serviceAvailable() ) {
-    qWarning() << "Nepomuk QueryServer interface not available!";
-  } else {
-    mSearchService = new Nepomuk::Search::QueryServiceClient( this );
-    connect( mSearchService, SIGNAL( newEntries( const QList<Nepomuk::Search::Result>& ) ),
-             this, SLOT( hitsAdded( const QList<Nepomuk::Search::Result>& ) ) );
-  }
+  mSearchService = new Nepomuk::Search::QueryServiceClient( this );
+  connect( mSearchService, SIGNAL( newEntries( const QList<Nepomuk::Search::Result>& ) ),
+           this, SLOT( hitsAdded( const QList<Nepomuk::Search::Result>& ) ) );
 }
 
 NepomukSearch::~NepomukSearch()
@@ -66,7 +62,10 @@ QStringList NepomukSearch::search( const QString &query )
     return QStringList();
   }
 
-  mSearchService->blockingQuery( query );
+  if ( !mSearchService->blockingQuery( query ) ) {
+    qWarning() << Q_FUNC_INFO << "Calling blockingQuery() failed!";
+    return QStringList();
+  }
 
   return mMatchingUIDs.toList();
 }

this bug (or a similar one) seems to be still present in kmail 4.4.8. virtuoso-t was using a full CPU core and kmail's composer was hanging during that time - nothing greyed out but unusable. Killing virtuoso-t got kmail to open the "sign" window (GPG signing) and finally send the email. This just happens occasionally...

System: openSUSE 11.3 (x86_64), kernel-desktop 2.6.34.7-0.5.1

> zypper search -is akonadi
S | Name                         | Type    | Version   | Arch   | Repository       
--+------------------------------+---------+-----------+--------+------------------
i | akonadi-runtime              | package | 1.4.1-2.2 | x86_64 | (System Packages)
i | libakonadi4                  | package | 4.5.4-2.5 | x86_64 | (System Packages)
i | libakonadiprotocolinternals1 | package | 1.4.1-2.2 | x86_64 | (System Packages)
> zypper search -is kmail
S | Name  | Type    | Version   | Arch   | Repository       
--+-------+---------+-----------+--------+------------------
i | kmail | package | 4.4.8-4.1 | x86_64 | (System Packages)

> zypper se -is virtuoso
S | Name                     | Type    | Version    | Arch   | Repository                            
--+--------------------------+---------+------------+--------+---------------------------------------
i | soprano-backend-virtuoso | package | 2.5.0-15.2 | x86_64 | (System Packages)                     
i | virtuoso-drivers         | package | 6.1.2-8.1  | x86_64 | openSUSE BuildService - KDE:Release:45
i | virtuoso-server          | package | 6.1.2-8.1  | x86_64 | openSUSE BuildService - KDE:Release:45

With the newer Akonadi drivers it seems it's working for me again.

akonadi-runtime           | 1.4.95-78.1
libakonadi4               | 4.5.95-198.1

soprano-backend-virtuoso  | 2.5.63-94.1
virtuoso                  | 6.1.2-15.1
virtuoso-drivers          | 6.1.2-15.1
virtuoso-server           | 6.1.2-15.1

This bug occured again with KDE 4.6.2 and KMail 1.13.7.

soprano-backend-virtuoso | 2.5.63-97.3
virtuoso                 | 6.1.2-19.1 
virtuoso-drivers         | 6.1.2-19.1 
virtuoso-server          | 6.1.2-19.1 

akonadi-runtime               | 1.5.2-93.1 
libakonadi4                   | 4.6.2-218.1
libakonadiprotocolinternals1  | 1.5.2-93.1

The nepomuk jobs are now async.

Thank you all for the fix!