Bug 276593 - Nepomukservicestub uses too much CPU
Summary: Nepomukservicestub uses too much CPU
Alias: None
Product: nepomuk
Classification: Miscellaneous
Component: general (show other bugs)
Version: unspecified
Platform: Archlinux Linux
: NOR normal
Target Milestone: ---
Assignee: Sebastian Trueg
: 276083 (view as bug list)
Depends on:
Reported: 2011-06-27 08:12 UTC by Hrvoje Senjan
Modified: 2011-07-28 20:42 UTC (History)
10 users (show)

See Also:
Latest Commit:
Version Fixed In:

top (56.82 KB, application/octet-stream)
2011-06-27 08:12 UTC, Hrvoje Senjan
Nepomukstorage gdb (54.61 KB, application/octet-stream)
2011-07-07 16:53 UTC, Hrvoje Senjan
strigiservice gdb (32.03 KB, application/octet-stream)
2011-07-07 16:54 UTC, Hrvoje Senjan
nepomukstorage backtrace (30.51 KB, text/plain)
2011-07-07 17:55 UTC, Alejandro Nova
nepomukstrigiservice backtrace (5.74 KB, application/octet-stream)
2011-07-07 17:56 UTC, Alejandro Nova

Note You need to log in before you can comment on or make changes to this bug.
Description Hrvoje Senjan 2011-06-27 08:12:49 UTC
Created attachment 61360 [details]

Version:           unspecified (using Devel) 
OS:                Linux

Using Arch Linux packages form kde-unstable repo (KDE 4.6.90 aka KDE 4.7 RC1)

Nepomukservicestub and virtuoso-t are using large number of CPU forever. Leaved it all night and they where constantly using ~30-40% of both cores. 

Reproducible: Always

Steps to Reproduce:
Log in into KDE with semantic desktop enabled.

Actual Results:  
Unnecessary long CPU usage cause by nepomuk/virtuoso.

Expected Results:  
They should cool down after some time.

Workaround: kill nepomuk , and start it again - CPU usage is at ~0-1%.

Could be related to https://bugs.kde.org/show_bug.cgi?id=276083

Atached 3 iterations of top
Comment 1 Hrvoje Senjan 2011-06-27 08:18:21 UTC
Qt: 4.7.3
KDE Development Platform: 4.6.90 (4.7 RC1)
Nepomuk Service Stub: 0.2

nepomukserver -v               
Qt: 4.7.3                                                                       
KDEDevelopmentPlatform:4.6.90(4.7RC1)                                             Nepomuk Server: 0.2 

Qt:4.7.3                                                                        KDEDevelopmentPlatform:4.6.90(4.7RC1)                                     
NepomukIndexer: 1.0

VirtuosoOpenSourceEdition(multithreaded)                                        Version as of Apr 12 2011

sopranod 2.6.51
Comment 2 Hrvoje Senjan 2011-06-29 10:17:16 UTC
This is obviously related to strigi search - altough strigi reports it's finished scanning , it still hogs the CPU - turning off strigi within systemsettings , or suspending it "solves" the issue
Comment 3 Dominik Cermak 2011-07-02 16:03:47 UTC
*** This bug has been confirmed by popular vote. ***
Comment 4 Alejandro Nova 2011-07-05 16:02:02 UTC
I was bitten by this one, hard. Some preliminary discoveries, and I'm confirming this too.

1. This bug is restricted to Strigi. Enabling Nepomuk without Strigi indexing yields normal CPU usage, and even Bangarang Nepomuk Writer (a different file indexing tool for Nepomuk) works.
2. I tried to start Strigi on its own via strigidaemon, and it stalls.
3. Strigi isn't indexing. Those five seconds are: Strigi loading, Strigi looping, and Nepomuk indexing your Digikam database (yes, every other NepomukService is working great).
4. Strace gives me this, in a loop. This is causing your CPU usage.

read(7, 0x243f154, 4096)                = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLIN}, {fd=7, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=15, events=POLLIN}], 7, 0) = 0 (Timeout)
read(7, 0x243f154, 4096)                = -1 EAGAIN (Resource temporarily unavailable)
select(16, [15], [15], NULL, {600, 0})  = 1 (out [15], left {599, 999995})
write(15, "\21\0\234\364\264HX\1\0\0select distinct ?g whe"..., 360) = 360
select(16, [15], [], NULL, {600, 0})    = 1 (in [15], left {599, 999706})
ioctl(15, FIONREAD, [13])               = 0
read(15, "\225#Bx\0\0\0\0\0\0\0\0\0", 13) = 13
select(16, [15], [15], NULL, {600, 0})  = 1 (out [15], left {599, 999994})
write(15, "\22\0\225#Bx", 6)            = 6
select(16, [15], [], NULL, {600, 0})    = 1 (in [15], left {599, 999995})
ioctl(15, FIONREAD, [10])               = 0
read(15, "\0\0\0\0\0\0\0\0\0\0", 10)    = 10
select(16, [15], [15], NULL, {600, 0})  = 1 (out [15], left {599, 999995})
write(15, "\26\0\225#Bx", 6)            = 6
select(16, [15], [], NULL, {600, 0})    = 1 (in [15], left {599, 999993})
ioctl(15, FIONREAD, [9])                = 0
read(15, "\0\0\0\0\0\0\0\0\0", 9)       = 9

5. There aren't error messages in nepomukserver console output (apart from a brown-paper bug telling me about how Nepomuk can't find "/nrio.trig", I workarounded that bug symlinking nrio.trig literally into the root folder). So, it may be some config missing for Strigi, Strigi hanging, and Nepomuk trying to restart Strigi, with no results. Those restarts are the source for your CPU usage.
Comment 5 Alejandro Nova 2011-07-05 16:16:25 UTC
6. My config: 

[phoebus@faeris /]$ nepomukindexer -v
Qt: 4.8.0
KDE Development Platform: 4.6.90 (4.7 RC1)
NepomukIndexer: 1.0

[phoebus@faeris /]$ nepomukservicestub -v
Qt: 4.8.0
Plataforma de desarrollo de KDE: 4.6.90 (4.7 RC1)
Matriz de servicio Nepomuk: 0.2

[phoebus@faeris /]$ nepomukserver -v
Qt: 4.8.0
Plataforma de desarrollo de KDE: 4.6.90 (4.7 RC1)
Servidor de Nepomuk: 0.2

Virtuoso is at:
[/usr/bin/nepomukservicestub] "11:38:06 OpenLink Virtuoso Universal Server"
[/usr/bin/nepomukservicestub] "11:38:06 Version 06.01.3127-pthreads for Linux as of Feb 15 2011"
[/usr/bin/nepomukservicestub] Using Virtuoso Version: "" 

I've replaced kde-runtime with today's git 4.7 branch, looking for a solution. Soprano also is today's git.
Comment 6 Alejandro Nova 2011-07-07 04:37:13 UTC
Can you check if nepomukindexer is running? For me, it isn't running.

If it's not running, then the cause of this bug is clear and the fix is quick (nepomukstrigiservice fails to start nepomukindexer, it tries to restart nepomukindexer a lot of times per second, so that causes your CPU usage). A trip to Mandriva 2011 made some things clearer. Is anybody out there?
Comment 7 Alejandro Nova 2011-07-07 04:37:51 UTC
BTW, as the last info bit, I'm on Fedora. This is not restricted to Arch Linux.
Comment 8 Andreas Kuhl 2011-07-07 06:24:20 UTC
@Alexjandro: I guess you're right: nepomukservice strigiindexer is running but not nepomukindexer. 

For me, restarting Nepumukserver does not fix the issue. I use nepomukcontroller to suspend indexing until this is fixed.
Comment 9 Martin Airs 2011-07-07 12:39:53 UTC
I also have this bug, for me the proccess..

nepomukservicestub nepomukstorage

is using lots of cpu, and nepomukindexer isn't running

I am using F15 kde 4.6.90
Comment 10 Alejandro Nova 2011-07-07 14:56:19 UTC
If we have confirmations, then this bug is triaged. I don't have enough privileges to mark this bug as TRIAGED, so, please, may someone with enough privileges change the status to TRIAGED and the description to something like "nepomukindexer doesn't start when called by nepomukstrigiservice, high CPU usage"?
Comment 11 Vishesh Handa 2011-07-07 15:09:59 UTC
staces do not help. Could one of you please provide backtraces of the strigi service, and the storage service?

Steps on how to get a backtrace -

1.) ps aux | grep nepomuk
2.) Note down the PID of "nepomukservicestub nepomukstorage", and "nepomukservicestub nepomukstrigiservice"
3.) Run gdb. Then "attach <PID>", and "thread apply all backtrace"
4.) Do this for both the storage and the strigi service
Comment 12 Hrvoje Senjan 2011-07-07 16:53:37 UTC
Created attachment 61681 [details]
Nepomukstorage gdb
Comment 13 Hrvoje Senjan 2011-07-07 16:54:07 UTC
Created attachment 61682 [details]
strigiservice gdb
Comment 14 Vishesh Handa 2011-07-07 17:28:59 UTC
Thanks, but they are missing debugging symbols.

You'll need to install Soprano, and kde-runtime debugging symbols.
Comment 15 Alejandro Nova 2011-07-07 17:55:49 UTC
Created attachment 61688 [details]
nepomukstorage backtrace

nepomukstorage backtrace, with all debug symbols. Threads 3-16 look identical and triggering a loop, so, that reinforces my conclusion. These backtraces will help you to catch the offending code.

Thanks in advance, Vishesh.
Comment 16 Alejandro Nova 2011-07-07 17:56:22 UTC
Created attachment 61689 [details]
nepomukstrigiservice backtrace

nepomukstrigiservice backtrace with all debug symbols.
Comment 17 Sebastian Trueg 2011-07-08 09:47:32 UTC
Git commit 327ec9a67af653467b670115ebb3e26b00183c1d by Sebastian Trueg.
Committed on 07/07/2011 at 17:33.
Pushed by trueg into branch 'master'.

Throttle the IndexCleaner the same way we do with the IndexScheduler.

BUG: 276593

M  +8    -2    nepomuk/services/strigi/indexcleaner.cpp     
M  +11   -0    nepomuk/services/strigi/indexcleaner.h     
M  +9    -7    nepomuk/services/strigi/indexscheduler.cpp     
M  +1    -3    nepomuk/services/strigi/indexscheduler.h     

Comment 18 Andreas Kuhl 2011-07-08 11:16:57 UTC
Just compiled from trunk and can confirm that this issue seems fixed: Restarting Nepomuk now does cause constant 25% CPU load by virtuoso-t and the indexer remains inactive. A newly created text file with a hash as content is immediatly found by dolphin search.
Comment 19 Andreas Kuhl 2011-07-08 11:17:26 UTC
Thx for your great work to everybody involved! :)
Comment 20 Andreas Kuhl 2011-07-08 11:22:08 UTC
(In reply to comment #18)
> Restarting Nepomuk now does cause constant 25% CPU load

Ah, typo!!! Should have been: "...now does NOT cause..."
Comment 21 Alejandro Nova 2011-07-08 12:50:30 UTC
I'll wait a little for your packages, Rex. I see them coming through Koji :)

Thanks to all, Sebastian, Vishesh, and all people here.
Comment 22 Alejandro Nova 2011-07-10 20:12:36 UTC
More about this.

I tested the fix and this bug is really fixed, nepomukindexer successfully indexes and if I run nepomukindexer in a file, I get a correct result. However, those results are not added to the database, and I can't use Strigi indexing. Should I file a new bug?
Comment 23 Hussam Al-Tayeb 2011-07-19 11:08:53 UTC
(In reply to comment #22)
> More about this.
> I tested the fix and this bug is really fixed, nepomukindexer successfully
> indexes and if I run nepomukindexer in a file, I get a correct result. However,
> those results are not added to the database, and I can't use Strigi indexing.
> Should I file a new bug?

could be relaxed to https://bugs.kde.org/show_bug.cgi?id=278080
I just filed that. strigi cases a nepomuk crash when it tries to index files.
Comment 24 Vishesh Handa 2011-07-28 20:42:41 UTC
*** Bug 276083 has been marked as a duplicate of this bug. ***