Bug 276593

Versions:
nepomukservicestub-v                         
Qt: 4.7.3
KDE Development Platform: 4.6.90 (4.7 RC1)
Nepomuk Service Stub: 0.2


nepomukserver -v               
Qt: 4.7.3                                                                       
KDEDevelopmentPlatform:4.6.90(4.7RC1)                                             Nepomuk Server: 0.2 


nepomukindexer-v              
Qt:4.7.3                                                                        KDEDevelopmentPlatform:4.6.90(4.7RC1)                                     
NepomukIndexer: 1.0



VirtuosoOpenSourceEdition(multithreaded)                                        Version 6.1.3.3127-pthreads as of Apr 12 2011


sopranod 2.6.51

This is obviously related to strigi search - altough strigi reports it's finished scanning , it still hogs the CPU - turning off strigi within systemsettings , or suspending it "solves" the issue

*** This bug has been confirmed by popular vote. ***

I was bitten by this one, hard. Some preliminary discoveries, and I'm confirming this too.

1. This bug is restricted to Strigi. Enabling Nepomuk without Strigi indexing yields normal CPU usage, and even Bangarang Nepomuk Writer (a different file indexing tool for Nepomuk) works.
2. I tried to start Strigi on its own via strigidaemon, and it stalls.
3. Strigi isn't indexing. Those five seconds are: Strigi loading, Strigi looping, and Nepomuk indexing your Digikam database (yes, every other NepomukService is working great).
4. Strace gives me this, in a loop. This is causing your CPU usage.

read(7, 0x243f154, 4096)                = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLIN}, {fd=7, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=15, events=POLLIN}], 7, 0) = 0 (Timeout)
read(7, 0x243f154, 4096)                = -1 EAGAIN (Resource temporarily unavailable)
select(16, [15], [15], NULL, {600, 0})  = 1 (out [15], left {599, 999995})
write(15, "\21\0\234\364\264HX\1\0\0select distinct ?g whe"..., 360) = 360
select(16, [15], [], NULL, {600, 0})    = 1 (in [15], left {599, 999706})
ioctl(15, FIONREAD, [13])               = 0
read(15, "\225#Bx\0\0\0\0\0\0\0\0\0", 13) = 13
select(16, [15], [15], NULL, {600, 0})  = 1 (out [15], left {599, 999994})
write(15, "\22\0\225#Bx", 6)            = 6
select(16, [15], [], NULL, {600, 0})    = 1 (in [15], left {599, 999995})
ioctl(15, FIONREAD, [10])               = 0
read(15, "\0\0\0\0\0\0\0\0\0\0", 10)    = 10
select(16, [15], [15], NULL, {600, 0})  = 1 (out [15], left {599, 999995})
write(15, "\26\0\225#Bx", 6)            = 6
select(16, [15], [], NULL, {600, 0})    = 1 (in [15], left {599, 999993})
ioctl(15, FIONREAD, [9])                = 0
read(15, "\0\0\0\0\0\0\0\0\0", 9)       = 9

5. There aren't error messages in nepomukserver console output (apart from a brown-paper bug telling me about how Nepomuk can't find "/nrio.trig", I workarounded that bug symlinking nrio.trig literally into the root folder). So, it may be some config missing for Strigi, Strigi hanging, and Nepomuk trying to restart Strigi, with no results. Those restarts are the source for your CPU usage.

6. My config: 

[phoebus@faeris /]$ nepomukindexer -v
Qt: 4.8.0
KDE Development Platform: 4.6.90 (4.7 RC1)
NepomukIndexer: 1.0

[phoebus@faeris /]$ nepomukservicestub -v
Qt: 4.8.0
Plataforma de desarrollo de KDE: 4.6.90 (4.7 RC1)
Matriz de servicio Nepomuk: 0.2

[phoebus@faeris /]$ nepomukserver -v
Qt: 4.8.0
Plataforma de desarrollo de KDE: 4.6.90 (4.7 RC1)
Servidor de Nepomuk: 0.2

Virtuoso is at:
[/usr/bin/nepomukservicestub] "11:38:06 OpenLink Virtuoso Universal Server"
[/usr/bin/nepomukservicestub] "11:38:06 Version 06.01.3127-pthreads for Linux as of Feb 15 2011"
[/usr/bin/nepomukservicestub] Using Virtuoso Version: "6.1.2.3127-pthreads" 

I've replaced kde-runtime with today's git 4.7 branch, looking for a solution. Soprano also is today's git.

Can you check if nepomukindexer is running? For me, it isn't running.

If it's not running, then the cause of this bug is clear and the fix is quick (nepomukstrigiservice fails to start nepomukindexer, it tries to restart nepomukindexer a lot of times per second, so that causes your CPU usage). A trip to Mandriva 2011 made some things clearer. Is anybody out there?

BTW, as the last info bit, I'm on Fedora. This is not restricted to Arch Linux.

@Alexjandro: I guess you're right: nepomukservice strigiindexer is running but not nepomukindexer. 

For me, restarting Nepumukserver does not fix the issue. I use nepomukcontroller to suspend indexing until this is fixed.

I also have this bug, for me the proccess..

nepomukservicestub nepomukstorage

is using lots of cpu, and nepomukindexer isn't running

I am using F15 kde 4.6.90

If we have confirmations, then this bug is triaged. I don't have enough privileges to mark this bug as TRIAGED, so, please, may someone with enough privileges change the status to TRIAGED and the description to something like "nepomukindexer doesn't start when called by nepomukstrigiservice, high CPU usage"?

staces do not help. Could one of you please provide backtraces of the strigi service, and the storage service?

Steps on how to get a backtrace -

1.) ps aux | grep nepomuk
2.) Note down the PID of "nepomukservicestub nepomukstorage", and "nepomukservicestub nepomukstrigiservice"
3.) Run gdb. Then "attach <PID>", and "thread apply all backtrace"
4.) Do this for both the storage and the strigi service

Created attachment 61681 [details]
Nepomukstorage gdb

Created attachment 61682 [details]
strigiservice gdb

Thanks, but they are missing debugging symbols.

You'll need to install Soprano, and kde-runtime debugging symbols.

Created attachment 61688 [details]
nepomukstorage backtrace

nepomukstorage backtrace, with all debug symbols. Threads 3-16 look identical and triggering a loop, so, that reinforces my conclusion. These backtraces will help you to catch the offending code.

Thanks in advance, Vishesh.

Created attachment 61689 [details]
nepomukstrigiservice backtrace

nepomukstrigiservice backtrace with all debug symbols.

Git commit 327ec9a67af653467b670115ebb3e26b00183c1d by Sebastian Trueg.
Committed on 07/07/2011 at 17:33.
Pushed by trueg into branch 'master'.

Throttle the IndexCleaner the same way we do with the IndexScheduler.

BUG: 276593

M  +8    -2    nepomuk/services/strigi/indexcleaner.cpp     
M  +11   -0    nepomuk/services/strigi/indexcleaner.h     
M  +9    -7    nepomuk/services/strigi/indexscheduler.cpp     
M  +1    -3    nepomuk/services/strigi/indexscheduler.h     

http://commits.kde.org/kde-runtime/327ec9a67af653467b670115ebb3e26b00183c1d

Just compiled from trunk and can confirm that this issue seems fixed: Restarting Nepomuk now does cause constant 25% CPU load by virtuoso-t and the indexer remains inactive. A newly created text file with a hash as content is immediatly found by dolphin search.

Thx for your great work to everybody involved! :)

(In reply to comment #18)
> Restarting Nepomuk now does cause constant 25% CPU load

Ah, typo!!! Should have been: "...now does NOT cause..."

I'll wait a little for your packages, Rex. I see them coming through Koji :)

Thanks to all, Sebastian, Vishesh, and all people here.

More about this.

I tested the fix and this bug is really fixed, nepomukindexer successfully indexes and if I run nepomukindexer in a file, I get a correct result. However, those results are not added to the database, and I can't use Strigi indexing. Should I file a new bug?

(In reply to comment #22)
> More about this.
> 
> I tested the fix and this bug is really fixed, nepomukindexer successfully
> indexes and if I run nepomukindexer in a file, I get a correct result. However,
> those results are not added to the database, and I can't use Strigi indexing.
> Should I file a new bug?

could be relaxed to https://bugs.kde.org/show_bug.cgi?id=278080
I just filed that. strigi cases a nepomuk crash when it tries to index files.

*** Bug 276083 has been marked as a duplicate of this bug. ***