Bug 324307

In my case, originally nepomuk was bombarding systemd-journal but now it isn't running at all. It is checked-in in desktop search settings. When I start nepomukcleaner nothing happens. Dolphin or KDE menu also don't show filters and recent items respectively.

Yeah, just telling us it crashes for you is not very helpful. We need a bit more information here. On what file is it segfaulting? Can you get a backtrace? Maybe you could even send us one of these core dumps you have been gathering? 

Also what link? You didn't provide one.

Is there a wiki or anything I can use to provide you better data?

This page is useful in general:
http://techbase.kde.org/Development/Tutorials/Debugging/How_to_create_useful_crash_reports

For bugs like this one the best thing you can do if to attach a copy of the file nepomukindexer crashes on to the bug report. You can find this by looking at the process list (press ctrl+esc). There will be a process with the command "nepomukindexer FILENAME". Attach that filename to this report.

If you are using arch like the original reporter, there are no debug symbols, so to get a useful backtrace/core dump you have to recompile using abs. To do this, in /var/abs/extra/nepomuk-core/PKGBUILD, change:     "-DCMAKE_BUILD_TYPE=Release \" to "-DCMAKE_BUILD_TYPE=Debug \" (note the backslash!) and add a line with options=(!strip) near the top.

In your case, Sudhir, I think nepomuk has crashed somehow. Try disabling it in systemsettings, hitting apply, and re-enabling it. If that doesn't work, try removing your database (WARNING! this will delete your tags) with 

rm -R $HOME/.kde4/share/apps/nepomuk/repository/main/data/virtuosobackend

and follow the above instructions.

Also, helpful bug reports are really appreciated - they're the best way we have to improve the software. If you can send me the file it crashes on, I can probably fix this pretty quickly, so good luck!

We also already fixed crashes in the epub extractor, the office 2007 extractor and the taglib (ie, music files) extractor since 4.11, so it is possible the problem will go away in 4.11.1

(In reply to comment #2)
> Yeah, just telling us it crashes for you is not very helpful. We need a bit
> more information here. On what file is it segfaulting? Can you get a
> backtrace? Maybe you could even send us one of these core dumps you have
> been gathering? 
> 
> Also what link? You didn't provide one.

The link is right there, in the url section of the bug report: https://bbs.archlinux.org/viewtopic.php?pid=1319145

Besides, other users are having the exact same issue, so I didn't think it was due to a specific file of mine.

Finally, I am unable to reproduce at the moment. Strange, because I had this for days.
Maybe the file that was causing this was deleted. I'll add more info and hopefully a useful trace when I can reproduce this.

> Besides, other users are having the exact same issue, so I didn't think it
> was due to a specific file of mine.

Probably because they have a similar type of file - nepomukindexer is the process that does the actual indexing.

> Finally, I am unable to reproduce at the moment. Strange, because I had this
> for days.
> Maybe the file that was causing this was deleted. I'll add more info and
> hopefully a useful trace when I can reproduce this.

Probably. It's most likely an epub or an office document, since those are the new indexers, but it could also be webm or flac.

> > Finally, I am unable to reproduce at the moment. Strange, because I had this
> > for days.

If you want to check for sure, move your db and get it to reindex everything, then see if it crashes.

Here are some messages and addresses if it helps.



013-09-02T14:49:12.145135-07:00 poulsbo kernel: [  170.549546] nepomukindexer[2376]: segfault at 8 ip 00007f622a6cb261 sp 00007fff8490c598 error 4 in libc-2.17.so[7f622a647000+1a3000]
2013-09-02T14:49:15.498135-07:00 poulsbo kernel: [  173.902521] nepomukindexer[2381]: segfault at 560 ip 00007f2aa5a6096e sp 00007fffda4fda10 error 4 in libQtGui.so.4.8.5[7f2aa564a000+a6b000]
2013-09-02T14:49:18.774151-07:00 poulsbo kernel: [  177.178976] nepomukindexer[2385]: segfault at 8 ip 00007f020d74b261 sp 00007fff5d80ff68 error 4 in libc-2.17.so[7f020d6c7000+1a3000]
2013-09-02T14:49:22.043141-07:00 poulsbo kernel: [  180.447747] nepomukindexer[2389]: segfault at 8 ip 00007f913b3ba261 sp 00007fff1b325838 error 4 in libc-2.17.so[7f913b336000+1a3000]
2013-09-02T14:49:25.238146-07:00 poulsbo kernel: [  183.642523] nepomukindexer[2393]: segfault at 560 ip 00007f0910fc696e sp 00007fffd787f0f0 error 4 in libQtGui.so.4.8.5[7f0910bb0000+a6b000]
2013-09-02T14:49:28.583244-07:00 poulsbo kernel: [  186.987844] nepomukindexer[2397]: segfault at 560 ip 00007f23523da96e sp 00007fff773128f0 error 4 in libQtGui.so.4.8.5[7f2351fc4000+a6b000]
2013-09-02T14:49:31.902151-07:00 poulsbo kernel: [  190.306155] nepomukindexer[2401]: segfault at 8 ip 00007f762834d261 sp 00007fffd315fb38 error 4 in libc-2.17.so[7f76282c9000+1a3000]
2013-09-02T14:49:38.630137-07:00 poulsbo kernel: [  197.034574] nepomukindexer[2409]: segfault at 560 ip 00007f71d23b096e sp 00007fffe679c3a0 error 4 in libQtGui.so.4.8.5[7f71d1f9a000+a6b000]
2013-09-02T14:49:41.781137-07:00 poulsbo kernel: [  200.185714] nepomukindexer[2413]: segfault at 8 ip 00007f68c09cf261 sp 00007fff59090958 error 4 in libc-2.17.so[7f68c094b000+1a3000]
2013-09-02T14:49:44.949141-07:00 poulsbo kernel: [  203.353580] nepomukindexer[2418]: segfault at 8 ip 00007f6a26846261 sp 00007fff0ccd3d98 error 4 in libc-2.17.so[7f6a267c2000+1a3000]

I was able to get Dolphin filters back by removing virtuosobackend and it worked fine for a few days and today the bug spawned again. I haven't had time to go over the wiki posted by Simeon. As soon as I have time later this week I will post as much info as I can gather. I currently have 1.8GB of /var/log/journal.

Don't worry about the wiki page - just try and find the file nepomukindexer is crashing on and attach it to the bug report. You can do this by looking at the output of

"ps -ef | grep nepomuk"

If you want to stop the journal going berserk, you can turn off core dump collection by following the instructions here:

http://mailman.archlinux.org/pipermail/arch-general/2012-December/032375.html

Sorry, but "ps -ef | grep nepomuk" gives not a single clue as to what file it is crashing on.
What did you expect to see hare?

nepomukindexer is respawning every 8 seconds
, only to segfault again.

n(In reply to comment #12)
> Sorry, but "ps -ef | grep nepomuk" gives not a single clue as to what file
> it is crashing on.
> What did you expect to see hare?

The indexing service (nepomukfileindexer) makes a list of not-yet-indexed files and then calls "nepomukindexer $file". I want a copy of the $file the nepomukindexer is crashing on, so I want the command line arguments of nepomukindexer.

If ps is not working, could you perhaps try replacing /usr/bin/nepomukindexer  with an (executable) script like:

#!/usr/bin/bash

echo $@ >> some-file-you-can-write-to

then some-file-you-can-write-to should contain the file it crashes on.

(but keep the old /usr/bin/nepomukindexer around somewhere)

> If ps is not working, could you perhaps try replacing
> /usr/bin/nepomukindexer  with an (executable) script like:
> 
> #!/usr/bin/bash
> 
> echo $@ >> some-file-you-can-write-to
> 
Better make that:

#!/bin/bash

echo $@ >> logfile

ps -ef | grep nepomuk
donnie     759   671  0 19:54 ?        00:00:00 kdeinit4: nepomukserver [kdeinit]              
donnie     762   759  0 19:54 ?        00:00:00 /usr/bin/nepomukstorage
donnie    1783   727  2 19:54 ?        00:00:05 /usr/bin/akonadi_nepomuk_feeder --identifier akonadi_nepomuk_feeder
donnie    3812   759  0 19:55 ?        00:00:00 /usr/bin/nepomukfileindexer
donnie    3813   759  0 19:55 ?        00:00:00 /usr/bin/nepomukfilewatch
donnie   12348  3184  0 19:58 pts/1    00:00:00 grep nepomuk

Some htop screenshots.
http://i.imgur.com/6hVFWRE.png
http://i.imgur.com/y90w9ie.png

So I have installed nepomuk-core with following settings in PKGBUILD

options=(debug !strip)

and

-DCMAKE_BUILD_TYPE=Debug \

Now I am not sure how to get the actual backtraces.

gdb nepomukindexer
(gdb) run
Starting program: /usr/bin/nepomukindexer 
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Must input url of the file to be indexed[Inferior 1 (process 28307) exited with code 01]

Ok, they are ebooks

poulsbo:~ # cd /usr/bin/
poulsbo:/usr/bin # cp nepomukindexer keep-nepomukindexer 
poulsbo:/usr/bin # pico fake-nepomukindexer 
poulsbo:/usr/bin # chmod 755 fake-nepomukindexer            
poulsbo:/usr/bin # cp fake-nepomukindexer nepomukindexer 

enable nepomuk in KDE settings:




jsa@poulsbo:~> tailf logfile
/home/jsa/Documents/eBooks/Floyd Gibbons/''And they thought we wouldn't fight'' (300)/''And they thought we wouldn't fight'' - Floyd Gibbons.epub
/home/jsa/Dropbox/eBooks-Backup-Library/William Andrew Johnston/The Apartment Next Door (297)/The Apartment Next Door - William Andrew Johnston.epub
/home/jsa/Dropbox/eBooks-Backup-Library/Tim Dorsey/We Came to Observe You (530)/We Came to Observe You - Tim Dorsey.epub
/home/jsa/Dropbox/eBooks-Backup-Library/Frank Richard Stockton/The late Mrs. Null (329)/The late Mrs. Null - Frank Richard Stockton.epub
/home/jsa/Dropbox/eBooks-Backup-Library/Rene Gutteridge/Listen (521)/Listen - Rene Gutteridge.epub
/home/jsa/Dropbox/eBooks-Backup-Library/Baron Spiegel von Und Zu Peckelsheim/The Adventures of the U-202 Submarine_ A (453)/The Adventures of the U-202 Submarine_ A - Baron Spiegel von Und Zu Peckelsheim.epub


It goes on like that for some length of time... several hundred ebooks...
I let it run its course. It started repeating on exactly 11 ebooks, none of the other several hundred ebooks and no other files.

Email me if you need one of these books.  I have several that are public domain in that list.

It turns out there are in fact three separate nepomuk bugs at work here.

First, there is a straightforward crash in the epub indexer

Second, there was a bug where files without metadata weren't being pushed correctly.

Third, I actually implemented for 4.11 a feature whereby files which crashed the indexer would not be indexed again, but it turns out this feature did not work at all - for three entirely separate reasons, no less.

So, thank you for your patience everyone. I have fixed the bugs I found, and sent the non-trivial parts to Vishesh for review. Unfortunately we've missed the 4.11.1 release date, so these will have to wait until 4.11.2.

Thank you very much.
Disabling indexing for now, and I'll wait for the dot releases.

Epubs worked until yesterday. At least most of them did. 
I also had a mountain of source code indexed, but I can't seem to search for content in that either since the last time I applies opensuse updates late yesterday.

(In reply to comment #20)
> Epubs worked until yesterday. At least most of them did. 
> I also had a mountain of source code indexed, but I can't seem to search for
> content in that either since the last time I applies opensuse updates late
> yesterday.

This would be 4.11 release? Everything gets re-indexed with it (more reliably) - source code is not now indexed by default, you have to turn on the "index source code" option in the indexing preferences.

Git commit 37f5c9df266e93e4257dfd4c39bdcb3061a837a1 by Simeon Bird.
Committed on 03/09/2013 at 04:08.
Pushed by sbird into branch 'KDE/4.11'.

epubextractor: Fix crash on some invalid files where epub_get_data
returns an error.

M  +3    -2    services/fileindexer/indexer/epubextractor.cpp

http://commits.kde.org/nepomuk-core/37f5c9df266e93e4257dfd4c39bdcb3061a837a1

Git commit 9a4dcb1bffaef6c70a256fb3c2a759e6586e0da5 by Simeon Bird.
Committed on 03/09/2013 at 04:43.
Pushed by sbird into branch 'KDE/4.11'.

Indexer and epubextractor: check more carefully that graphs are
not empty.

In particular, in the indexer we check that the graph is
non-empty, but then we remove the plain text content, so we need to
check again.
REVIEW: 112476

M  +2    -1    services/fileindexer/indexer/epubextractor.cpp
M  +19   -12   services/fileindexer/indexer/indexer.cpp

http://commits.kde.org/nepomuk-core/9a4dcb1bffaef6c70a256fb3c2a759e6586e0da5

Git commit d14700f5a02653260aa1129eb718f69d33e5c8a5 by Simeon Bird.
Committed on 03/09/2013 at 06:05.
Pushed by sbird into branch 'KDE/4.11'.

FileIndexer: Really do not index file multiple times

In commit c441198286562924ad3980527913b4888bdcb7a6
I attempted to implement not re-indexing files which failed to index.

However, my code made no sense!

It attempted to set kext::indexingLevel to -1 if the nepomukindexer job
returned with an error. But, the KJob never set an error even if the
indexer returned 1, so nothing happened.

Even when this was fixed, it still didn't work, because I was updating
the indexingLevel of the current filename, not the resource URI of the
current filename.

So, to summarise, I was silly. This one works, and an easy way to test
it is to replace /usr/bin/nepomukindexer with:

 #!/bin/bash
 exit 1

M  +11   -6    services/fileindexer/fileindexingjob.cpp
M  +11   -3    services/fileindexer/fileindexingqueue.cpp

http://commits.kde.org/nepomuk-core/d14700f5a02653260aa1129eb718f69d33e5c8a5

Git commit 6e6e676bc519eae62d66a6b69c1a83788c7a570a by Simeon Bird.
Committed on 03/09/2013 at 06:29.
Pushed by sbird into branch 'KDE/4.11'.

FileIndexer: Check for nepomukindexer crashing.

It is not enough to check that the indexer returns 0.
We must also check that it did not crash, and, if it did,
return an error.

This patch also defines an error enum for FileIndexingJob.
REVIEW: 112475

M  +8    -4    services/fileindexer/fileindexingjob.cpp
M  +11   -2    services/fileindexer/fileindexingjob.h

http://commits.kde.org/nepomuk-core/6e6e676bc519eae62d66a6b69c1a83788c7a570a