Version: unspecified (using Devel)
As long as I test, some specific mp3 file on my system will cause this problem. I attach the test file for test.
Steps to Reproduce:
1. Create a folder
2. Place attached mp3 in it
3. Make nepomuk only index it (for quick)
Index this file without problem.
Nepomukindexer will take 100% cpu and never ends.
Created attachment 60371 [details]
Bug mp3 file.
Actually It works will in 4.6.3.
I can confirm this bug on Arch Linux 64 bit running KDE 4.7.1. Indexing mp3 files causes nepomukindexer at random mp3 files to spike the cpu. Looking at top I see 3 or more etries all of a sudden of nepomukindexer taking 99 or 100 % cpu power.
The same also happens on my Kubuntu installation, KDE SC 4.7.1, 64bit
The buggy file in #2 still trigger this.
The backtrace while I attach the nepomukindexer.
#0 0x00007f78866f2b17 in access () from /lib/libc.so.6
#1 0x00007f78874a74b1 in KStandardDirs::exists (fullPath=...)
#2 0x00007f78874a77d8 in KStandardDirs::realPath (dirname=<value optimized out>)
#3 0x00007f78874ac5c0 in KStandardDirs::KStandardDirsPrivate::resourceDirs (this=0x246b8c0, type=0x7f78875a6309 "xdgdata-mime",
subdirForRestrictions=...) at /chakra/desktop-testing/kdelibs/src/kdelibs-4.7.1/kdecore/kernel/kstandarddirs.cpp:1165
#4 0x00007f78874ae44c in KStandardDirs::findAllResources (this=<value optimized out>, type=0x7f78875a6309 "xdgdata-mime", filter=..., options=...,
relList=...) at /chakra/desktop-testing/kdelibs/src/kdelibs-4.7.1/kdecore/kernel/kstandarddirs.cpp:872
#5 0x00007f78874ae682 in KStandardDirs::findAllResources (this=<value optimized out>, type=<value optimized out>, filter=<value optimized out>,
options=<value optimized out>) at /chakra/desktop-testing/kdelibs/src/kdelibs-4.7.1/kdecore/kernel/kstandarddirs.cpp:897
#6 0x00007f78874c6f57 in KMimeTypeRepository::checkMimeTypes (this=<value optimized out>)
#7 0x00007f78874c71c6 in KMimeTypeRepository::checkEssentialMimeTypes (this=0x25f0aa0)
#8 0x00007f78874c262e in KMimeType::findByUrlHelper (_url=..., mode=0, is_local_file=true, device=0x7fff347977b0, accuracy=0x0)
#9 0x00007f78874c370b in KMimeType::findByUrl (url=..., mode=0, is_local_file=<value optimized out>, fast_mode=<value optimized out>,
accuracy=<value optimized out>) at /chakra/desktop-testing/kdelibs/src/kdelibs-4.7.1/kdecore/services/kmimetype.cpp:323
#10 0x00000000004090cc in _start ()
Cannot reproduce with libstreamanalyzer (strigi) 0.7.6. Please make sure to have an up-to-date version since it is the actual problem.
Just tried this with strigi 0.7.2 on Arch Linux 64 bit with KDE 4.7.2 and this is NOT fixed. Nepomukindexer still hangs at random mp3 files but the cpu doesn't spike as high as before (with 0.7.5).
I came from chakra and the strigi check out from git on 20110925.
May try debug later.
Any progress on this because this isn't resolved.
This issue still exists in KDE 4.7.2 with strigi 0.7.6. I'm using 64bit Arch Linux.
I am working on it. Need to get some other trouble out of the way first though... please be a little more patient... I know you already have done that for quite a while... :)
Ok, this was simpler than I thought: works perfectly here with the upcoming 4.7.3 and strigi trunk (upcoming 0.7.7). Since we only have 3 more days until 4.7.3 lets hope that the release will fix it for you, too.
OK, will check for myself when KDE 4.7.3 and strigi 0.7.7 hit the Arch Linux repos.
Just grab the strigi git code and compiles, seems problem still exists.
But as the previous backtrace indicates, I'm not sure this is a strigi problem or kdelibs/kde-runtime problem.
Would wait for 4.7.3.
Just upgrade to KDE 4.7.3 Chakra... no luck.
And strigi from anongit.kde.org (is this the correct place?)
Try debug a lot... seems after strigi index, the memory is already mess up.. like toLocal8bit cannot work use the correct codec. Though valgrind doesn't show something interesting.
I am not sure what to do here now since I cannot reproduce the problem.... :/
emm.. is it correct to get the strigi code from anongit.kde.org?
(In reply to comment #16)
> emm.. is it correct to get the strigi code from anongit.kde.org?
Yes, but be aware that the strigi repo is just a wrapper for a set of submodules. So unless you fetch those you do not have the updated libs. I recommend to instead clone the "libstreams" and "libstreamanalyzer" repos.
Still exists in KDE 4.7.3 in Arch Linux. Strigi is still 0.7.6 though. It seems like every mp3 that hangs the indexer is encoded as variable bit rate, have ID3v1.1 and ID3v2.4.0 tags and are Joint Stereo. Files with ID3v2.3.0 tags are indexed correctly.
`file` says "Audio file with ID3 version 2.4.0, contains: MPEG ADTS, layer III, v1, XXX kbps, 44.1 kHz, JntStereo" for each file.
Every file seems to have the ID3v2 number of tracks in the album field set.
Converting the ID3v2.4.0 tag to 2.3.0 using Kid3 solves the problem and the files stop hanging the indexer.
I am fairly certain this will be fixed with strigi 0.7.7.
No luck, just grab strigi today's git code, with KDE 4.8 beta 1.
If I follow #19 comment, no more hang in nepomukindexer, but with this error ..
nepomukindexer(5731)/nepomuk (strigi service) Nepomuk::StrigiIndexWriter::finishAnalysis: "Cannot set values for abstract property 'http://www.semanticdesktop.org/ontologies/2009/02/19/nmm#setSize'."
(In reply to comment #21)
> No luck, just grab strigi today's git code, with KDE 4.8 beta 1.
> If I follow #19 comment, no more hang in nepomukindexer, but with this error ..
> nepomukindexer(5731)/nepomuk (strigi service)
> Nepomuk::StrigiIndexWriter::finishAnalysis: "Cannot set values for abstract
> property 'http://www.semanticdesktop.org/ontologies/2009/02/19/nmm#setSize'."
Please update shared-desktop-ontologies to 0.8.0.
But still the problem exists, the ID3v2 2.4 index is break.
(In reply to comment #23)
> But still the problem exists, the ID3v2 2.4 index is break.
You have updated sdo now?
Did you install it locally or globally?
Well, the original problem of this bug report is .. the attached mp3 file cannot be indexed (nepomukindexer hang and take one core).
#19 suggest to convert id3v2 2.4 tag to 2.3, so if I convert to id3v2 2.3, the hang problem solved, but that didn't resolve the original problem: nepomukindexer hang with id3v2 2.4....
No matter I update s-d-o or not (actually I update it, the hang problem is still there), that's nothing to do with v2.4 tag.
Seems that I did speak prematurely: actually indexing did not work here either. However, it did not hang but simply terminated with an error. This led to a fix in shared-desktop-ontologies. I just released 0.8.1.
I root down the cause for hang for me.
After a call to UTF8Converter with iconv, the QTextCodec::codecForLocale cannot work correctly.
But all the call to iconv seems to use correctly, I wonder is it a glibc bug or qt bug...
I have glibc 2.14.1 here.
This is my test case (already strigi free).
The last qDebug() << 111111; will output wrong (also for QString)
Which part should I report the bug?.... Qt / glibc ...
(In reply to comment #28)
> This is my test case (already strigi free).
> The last qDebug() << 111111; will output wrong (also for QString)
> Which part should I report the bug?.... Qt / glibc ...
To be honest I do not know.
Created attachment 66082 [details]
A walkaround for this bug
Don't know why, but iconv UTF-16 will cause this bug, at least for Chakra and Archlinux (they both use glibc 2.14.1), not tested on other distribution, anyway we can detect the endian by hand, and seems it can fix this problem.
Ok, the reason is also found. iconv_open with glibc 2.14 has a bug (maybe not a bug, but behaviour is different from glibc 2.13) that will remember the endianness even another iconv_open is used. So this walkaround is required for all 2.14+ glibc.
I put a upstream report here.
This is the error I get with s-d-o 0.8.1 (with upper patched git strigi).
Nepomuk::StrigiIndexWriter::finishAnalysis: "<http://www.semanticdesktop.org/ontologies/2009/02/19/nmm#setNumber> has a rdfs:domain of <http://www.semanticdesktop.org/ontologies/2009/02/19/nmm#MusicPiece>. <_:b> only has the following types <http://www.semanticdesktop.org/ontologies/2009/02/19/nmm#MusicAlbum>, <http://www.w3.org/2000/01/rdf-schema#Resource>"
Ok, #32 problem is resolved with git libstreamanalzyer, just noticed.
Not very familiar with git submodule...
I just installed strigi 0.7.7 on my Arch 64 machine and I have shared-desktop-ontologies 0.8.1 and my mp3 files still hang...
(In reply to comment #34)
> I just installed strigi 0.7.7 on my Arch 64 machine and I have
> shared-desktop-ontologies 0.8.1 and my mp3 files still hang...
glibc 2.15 will hopefully resolve this problem.
Awesome debugging and you beat the Drepper, Weng!