Bug 163050

I can reproduce it (checked with tgz files).

If moving the mouse over another file before the freeze ends, I could make dolphin crash. (the bt seems to indicate it's a strigi crash though)

Confirmed today using r815560.

I can only confirm if I have the information panel activated (r815688). Otherwise it's perfectly fine.

@Christophe, Strigi could really be the issue, i also opened this bug http://bugs.kde.org/show_bug.cgi?id=163102 which is a crash caused by strigi, or so it seems by the backtrace (information panel activated only).

I checked the code in Dolphin and no changes have been done in this regard since KDE 4.0, where this issue does not occur. The following code is invoked by Dolphin when the Information Panel is open:

const KFileMetaInfo::WhatFlags flags = KFileMetaInfo::Fastest | KFileMetaInfo::TechnicalInfo | KFileMetaInfo::ContentInfo;
const KFileMetaInfo fileMetaInfo(path, QString(), flags);

When hovering large files (e. g. compressed files or video files) the call of KFileMetaInfo constructor might take several seconds or minutes, although KFileMetaInfo::Fastest is used.

The issue seems not be related to Strigi or Nepomuk (turning off both of them still leads to a blocking behavior).

*** Bug 163102 has been marked as a duplicate of this bug. ***

As I said in the other bug, the time of the freeze id proportional to the size of the video: the bigger the longer freeze (about 35 sec for a 700mb video).

Also, this happens when hovering /dev/zero

*** Bug 162998 has been marked as a duplicate of this bug. ***

Seems to be a Strigi bug (I think Strigi is still used for KFileMetaInfo even if you do not use it for disk indexing).

I run Dolphin in callgrind while getting info on /dev/zero and it looks like the problem is in src/streamanalyzer/streamanalyzer.cpp.

Namely near the bottom of the analyze() function:

        do {
            if (!idx.config().indexMore()) {
                removeIndexable(idx.depth());
                return -1;
            }
            ready = input->size() != -1;
            vector<StreamThroughAnalyzer*>::iterator ts;
            for (ts = tIter->begin(); ready && ts != tIter->end(); ++ts) {
                ready = (*ts)->isReadyWithStream();
            }
            if (!ready) {
                input->skip(skipsize);
                if (skipsize < 131072) {
                    skipsize *= 4;
                }
            }
        } while (!ready && input->status() == Ok);

This loop runs forever.  Presumably /dev/zero is always ready to read, with an OK status since it cannot EOF.

This probably also causes the slowdown on the huge .avi files as strigi reads *the entire file* for some reason (even according to the comments).  I'm not exactly sure why the code makes it to this case however.

The last commit involving the analyze function doesn't seem to have caused it.  I haven't checked the latest commit yet (something about enabling more analyzers).

P.S. does strigi have a entry in bugs.kde.org?  I can't seem to find a way to reassign this to strigi.  We do have a nepomuk but with no strigi subcategory.

I suspect based on the timing that it is commit 814603 which enabled a lot of stream analyzers which is making the bug apparent.  Even reverting that however, /dev/zero causes this bug as does pretty much any large movie that I try.

It would be helpful to see how 'xmlindexer' performs on the same file.
There certainly is no need for dolphin to do a very thorough analysis of big files, so we need to adapt the analyzer configuration to limit the analysis and make sure the analyzers involved respect the setting.

Currently the default configuration is used in predicateproperties.cpp:117.
For performance one could do:
class ShallowAnalysisConfiguration : public Strigi::AnalyzerConfiguration {
    /** Limit the amount of data we will read from a stream.
        This is a suggestion to analyzers which they should follow.
        Only index real files. We do not look at subfiles.
        This setting is needed because by default all subfiles are examined.
    **/
    int64_t maximalStreamReadLength(const Strigi::AnalysisResult& ar) {
        // 64k should be enough
        return (ar.depth() == 0) ?65536 :0;
    }
};

That's nice for Dolphin (well KMetaFileInfo really, it can use the "fastest" hint flag to setup shallow analysis), but what happens if Strigi tries to index a file that's not a file i.e. /dev/zero?

An infinite loop isn't a great idea for something in kdelibs IMHO.  Does Strigi have code to determine if it's reading from a regular file or a device, network socket, etc?  I think that would be an easy way to skip files like that in case someone ends up trying to get meta info in the /dev directory.  The shallow analysis presumably would take care of the large movies and such.

For Strigi, the problem of files like /dev/null has been solved in fileinputstream.cpp:66:
    // determine file size. if the stream is not seekable, the size will be -1
    if (fseeko(file, 0, SEEK_END) == -1) {
        m_size = -1;
    } else {
        m_size = ftello(file);
        fseeko(file, 0, SEEK_SET);
        // if the file has size 0, make sure that it's really empty
        // this is useful for filesystems like /proc that report files as size 0
        // for files that do contain content
        if (m_size == 0) {
            char dummy[1];
            size_t n = fread(dummy, 1, 1, file);
            if (n == 1) {
                m_size = -1;
                fseeko(file, 0, SEEK_SET);
            } else {
                fclose(file);
                file = 0;
                return;
            }
        }
    }

In kdelibs, a stream is created from a QIODevice. We need a way to tell if the size of the stream is really finite. /dev/null is size 0, but you can keep reading from it. If the stream is an infinite stream, like /dev/null is, but reports that is size zero, the underlying QIODevice should be closed and the BufferedStream should make 0 bytes available.

The relevant code in kdelibs is kfilemetainfo.cpp:51.

SVN commit 817857 by mpyne:

Don't allow KFileMetaInfo to try an analyze non-sequential devices.  This fixes stuff like /dev/zero,
/dev/random (character devices), pipes, and other weird non-regular files.

This helps but does not fix bug 163050

CCBUG:163050


 M  +13 -1     kfilemetainfo.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=817857

SVN commit 817893 by mpyne:

Backport fix keeping non-sequential files from being analyzed by KFileMetaInfo to 4.0.

CCBUG:163050


 M  +11 -1     kfilemetainfo.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=817893

*** Bug 163570 has been marked as a duplicate of this bug. ***

*** Bug 163838 has been marked as a duplicate of this bug. ***

Bug still present on 4.0.82(relased on 11/06)

loading writer for key  "http://freedesktop.org/standards/xesam/1.0/core#mediaDuration"
loading writer for key  "http://freedesktop.org/standards/xesam/1.0/core#verticalResolution"
loading writer for key  "http://freedesktop.org/standards/xesam/1.0/core#sourceModified"
loading writer for key  "http://freedesktop.org/standards/xesam/1.0/core#size"
loading writer for key  "http://freedesktop.org/standards/xesam/1.0/core#frameRate"
loading writer for key  "http://freedesktop.org/standards/xesam/1.0/core#audioCodec"
loading writer for key  "http://freedesktop.org/standards/xesam/1.0/core#name"
loading writer for key  "http://freedesktop.org/standards/xesam/1.0/core#videoCodec"
loading writer for key  "http://freedesktop.org/standards/xesam/1.0/core#url"
loading writer for key  "http://strigi.sf.net/ontologies/0.9#parentUrl"
loading writer for key  "http://www.w3.org/1999/02/22-rdf-syntax-ns#type"
loading writer for key  "http://strigi.sf.net/ontologies/0.9#depth"
loading writer for key  "http://freedesktop.org/standards/xesam/1.0/core#fileExtension"
resetting is impossible!! pos: 8202 status: 0

I was going to take a look and fix this issue when saw Jos' commit 2 days ago (http://websvn.kde.org/trunk/kdesupport/strigi/src/streamanalyzer/streamanalyzer.cpp?r1=817382&r2=817855). This should fix the bug. Please retry with trunk (NOTE !: you need to update kdesupport, and use the strigi from kdesupport). I have not enough big files to check it out here now. But that fix is what I also had in mind, and it should fix the issue. Probably he forgot to close this bug as fixed.

It crashes for me on some files (Jos already knows about this however) but when it does work it goes very quickly now, closing the bug.

The crashing is a separate bug:
http://bugs.kde.org/show_bug.cgi?id=163838
To fix that I must be able to reproduce it, which I've not managed so far.

> To fix that I must be able to reproduce it, which I've not managed so far.

Bug #163838 crashs on an AVI file, I could reproduce a crash with the same backtrace when hovering a large .tar.gz file (compressed directory [400 MB - mostly e-mails]: tar -czvf peterDotKde.tgz /home/peter/.kde)

Is this really solved? I still have 7 second freezes with 230 MiB .ogm files that only happen when the information tab is activated in trunk rev. 820931.

I don't know much, but As far as I understand, you must use strigi from kde subversion repository.
I use gentoo and strigi is installed by the distro packages, so the bug is still there in my system.

If the bug is still there, please look what's taking up the cpu with 'sysprof'.

Created attachment 25369 [details]
sysprof dolphin profile 

I'm no expert with sysprof but it seems the function QIODevice::read(char*,long
long) is the worst offender with 16%.

I really wonder if for those that this problem is still happening you updated kdesupport, or not. That is vital for determining if the problem is still happening or not.

Right, I forgot to mention that. I didn't install strigi from arch's repos, but from kdesupport.

svn info @ kdesupport : Revision: 820930
svn info @ KDE/kdebase/apps/dolphin : Revision: 820931

At this very moment, kdesupport is 820853...

Updated kdesupport some minutes ago and Dolphin hangs for a few seconds when hovering http://www.file-factory.com/old_de/downloads/aKademy2006-Plasma.avi if the information panel is open (KFileMetaInfo is used).

Sorry, I forgot to mention in comment #30: it's important to test this when having a cold disk cache. After hovering the file once it blocks only around ~ 0,5 second (with cold disk cache: ~2 - 3 seconds).

SVN commit 821198 by vandenoever:

Additional fix that limits the amount of data being read while doing a quick analysis of a file. As discussed on bugs.kde.org.

CCBUG:163050 



 M  +14 -1     predicateproperties.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=821198

Good job, now it doesn't hang and seems to work as it should, although the crashes on certain large files continue to exist but that's a different bug.

*** Bug 164981 has been marked as a duplicate of this bug. ***

same bug here, but strigi is disabled, so it can't be it

> same bug here, but strigi is disabled, so it can't be it 

AFAIK disabling strigi only disables the indexing functionality. Strigi still is used for getting the meta data in this case (@Jos: please correct me if I'm wrong).

still crash here ? what is fix I can't see?

*** Bug 165030 has been marked as a duplicate of this bug. ***

*** Bug 165084 has been marked as a duplicate of this bug. ***

I've updated from SVN yesterday and this still happens :S As before, if I hover over a "large" video file (700mb) dolphin freezes for about 30 seconds.

This bug is not fixed yet (about 6 hours old build of kdesupport, 20 hours of the rest).

It is still crashing on every single avi file I have. And it does't read a lot of data until it crashes. (when selecting one of them in dolphin with Info-Sidebar enabled).

Here the backtrace:

KCrash handler]
#6  0xb80cb424 in __kernel_vsyscall ()
#7  0xb662e740 in raise () from /lib/libc.so.6
#8  0xb6630078 in abort () from /lib/libc.so.6
#9  0xb662768e in __assert_fail () from /lib/libc.so.6
#10 0xb6c16068 in DataEventInputStream (this=0x2000, i=0x6, h=@0x8b1b30c)
    at /home/kde-devel/kdesvn/kdesupport/strigi/src/streams/dataeventinputstream.cpp:30
#11 0xb6c4f0c0 in Strigi::EventThroughAnalyzer::connectInputStream (
    this=0x8b1b308, in=0xbf9e52c8)
    at /home/kde-devel/kdesvn/kdesupport/strigi/src/streamanalyzer/eventthroughanalyzer.cpp:52
#12 0xb6c74fdb in Strigi::StreamAnalyzerPrivate::analyze (this=0x8aac388, 
    idx=@0xbf9e5314, input=0xbf9e52c8)
    at /home/kde-devel/kdesvn/kdesupport/strigi/src/streamanalyzer/streamanalyzer.cpp:392
#13 0xb77debd9 in KFileMetaInfoPrivate::init (this=0x8a90108, 
    stream=@0xbf9e5378, url=@0xbf9e5370, mtime=1196678426)
    at /home/kde-devel/kdesvn/kdelibs/kio/kio/kfilemetainfo.cpp:201
#14 0xb77e07d5 in KFileMetaInfo (this=0xbf9e5438, path=@0xbf9e543c)
    at /home/kde-devel/kdesvn/kdelibs/kio/kio/kfilemetainfo.cpp:224
#15 0x0808307c in InfoSidebarPage::showMetaInfo (this=0x854db18)
    at /home/kde-devel/kdesvn/kdebase/apps/dolphin/src/infosidebarpage.cpp:388
#16 0x08083428 in InfoSidebarPage::showItemInfo (this=0x854db18)
    at /home/kde-devel/kdesvn/kdebase/apps/dolphin/src/infosidebarpage.cpp:223
#17 0x08083cd3 in InfoSidebarPage::qt_metacall (this=0x854db18, 
    _c=QMetaObject::InvokeMetaMethod, _id=4, _a=0xbf9e55b8)
    at /home/kde-devel/kdesvn/build/kdebase/apps/dolphin/src/infosidebarpage.moc:93

Daniel, are you sure you're not talking about bug 163838?  This bug is not for crashing, it's for trying to read huge files in their entirety and causing slowdown/locked-up appearance.

Michael/Daniel : bug 164296 not 163838. closing this one again.

since today i dont have this probleme anymore !!!   :D

Mandriva 2009.0 kde 4.098 rc1

Just installed from: http://www.kubuntu.org/news/kde-4.1rc1
Placing cursor over a large file (such as 80MB or more) freezes dolphin for some time.

*** Bug 166688 has been marked as a duplicate of this bug. ***

*** Bug 166648 has been marked as a duplicate of this bug. ***

i still experience this bug with 4.0.98

@Niklas : What does strigidaemon --version returns ? 

wow that was fast, and right you are, i thought strigi was updated as well, but no it's still 0.5.10 so i'm building from svn now, but when you say it's fixed i guess it really is fixed ;)

Hum, 0.5.10 is the correct version. but maybe your distro packagers didn't add the latest patches.

For me, that issue still occurs.
Strigi version: 0.5.10
Kde version: 4.1 rc1
Installed from: kubuntu packages

I have a similar problem with KDE 4.1 RC1, the problem description is in bug 166648, using strigidaemon 0.5.10, opensuse factory packages.

For me, that issue still occurs.
Strigi version: 0.5.10
Kde version: 4.1 rc1
Installed from: kubuntu packages

For my personal experience (gentoo distro) I can say, that strigi is installed from the distro repository that doesn't have anything to do with the fixes/patches that are on the svn trunk of kde4.
This is the cause of the non-disappearance of the bug in many distros.
The only way to test it ... is get strigi from kde4 svn , and installed it manually overriding the strigi version of your distro.
The version number in my system (0.5.10) doesn't say anything...

*** Bug 166954 has been marked as a duplicate of this bug. ***

Does the recently released Strigi version (0.5.11) contain the fix for this bug?

*** Bug 167221 has been marked as a duplicate of this bug. ***

This was for my report of http://bugs.kde.org/show_bug.cgi?id=167221 now marked as duplicate.


dolphin: 1.0.99
kde: 4.0.98 (Debian experimental binaries)
libstrigihtmlgui0: 0.5.10
libstrigiqtdbusclient0: 0.5.10
strigi-applet: 0.5.7
strigi-client: 0.5.10
strigi-daemon: 0.5.10
strigi-plugins: 0.5.7
strigi-utils: 0.5.10

nepomuk is enabled and running
strigi is not enabled and not running


As it was hard to explain I recorded a video where you can see how freeze happens when first selecting a file and second time does well (enjoy):
http://es.youtube.com/watch?v=Wv8ozHled3I



For me console output has nothing strange:
$ dolphin
dolphin(11309) MetaDataWidget::setFile: KUrl("file:///home/rap")
rap@neonlight:~$ dolphin(11309) MetaDataWidget::setFile: KUrl("file:///home/rap")
dolphin(11309) KMimeTypeFactory::parseMagic: Now parsing  "/usr/share/mime/magic"
dolphin(11309) KMimeTypeFactory::parseMagic: Now parsing  "/home/rap/.local/share/mime/magic"
dolphin(11309) MetaDataWidget::setFile: KUrl("file:///home/rap/Movies")
dolphin(11309) MetaDataWidget::setFile: KUrl("file:///home/rap/Music")
....

since today the same probleme came back !! nnnnnnnoooooooooooooooooonnnnnnnnnnn 
i'll wait the next update !!!

I appear to still be having this issue with kde 4.1.0 and strigi 0.5.11.  Here is output from dolphin in terminal after mousing over a 44.8Mib pdf file.  29Mib pdf file will freeze it too. turning off the information panel fixes the issue.

output:

WARNING: field 'http://strigi.sf.net/ontologies/0.9#debugParseError' is not defined in any rdfs ontology database. 
/usr/lib/strigi/strigila_cpp.so                                                                                    
/usr/lib/strigi/strigila_deb.so                                                                                    
/usr/lib/strigi/strigila_xpm.so                                                                                    
/usr/lib/strigi/strigita_au.so                                                                                     
/usr/lib/strigi/strigila_namespaceharvester.so                                                                     
/usr/lib/strigi/strigita_gif.so                                                                                    
/usr/lib/strigi/strigita_pcx.so                                                                                    
/usr/lib/strigi/strigita_xbm.so                                                                                    
/usr/lib/strigi/strigila_txt.so                                                                                    
/usr/lib/strigi/strigita_avi.so                                                                                    
/usr/lib/strigi/strigita_avi.so: undefined symbol: strigiAnalyzerFactory                                           
/usr/lib/strigi/strigita_dds.so                                                                                    
/usr/lib/strigi/strigita_ico.so                                                                                    
/usr/lib/strigi/strigita_rgb.so                                                                                    
/usr/lib/strigi/strigita_sid.so                                                                                    
/usr/lib/strigi/strigita_wav.so                                                                                    
/usr/lib/strigi/strigita_wav.so: undefined symbol: strigiAnalyzerFactory                                           
WARNING: field 'maxLineLength' is not defined in any rdfs ontology database.                                       
WARNING: field 'line ending format' is not defined in any rdfs ontology database.                                  
WARNING: field 'dds volume depth' is not defined in any rdfs ontology database.                                    
WARNING: field 'dds mipmap count' is not defined in any rdfs ontology database.                                    
WARNING: field 'dds image type' is not defined in any rdfs ontology database.                                      
WARNING: field 'document.stats.image_count' is not defined in any rdfs ontology database.                          
WARNING: field 'document.stats.image_name' is not defined in any rdfs ontology database.                           
WARNING: field 'document.stats.image_shared_rows' is not defined in any rdfs ontology database.                    
WARNING: field 'ole.category' is not defined in any rdfs ontology database.                                        
WARNING: field 'ole.presentationtarget' is not defined in any rdfs ontology database.                              
WARNING: field 'ole.manager' is not defined in any rdfs ontology database.                                         
WARNING: field 'ole.company' is not defined in any rdfs ontology database.
WARNING: field 'document.stats.table_count' is not defined in any rdfs ontology database.
WARNING: field 'document.stats.object_count' is not defined in any rdfs ontology database.
WARNING: field 'http://rdf.openmolecules.net/0.9#moleculeCount' is not defined in any rdfs ontology database.
dolphin(12610) KFileMetaInfoPrivate::init: KUrl("file:///home/vblanton/music/piano/piano sheet music/12 FAKEBOOKS/realbk2.pdf")

*** Bug 168884 has been marked as a duplicate of this bug. ***

*** Bug 169706 has been marked as a duplicate of this bug. ***

The same problem appears for *.mp4 files (with extension mp4) in kde 4.2.1 .

I also experience this problem in 4.2.4 with .mp4 files.