Bug 346403

On further investigation, the syscalls made during the freeze are all stat calls to /etc/localtime, which appears to be done for each file or directory. My memory on the xml thing was wrong, apologies for that.

Moved this to the product kfile, since kdialog simply builds on top of that.

The right product is kio, reassigning.

I can reproduce major slowness when clicking on the button to turn previews off in the file dialog, and I have just fixed that. However I don't see the slowness otherwise. Is this with previews on or off ? What type of files, in terms of extension and content ?

This is with the side-panel preview on. The folder is full of mainly PNG images. The images are sorted by filename.

I've just tried again with kdialog and compared it to a minimal file widget application I wrote based on the kio tests, and while kdialog now seems to no longer hang (although this isn't git HEAD), the test application still does.

Correction: kdialog now seems to hang again, so seemingly this isn't consistently reproducible.
It still only hangs if the folder has a lot of files in it, though.
After a few tries, I've also gotten it to hang on my /usr/bin/ (5490 items), so it seems the filetype does not matter.

Created attachment 101545 [details]
gdb backtrace of interrupted process while it hangs

Here's the gdb backtrace of a hanging kdialog process (built as debug), however kde frameworks and Qt aren't built with debug symbols because Arch Linux doesn't provide pre-compiled debug packages, I'll try to build everything with debug symbols someday, but since this will probably take a long time I haven't done so yet.

Created attachment 101546 [details]
A different backtrace from another hanging kdialog process

Sorry for the many successive comments, but here's another backtrace, this time hanging in libharfbuzz.

It looks like this might be an issue with my system's fonts, which would explain why you might not be able to reproduce it.

Git commit 09cd8420e6c901a482c346fdaf4bc1cc38587646 by David Faure.
Committed on 13/10/2016 at 16:52.
Pushed by dfaure into branch 'master'.

Major performance improvement when turning previews off in the file dialog.

Added a method for this KDirModel so it can emit dataChanged only once
(or once per dir in a tree) rather than for every item.
REVIEW: 129159

M  +0    -10   src/filewidgets/kdiroperator.cpp
M  +1    -24   src/filewidgets/kfilepreviewgenerator.cpp
M  +27   -1    src/widgets/kdirmodel.cpp
M  +7    -0    src/widgets/kdirmodel.h

http://commits.kde.org/kio/09cd8420e6c901a482c346fdaf4bc1cc38587646

Thanks for the backtraces. They don't necessarily indicate "hanging in libharfbuzz".
More likely, KFilePreviewGenerator::Private::dispatchIconUpdateQueue() is triggering dataChanged for one item at a time, just like in the bug I fixed.

This confirms my suspicion that it's related to previews. I don't mean the side panel, I mean the icons themselves turning into a preview of the file (e.g. for images, you see the image in small instead of the generic "this is an image" icon). To confirm this, you can try again without previews enabled (although the actual action of disabling previews will be slow until you get my bugfix -- better do that in a small folder before going back to the big folder).

The way to find out if it's hanging in harfbuzz is simple: once you're in gdb, do
break KDirModel::itemChanged
cont
and you should see that it does continue, and it will hit itemChanged again, and again (once per file).
This goes back to the event loop between files though, i.e. it's supposed to be happening "in the background" while you can still use the dialog. That's definitely what I see here.

In /usr/bin the issue is: most files don't have an extension, so mimetype determination has to happen (by reading the contents of the file), and this too happens delayed (in the background), from the same KFilePreviewGenerator class (yes, it does more than the name indicate, it does mimetype determination and previews). So it's likely the same issue, just triggered slightly differently. I expect it to make the app more busy in fact, given that previews are generated by a separate process (kioslave) so in theory the app (kdialog here) should be waiting for the preview to come in (just a little bit).

Anyhow, I'm a bit puzzled because when previews are generated, then yes it's expected that the view will re-layout one item at a time. It's what happens here, it's what happens for you... but somehow the user visible behaviour seems different? Or my machine is too powerful for me to find it slow....

Actually, I already have previews turned off in the folders in which this happens. The fact that it happens inconsistently would also indicate that the problem is something else.

I'll try to mess around with gdb some more when I get the time to.

Seems i have same issue, in kdevelop 5.0.1 and also in previous version:

Step to reproduce:
1) if open a file in arch/arm/boot/dts/
2) close kdevelop
3) reopen it so that file is already visible
4) when the dialog file path already point to (a linux source tree) some file in "arch/arm/boot/dts/" so if do open and same path is auto-selected

i have to wait at least 20 seconds with dialog as freezed, un-scrollable and un-selectable.

I break gdb while waiting, the backtrace is below:

$ gdb kdevelop
GNU gdb (Debian 7.11.1-2) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from kdevelop...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/kdevelop 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffda57c700 (LWP 2485)]
[New Thread 0x7fffd8fc7700 (LWP 2486)]
[New Thread 0x7fffd120d700 (LWP 2488)]
[New Thread 0x7fffc97be700 (LWP 2489)]
[New Thread 0x7fffbdfde700 (LWP 2490)]
[New Thread 0x7fffbcb21700 (LWP 2501)]
Not loading plugin named "KDevNinjaBuilder" because it has been disabled!
QInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: Permesso negato
Invalid return type in method "core"
Invalid return type in method "extensions"
[New Thread 0x7fffb60e5700 (LWP 2573)]
[New Thread 0x7fffb58e4700 (LWP 2575)]
Not loading plugin named "KDevNinjaBuilder" because it has been disabled!
Invalid return type in method "core"
Invalid return type in method "extensions"
[New Thread 0x7fffb4c5b700 (LWP 2600)]
[New Thread 0x7fffa3fff700 (LWP 2601)]
[New Thread 0x7fffa37fe700 (LWP 2602)]
[New Thread 0x7fffa2ffd700 (LWP 2603)]
[New Thread 0x7fffa27fc700 (LWP 2604)]
[New Thread 0x7fffa1ffb700 (LWP 2605)]
[New Thread 0x7fffa17fa700 (LWP 2607)]
[New Thread 0x7fffa0ff9700 (LWP 2608)]
[New Thread 0x7fff83fff700 (LWP 2609)]
[New Thread 0x7fff837fe700 (LWP 2610)]
[Thread 0x7fff837fe700 (LWP 2610) exited]
^C
Thread 1 "kdevelop" received signal SIGINT, Interrupt.
0x00007ffff585c780 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007ffff585c780 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007ffff5d29722 in QUrl::detach() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#2  0x00007ffff5d29a97 in QUrl::setScheme(QString const&) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#3  0x00007ffff5d2c9f8 in QUrl::fromLocalFile(QString const&) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007ffff131a875 in KFileItem::mostLocalUrl(bool*) const () from /usr/lib/x86_64-linux-gnu/libKF5KIOCore.so.5
#5  0x00007ffff131f893 in ?? () from /usr/lib/x86_64-linux-gnu/libKF5KIOCore.so.5
#6  0x00007ffff131fa0b in KFileItem::overlays() const () from /usr/lib/x86_64-linux-gnu/libKF5KIOCore.so.5
#7  0x00007ffff1947d98 in KDirModel::data(QModelIndex const&, int) const ()
   from /usr/lib/x86_64-linux-gnu/libKF5KIOWidgets.so.5
#8  0x00007ffff5d96f29 in QSortFilterProxyModel::data(QModelIndex const&, int) const ()
   from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#9  0x00007ffff18f944c in KFileItemDelegate::Private::decoration(QStyleOptionViewItem const&, QModelIndex const&) const () from /usr/lib/x86_64-linux-gnu/libKF5KIOWidgets.so.5
#10 0x00007ffff18f9998 in KFileItemDelegate::Private::initStyleOption(QStyleOptionViewItem*, QModelIndex const&) const () from /usr/lib/x86_64-linux-gnu/libKF5KIOWidgets.so.5
#11 0x00007ffff18fa2ea in KFileItemDelegate::sizeHint(QStyleOptionViewItem const&, QModelIndex const&) const ()
   from /usr/lib/x86_64-linux-gnu/libKF5KIOWidgets.so.5
#12 0x00007ffff69b5e49 in QTreeViewPrivate::widthHintForIndex(QModelIndex const&, int, QStyleOptionViewItem const&, int) const () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#13 0x00007ffff69b8dcf in QTreeView::sizeHintForColumn(int) const ()
   from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#14 0x00007ffff698a5d6 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#15 0x00007ffff51ad3b0 in ?? () from /usr/lib/x86_64-linux-gnu/libKF5KIOFileWidgets.so.5
#16 0x00007ffff5e00bd9 in QMetaObject::activate(QObject*, int, int, void**) ()
   from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#17 0x00007ffff5e7a4a6 in QAbstractItemModel::layoutChanged(QList<QPersistentModelIndex> const&, QAbstractItemModel::LayoutChangeHint) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#18 0x00007ffff5d9fe1e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007ffff5da19c8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#20 0x00007ffff5e00bd9 in QMetaObject::activate(QObject*, int, int, void**) ()
   from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#21 0x00007ffff5e7a3bc in QAbstractItemModel::dataChanged(QModelIndex const&, QModelIndex const&, QVector<int> const&) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#22 0x00007ffff194750c in KDirModel::itemChanged(QModelIndex const&) ()
   from /usr/lib/x86_64-linux-gnu/libKF5KIOWidgets.so.5
#23 0x00007ffff51d6306 in KFilePreviewGenerator::Private::dispatchIconUpdateQueue() ()
   from /usr/lib/x86_64-linux-gnu/libKF5KIOFileWidgets.so.5
#24 0x00007ffff51d6678 in KFilePreviewGenerator::Private::resolveMimeType() ()
   from /usr/lib/x86_64-linux-gnu/libKF5KIOFileWidgets.so.5
#25 0x00007ffff5e01729 in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#26 0x00007ffff670fbec in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#27 0x00007ffff671508f in QApplication::notify(QObject*, QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#28 0x00007ffff5dd3450 in QCoreApplication::notifyInternal2(QObject*, QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#29 0x00007ffff5dd53cc in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) ()
   from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#30 0x00007ffff5e290f3 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#31 0x00007fffee4b61a7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#32 0x00007fffee4b6400 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#33 0x00007fffee4b64ac in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#34 0x00007ffff5e294ff in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#35 0x00007ffff5dd119a in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#36 0x00007ffff691455c in QDialog::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) quit
A debugging session is active.

	Inferior 1 [process 2481] will be killed.

* the issue is not really happening always, strangely, i.e. i closed all files
and reopened kde, opening file seems to work fine now

I would like to help get this bug's status changed to confirmed. The following is enough to cause the bug to manifest:

$ mkdir ~/test
$ touch ~/test/{000..999}.txt
$ kdialog --getsavefilename ~/test/

Observations:

1) This causes a slowdown of 22–29 seconds on my i7 + SSD laptop while kdialog eats up 100% of a core. If you are not seeing the slowdown, you can of course add a few nines ;)

2) Subsequent runs of the kdialog line are always slow. 

3) The slowdown is always present, regardless of whether or not the "Show Previews" button is pressed. 

4) Toggling the "Show Previews" button after the dialog becomes responsive does NOT cause any additional delay (this is not just because my example involves only empty files—it happens in real directories, too).

5) I can also confirm that another aspect of the bug is still present: If we instead run 

$ kdialog --getsavefilename ~/

...THEN navigate to ~/test/ via the GUI, there is no slowdown whatsoever. I feel that this is a key point because it shows that there is nothing inherently slow about displaying a large number of files in kdialog.

Please take another look at this bug.



Arch Linux / Plasma 5.9.4 / Frameworks 5.32.0 / QT 5.8.0 / kdialog 1.0

I can confirm this with your example, using detail view, file preview NOT activated. Using KIO from 5.32.0 on openSuse 42.2

Fixed via https://phabricator.kde.org/D5774