Bug 398900

Summary: "Quick open" got much (100x) slower with 18.08 [regression]
Product: [Applications] kate Reporter: Ralf Jung <post>
Component: generalAssignee: KWrite Developers <kwrite-bugs-null>
Status: RESOLVED FIXED    
Severity: normal CC: averageradical, nate, neundorf
Priority: NOR    
Version: 18.08.0   
Target Milestone: ---   
Platform: Other   
OS: Linux   
See Also: https://bugs.kde.org/show_bug.cgi?id=387360
Latest Commit: https://cgit.kde.org/kate.git/commit/?id=d6e38c0cbd3d6d7f7658862757b8c33db4322d7a Version Fixed In: 18.12.0
Sentry Crash Report:

Description Ralf Jung 2018-09-21 08:14:13 UTC 
With Kate 18.04, I first discovered the "quick open" feature (Ctrl-Alt-O), and started to get used to it. Unfortunately, with Kate 18.08, it got so slow it is useless.

To reproduce, clone the Rust compiler (git clone https://github.com/rust-lang/rust/), ideally onto an SSD, and make sure to initialize all submodules (git submodule update --init --recursive). This will take a bit. Then open some files under src, and also open src/README.md. Then do Ctrl-Alt-O. This used to be instantaneous with previous versions of kate, now it takes around 10 seconds (so it is at least 100x slower).

Also, it is now listing every file twice when I start searching, once as `/home/r/src/rust/rustc//src/...` and once as `file://home/r/src/rust/rustc/src/...` (one has a duplicate slash, and one has a `file://` prefix). I do not this this was the case before the update.

I am on Debian testing.
Comment 1 Dominik Haumann 2018-09-23 17:14:01 UTC 
Alex, is this what you also experience?
Comment 2 Alexander Neundorf 2018-09-23 18:46:40 UTC 
(In reply to Dominik Haumann from comment #1)
> Alex, is this what you also experience?

No, but my colleague has probably this same problem.

"Quick open" blocks for 10 seconds or so.
He is using it inside an svn-checkout, and he has a cmake-generated project with the "svn" tag loaded.
It seems like, even though he disabled the automatic loading of svn-projects in the settings, kate starts to do a lot of stuff with svn when he tries Quick Open.
Comment 3 Alexander Neundorf 2018-09-26 20:57:48 UTC 
(In reply to Alexander Neundorf from comment #2)
> (In reply to Dominik Haumann from comment #1)
> > Alex, is this what you also experience?
> 
> No, but my colleague has probably this same problem.
> 
> "Quick open" blocks for 10 seconds or so.
> He is using it inside an svn-checkout, and he has a cmake-generated project
> with the "svn" tag loaded.
> It seems like, even though he disabled the automatic loading of svn-projects
> in the settings, kate starts to do a lot of stuff with svn when he tries
> Quick Open.

My colleague is using 18.04, and it is already slow for him.
I want to have a look at his system next week.
Comment 4 Nate Graham 2018-09-27 17:34:27 UTC 
*** Bug 399151 has been marked as a duplicate of this bug. ***
Comment 5 Alexander Neundorf 2018-10-16 19:51:18 UTC 
(In reply to Alexander Neundorf from comment #3)
...
> My colleague is using 18.04, and it is already slow for him.
> I want to have a look at his system next week.


It's strange.
It's slow on that machine (a few seconds), and fast on my machine, using the same project.
It's slow on that machine when a file which is part of a kate project is open (which contains MANY files), it's fast if e.g. a file in the kate source tree is open. Will look more into it.
Comment 6 Dominik Haumann 2018-10-17 17:52:03 UTC 
What about this change: https://phabricator.kde.org/D15804 ? Could you try this patch?
Comment 7 Alexander Neundorf 2018-10-17 20:39:42 UTC 
I'll give it a try.

I had a closer look at the performance issue.
The project were it is slow is quite big, it contains around 19.000 files.
On the slow machine/installation, void KateQuickOpen::update() takes 10 seconds.
All the time is spent in the loop which puts the project files into the model:
foreach(const QString & file, projectFiles)

My two installations (a relatively old version and one from current git) both run 
fast, here this loop over 19.000 files takes 100 milliseconds. I.e. it is 100 times faster. In the slow install, it seems like most of the time is spent when putting the two QStandardItems into the model.

Below are three backtraces from within those 10 seconds.
They are all QStandardItemPrivate::setChild() which emits QAbstractItemModel::layoutChanged(...). Updating a layout 19.000 times probably can take some time. This would be around 0.5 milliseconds per update.
But why is it slow on one setup and fast on another ?
Could this be related to the "environment", the window manager (fast with kwin, slow with i3 and xfce?), Qt options, X driver ?

#0  0x00007ffff30d0d27 in QSortFilterProxyModelPrivate::create_mapping(this=this@entry=0x555556059490, source_parent=...) at itemmodels/qsortfilterproxymodel.cpp:375
#1  0x00007ffff30d15f9 in QSortFilterProxyModelPrivate::update_persistent_indexes(QVector<QPair<QModelIndex, QPersistentModelIndex> > const&) () at itemmodels/qsortfilterproxymodel.cpp:1079
#2  0x00007ffff30d1c79 in QSortFilterProxyModelPrivate::_q_sourceLayoutChanged (this=0x555556059490, sourceParents=..., hint=<optimized out>) at itemmodels/qsortfilterproxymodel.cpp:1454
#3  0x00007ffff30d6823 in QSortFilterProxyModel::qt_static_metacall (_o=<optimized out>, _c=<optimized out>, _id=<optimized out>, _a=<optimized out>) at itemmodels/qsortfilterproxymodel.h:174
#4  0x00007ffff311ae1e in QMetaObject::activate(QObject*, int, int, void**)() at kernel/qobject.cpp:3771
#5  0x00007ffff311b4ab in QMetaObject::activate (sender=<optimized out>, m=m@entry=0x7ffff337bbe0 <QAbstractItemModel::staticMetaObject>, local_signal_index=local_signal_index@entry=3, argv=argv@entry=0x7fffffffd3f0) at kernel/qobject.cpp:3633
#6  0x00007ffff30ae5ba in QAbstractItemModel::layoutChanged(this=<optimized out>, _t1=..., _t2=<optimized out>, _t2@entry=QAbstractItemModel::NoLayoutChangeHint) at .moc/moc_qabstractitemmodel.cpp:566
#7  0x00007ffff3ce89ae in QStandardItemPrivate::setChild(int, int, QStandardItem*, bool) () at itemmodels/qstandarditemmodel.cpp:147
#8  0x00005555555ca30c in KateQuickOpen::update (this=0x5555560493a0) at /usr/src/debug/kate-18.08.2-1.fc28.x86_64/kate/katequickopen.cpp:258
#9  0x00005555555944b4 in KateMainWindow::slotQuickOpen (this=0x555555a44fc0) at /usr/src/debug/kate-18.08.2-1.fc28.x86_64/kate/katemainwindow.cpp:1144


#0  0x00007ffff30be222 in mergeIndexes(QVector<QPersistentModelIndex> const&) () at itemmodels/qitemselectionmodel.cpp:967
#1  0x00007ffff30c4236 in QItemSelectionModelPrivate::_q_layoutChanged(QList<QPersistentModelIndex> const&, QAbstractItemModel::LayoutChangeHint) () at itemmodels/qitemselectionmodel.cpp:1085
#2  0x00007ffff30c489b in QItemSelectionModel::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) () at itemmodels/qitemselectionmodel.h:148
#3  0x00007ffff311ae1e in QMetaObject::activate(QObject*, int, int, void**)() at kernel/qobject.cpp:3771
#4  0x00007ffff311b4ab in QMetaObject::activate(sender=sender@entry=0x5555560613c0, m=m@entry=0x7ffff337bbe0 <QAbstractItemModel::staticMetaObject>, local_signal_index=local_signal_index@entry=3, argv=argv@entry=0x7fffffffd260) at kernel/qobject.cpp:3633
#5  0x00007ffff30ae5ba in QAbstractItemModel::layoutChanged (this=this@entry=0x5555560613c0, _t1=..., _t2=<optimized out>, _t2@entry=QAbstractItemModel::NoLayoutChangeHint) at .moc/moc_qabstractitemmodel.cpp:566
#6  0x00007ffff30d1ca4 in QSortFilterProxyModelPrivate::_q_sourceLayoutChanged (this=0x55555605b480, sourceParents=..., hint=<optimized out>) at itemmodels/qsortfilterproxymodel.cpp:1463
#7  0x00007ffff30d6823 in QSortFilterProxyModel::qt_static_metacall (_o=<optimized out>, _c=<optimized out>, _id=<optimized out>, _a=<optimized out>) at itemmodels/qsortfilterproxymodel.h:174
#8  0x00007ffff311ae1e in QMetaObject::activate(QObject*, int, int, void**)() at kernel/qobject.cpp:3771
#9  0x00007ffff311b4ab in QMetaObject::activate (sender=<optimized out>, m=m@entry=0x7ffff337bbe0 <QAbstractItemModel::staticMetaObject>, local_signal_index=local_signal_index@entry=3, argv=argv@entry=0x7fffffffd3f0) at kernel/qobject.cpp:3633
#10 0x00007ffff30ae5ba in QAbstractItemModel::layoutChanged(this=<optimized out>, _t1=..., _t2=<optimized out>, _t2@entry=QAbstractItemModel::NoLayoutChangeHint) at .moc/moc_qabstractitemmodel.cpp:566
#11 0x00007ffff3ce89ae in QStandardItemPrivate::setChild(int, int, QStandardItem*, bool) () at itemmodels/qstandarditemmodel.cpp:147
#12 0x00005555555ca322 in KateQuickOpen::update (this=0x555556056380) at /usr/src/debug/kate-18.08.2-1.fc28.x86_64/kate/katequickopen.cpp:259
#13 0x00005555555944b4 in KateMainWindow::slotQuickOpen (this=0x555555a44270) at /usr/src/debug/kate-18.08.2-1.fc28.x86_64/kate/katemainwindow.cpp:1144


#0  0x00007ffff30ceb72 in QVector<int>::fill (asize=-1, from=<optimized out>, this=0x55555654ccb0) at ../../include/QtCore/../../src/corelib/tools/qarraydata.h:61
#1  QSortFilterProxyModelPrivate::build_source_to_proxy_mapping (this=this@entry=0x55555605a9d0, proxy_to_source=..., source_to_proxy=...) at itemmodels/qsortfilterproxymodel.cpp:1041
#2  0x00007ffff30d0eb5 in QSortFilterProxyModelPrivate::create_mapping(this=this@entry=0x55555605a9d0, source_parent=...) at itemmodels/qsortfilterproxymodel.cpp:395
#3  0x00007ffff30d15f9 in QSortFilterProxyModelPrivate::update_persistent_indexes(QVector<QPair<QModelIndex, QPersistentModelIndex> > const&) () at itemmodels/qsortfilterproxymodel.cpp:1079
#4  0x00007ffff30d1c79 in QSortFilterProxyModelPrivate::_q_sourceLayoutChanged (this=0x55555605a9d0, sourceParents=..., hint=<optimized out>) at itemmodels/qsortfilterproxymodel.cpp:1454
#5  0x00007ffff30d6823 in QSortFilterProxyModel::qt_static_metacall(_o=<optimized out>, _c=<optimized out>, _id=<optimized out>, _a=<optimized out>) at itemmodels/qsortfilterproxymodel.h:174
#6  0x00007ffff311ae1e in QMetaObject::activate(QObject*, int, int, void**) () at kernel/qobject.cpp:3771
#7  0x00007ffff311b4ab in QMetaObject::activate (sender=<optimized out>, m=m@entry=0x7ffff337bbe0 <QAbstractItemModel::staticMetaObject>, local_signal_index=local_signal_index@entry=3, argv=argv@entry=0x7fffffffd3f0) at kernel/qobject.cpp:3633
#8  0x00007ffff30ae5ba in QAbstractItemModel::layoutChanged (this=<optimized out>, _t1=..., _t2=<optimized out>, _t2@entry=QAbstractItemModel::NoLayoutChangeHint) at .moc/moc_qabstractitemmodel.cpp:566
#9  0x00007ffff3ce89ae in QStandardItemPrivate::setChild(int, int, QStandardItem*, bool) () at itemmodels/qstandarditemmodel.cpp:147
#10 0x00005555555ca322 in KateQuickOpen::update (this=0x555556055640) at /usr/src/debug/kate-18.08.2-1.fc28.x86_64/kate/katequickopen.cpp:259
#11 0x00005555555944b4 in KateMainWindow::slotQuickOpen (this=0x555555a443a0) at /usr/src/debug/kate-18.08.2-1.fc28.x86_64/kate/katemainwindow.cpp:1144
Comment 8 Alexander Neundorf 2018-10-18 21:44:19 UTC 
The patch in https://phabricator.kde.org/D15804 indeed seems to fix the performance problem, and I haven't noticed any other problems. :-)
Comment 9 Nate Graham 2018-10-18 23:06:32 UTC 
Great, thanks for testing!


author	Tomaz Canabrava <tcanabrava@kde.org>	2018-10-16 14:08:54 (GMT)
committer	Tomaz Canabrava <tcanabrava@kde.org>	2018-10-17 10:01:15 (GMT)
commit	d6e38c0cbd3d6d7f7658862757b8c33db4322d7a (patch)
tree	30e3b36b4b6c8921eb5613d0b1aa3c2420c14695
parent	e2ebc77065c2dae681c5fecf3a1f898e065bc7f6 (diff)
Fix Duplicates on Quick Open File
Summary:
The logic on the quick open was overcomplex, and it led to
bugs while filling the document list.
The new logic fills a vector with all the files,
organizes them by Url, remove the files based on the
duplicated url's (so name matching doesn't matter) and only
then starts to populate the view.

I belive this is also faster than the old code as I minimized
the amount of memory allocations and deallocations.

I'm unsure about the use of QFileInfo in the end of the code as
we can have too many files there and creating a temporary
QFileInfo just to get the name is a huge overhead, we can do a split('/').last()
for the file name. I'll do some measurements with that.

Reviewers: cullmann, brauch, neundorf

Reviewed By: cullmann

Subscribers: gregormi, dhaumann, kwrite-devel

Tags: #kate

Differential Revision: https://phabricator.kde.org/D15804