Hi, I was trying to find a file and usually do: "find /home/mark -iname "*part_of_the_name*" which returns results within a few seconds. Mind you, my home folder is (when testing this) 381GB large and containing 1095284 files (find /home/mark -type f | wc -l). That count also return within about a second. That's just to give an impression of number of files that have to be searched through to find that file i was looking for. It takes about 2 seconds for the find command to find the files and be done. It takes about 25 seconds for Dolphin to do the exact same thing (i'm not searching through content, only filenames). I verified that dolphin is in fact using this KIO slave :) (as it could be baloo too) Now i was looking through the code of filenamesearch to figure out why it would be this slow. Obviously, there is some overhead when using KIO::ListDir or KCoreDirLister as it goes over the socket connection and does serialization. So i fully expect it to be (a bit) slower. As those paths have been massively optimized in the past (partly by me too) i kinda have a feeling what can be expected as slowdown. 2 or 3x slower can be expected, ~13x like i'm observing here is not the expected behavior and smells like a severe deficiency in this code path. My first gut feeling was to use KIO::listRecursive instead of manually implementing recursive folder iteration as is done now in filenamesearch. I was guessing that to be (a lot) faster as the KIO::listRecursive code simply has seen tons of optimizations over the past, well, decades. That turned out right, the time is then reduced to about ~8 seconds as opposed to the ~25 it was before. However, this introduces a much more complicated issue. You do not want the caching logic of KIO::listRecursive when searching for files. Sadly, there is no way to tell KIO::listRecursive to not use caching hence going the KCoreDirLister isn't such a bad idea (that's how it's done now). But there is more needed to improve searching. Right now if you search for an entry and it finds one within 300ms (mind you, KIO does 300ms batching of entries) but then don't find another entry for - say - 24 seconds, then you will not get any listEntry update for that time. Thus no GUI entry for an item that might be found all along. You will get the entry as soon as there is a new match (triggering the 300ms interval) or till finished() is called. This can be improved, but i'm not yet sure what would be wise. Leaving it open here as a braindump. Cheers, Mark
> You will get the entry as soon as there is a new match (triggering the 300ms interval) or till finished() is called. That is also an issue if you enter a folder with a slow mount inside. KIO goes like: while ((ep = QT_READDIR(dp)) != nullptr) QT_LSTAT ... listEntry(entry) So should lstat get stuck on a slow mount, none of the previous queued up files are emitted until lstat returns and the event loop continues to then find that the 300ms batch time has expired and send all entries along, i.e. enter a folder with 30 files where there is one folder pointing to a slow mount. You won't see any files until that slow mount has responded. Obvious solution: listEntries handling in a thread in SlaveBase :P
(In reply to Kai Uwe Broulik from comment #1) > Obvious solution: listEntries handling in a thread in SlaveBase :P Exactly my thought! And i actually have a threaded version - somewhat - working here locally, but that on it's own has some issues. - SlaveBase doesn't inherit from QObject - Due to above: no signals/slots directly I can still work around that. Just make a QThread in which a QTimer lives that does the 300ms interval. Something like: QThread *timerThread = new QThread(nullptr); // no No slavebase as parent, no QObject QTimer *timer = new QTimer(nullptr); timer->moveToThread(timerThread); That very timer should be started when the listEntry objects start flowing in and stopped when finished() is called. And that's where i am right now with another issue. In the approach from above the QTimer lives in another thread context. Aka, you cannot call _any_ functions on it from within SlaveBase. No start, stop, isActive, ... I am, right now, managing to start the timer like so: QObject::connect(timerThread, &QThread::started, [this]() { timer->start(); }); Note: the "this" is "SlaveBasePrivate", the timer and timerThread objects live in the SlaveBasePrivate scope. Do you have any idea how to start/stop a timer that lives in it's own QThread but start/stop them from within SlaveBase without signals?.... ;)
You could make SlaveBasePrivate a QObject.
(In reply to Kai Uwe Broulik from comment #3) > You could make SlaveBasePrivate a QObject. Hmm, i wonder if that's acceptable? Well, i can try if it works at all ;)
It surely won't change ABI because it's the Private part. The only downside I can see is some overhead of QObject creation? But then, how many of those objects will we have anyway?
(In reply to Kai Uwe Broulik from comment #5) > It surely won't change ABI because it's the Private part. The only downside > I can see is some overhead of QObject creation? But then, how many of those > objects will we have anyway? No it does not (without compile testing). I'd likely get into trouble with moc and signals as it's all in the cpp file. For that to work i'd have to give SlaveBasePrivate a header...
#include "slavebase.moc" at the end and it should just work
(In reply to Kai Uwe Broulik from comment #7) > #include "slavebase.moc" at the end and it should just work Ha, smart! Oke, so I've got that to work somewhat, but this isn't easy by far. The tricky thing is the slot connection for listEntries. They need to run on the main thread and will be executed by the main thread eventloop (via Qt::QueuedConnection). Which doesn't exist! I can call it from the thread's event loop but that makes the sockets _really_ mad and makes the slave crash, hehe :) Just handling the event in SlaveBase::dispatchLoop also isn't possible because, well, the main thread were that lives is blocked by the slave. The only solution i see now is lifting out all socket handling in a separate (QThread based?) class that isn't blocking and has an eventloop. But that is waaaaaaay beyond an easy fix. If you have more ideas, please do share!