SUMMARY When scanning a collection which includes an album with many sub-directories, the scanner's performance is heavily impacted. This regression was introduced in commit e00ade3c (first included in v6.4.0), due to calling function QDir::entryList too many times in the loop that iterates the collection (collectionscanner_scan.cpp:489 and potentially also 552). One possible fix is to cache the result of entryList, as well as the parent sub-album's directory from the previous iteration. In the next iteration, if the parent directory of the current sub-album is the same as the one saved from the previous iteration, use the cached result of entryList, instead of calling the function again. Since all sub-albums with the same root will be iterated together, this implementation should reduce significantly the performance penalty. STEPS TO REPRODUCE 1. Create the collection directory with many (tested with 9K) sub-directories. Each sub-directory shall contain a number of pictures. 2. Add the created directory to digikam and trigger a media scan. OBSERVED RESULT The scan takes a very long time to complete (from hours to days) or stalls indefinitely. EXPECTED RESULT The scan completes in a few minutes. SOFTWARE/OS VERSIONS Windows: - macOS: 10.14.6 Linux/KDE Plasma: - (available in About System) KDE Plasma Version: - KDE Frameworks Version: 5.63.0 Qt Version: 5.13.2 ADDITIONAL INFORMATION Tested with pre-built v6.4.0 on macOS and Linux, as well as self-compiled v7.0.0-git (5a1bc45b) on macOS.
To be more precise, the commit is this one : https://invent.kde.org/kde/digikam/commit/e00ade3c4bd32822db4531bf32906633c84e6c53 Gilles Caulier
Well, it's the only way to get the real file name of a folder under Windows. I am amazed that this code takes so long and MacOS. I had carried out intensive tests and there were hardly measurable delays under Linux. Because the folders should be long in the cache, because we already determine the number of folders. Ok, we can only allow this code for Windows. Maik
Git commit 2d214984e574e63c082e5461cf2c8d72abe95bc1 by Maik Qualmann. Committed on 20/11/2019 at 11:30. Pushed by mqualmann into branch 'master'. read whole dir only under Windows to get correct file name FIXED-IN: 7.0.0 M +2 -1 NEWS M +6 -4 core/libs/database/collection/collectionscanner_scan.cpp https://invent.kde.org/kde/digikam/commit/2d214984e574e63c082e5461cf2c8d72abe95bc1
Since we are running in least 3 times through network or local folders in the collection scanner, I've been planning a try with a cache for a while now. We'll see how much a QHash consumes with 100K QFileInfo objects and whether it significantly improves performance. Maik
Hi Maik Thanks for your fast support, the last commit fixes the issue in my setup. Since the collection I am working with is accessed through a FUSE mount, it is possible that filesystem caches are not being used. So I believe the issue is not macOS/Linux specific and could also arise on Windows for certain volume types. Daniel
*** Bug 420334 has been marked as a duplicate of this bug. ***