Bug 437771

Summary: scanning performance much too slow
Product: [Applications] digikam Reporter: Clara Spielmann <clara.lina.sophie>
Component: Database-ScanAssignee: Digikam Developers <digikam-bugs-null>
Status: REPORTED ---    
Severity: normal CC: caulier.gilles, iwannaberich, metzpinguin
Priority: NOR    
Version: 7.1.0   
Target Milestone: ---   
Platform: Microsoft Windows   
OS: Microsoft Windows   
Latest Commit: Version Fixed In:
Sentry Crash Report:

Description Clara Spielmann 2021-05-28 11:15:02 UTC
SUMMARY


STEPS TO REPRODUCE
1. First time scanning of 135.000 / 500 GB of images. 
2. CPU Intel Intel Core i5-8250U
3. 

OBSERVED RESULT
It takes days with a CPU rate of around 3 %

EXPECTED RESULT
It should be much faster, there should be an option to disable CPU limitation to such a low percentage.
PROBLEM: If you don't request performance, CPU stays in low frequency and the whole process takes even longer. This behaviour is unbearable.

SOFTWARE/OS VERSIONS
Windows: Windows 10 21H1
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

ADDITIONAL INFORMATION
In contrary to other bug reports to this subject, I want to express that thew scanning takes too few system resources to do its job.
Comment 1 Maik Qualmann 2021-05-28 11:56:19 UTC
There is no CPU limit on the first scan.
Which database type are you using (MySQL / SQLite)
Where is the database located? (Local, network)
Where is the collection located? (Local SSD / HDD, USB, network)

The digiKam version used is no longer up-to-date.

Maik
Comment 2 Maik Qualmann 2021-08-09 19:32:09 UTC
Git commit 03f233383d94c20099afe07e75edb727694e5c3c by Maik Qualmann.
Committed on 09/08/2021 at 19:29.
Pushed by mqualmann into branch 'master'.

first step for a faster scan of network drives
We only run through a network collection once in
the first scan to determine the number of files.
The possible number of images and albums is now
taken from the database.
This can be a little less precise under certain
circumstances, we will test it.
Related: bug 405235, bug 431037, bug 265241

M  +1    -1    core/libs/database/collection/collectionscanner.h
M  +11   -13   core/libs/database/collection/collectionscanner_scan.cpp
M  +25   -4    core/libs/database/collection/collectionscanner_utils.cpp
M  +33   -0    core/libs/database/coredb/coredb.cpp
M  +5    -0    core/libs/database/coredb/coredb.h

https://invent.kde.org/graphics/digikam/commit/03f233383d94c20099afe07e75edb727694e5c3c
Comment 3 Maik Qualmann 2021-08-16 20:11:54 UTC
Git commit d14487fc0da4e26bc7ba2a82730b7a36d1c4cff6 by Maik Qualmann.
Committed on 16/08/2021 at 20:09.
Pushed by mqualmann into branch 'master'.

use album modification date to speed up the initial scan
Database update from V13 to V14 is carried out, the
column "modificationDate" is added to the Albums table.
Related: bug 405235, bug 431037, bug 265241

M  +14   -4    core/data/database/dbconfig.xml.cmake.in
M  +1    -1    core/libs/database/collection/collectionscanner.h
M  +5    -0    core/libs/database/collection/collectionscanner_p.cpp
M  +53   -16   core/libs/database/collection/collectionscanner_scan.cpp
M  +45   -0    core/libs/database/coredb/coredb.cpp
M  +19   -1    core/libs/database/coredb/coredb.h
M  +9    -1    core/libs/database/coredb/coredbschemaupdater.cpp

https://invent.kde.org/graphics/digikam/commit/d14487fc0da4e26bc7ba2a82730b7a36d1c4cff6
Comment 4 Maik Qualmann 2021-08-18 17:34:22 UTC
Git commit 9a443ffcae0f417d355a16e5040225f3a8e7075a by Maik Qualmann.
Committed on 18/08/2021 at 17:33.
Pushed by mqualmann into branch 'master'.

use album date cache to reduce file reading
Related: bug 405235, bug 431037, bug 265241

M  +2    -0    core/libs/database/collection/collectionscanner_p.h
M  +37   -3    core/libs/database/collection/collectionscanner_scan.cpp
M  +1    -1    core/libs/database/utils/scan/scancontroller_progress.cpp
M  +10   -2    core/libs/database/utils/scan/scancontroller_start.cpp

https://invent.kde.org/graphics/digikam/commit/9a443ffcae0f417d355a16e5040225f3a8e7075a
Comment 5 caulier.gilles 2022-01-09 15:20:47 UTC
Clara,

Please give us a fresh feedback using current 7.5.0 pre-release installer for Windows available here :

https://files.kde.org/digikam/

It will includes all last changes from Maik listed in this file.

Thanks in advance and happy new year

GilleS Caulier
Comment 6 caulier.gilles 2023-05-01 07:53:04 UTC
@clara

digiKam 8.0.0 is out. This entry still valid with this release ?

Best regards

Gilles Caulier
Comment 7 caulier.gilles 2023-10-15 10:10:37 UTC
@clara,


This problem still reproducible with the new digiKam 8.2.0 pre-release Windows
installer available at usual place:

https://files.kde.org/digikam/

This new bundle is based on last Qt framework 5.15.11 and KDE framework 5.110.

Thanks in advance

Gilles Caulier