Bug 437771 - scanning performance much too slow
Summary: scanning performance much too slow
Status: REPORTED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Scan (show other bugs)
Version: 7.1.0
Platform: Microsoft Windows Microsoft Windows
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-05-28 11:15 UTC by Clara Spielmann
Modified: 2023-10-15 10:10 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Clara Spielmann 2021-05-28 11:15:02 UTC
SUMMARY


STEPS TO REPRODUCE
1. First time scanning of 135.000 / 500 GB of images. 
2. CPU Intel Intel Core i5-8250U
3. 

OBSERVED RESULT
It takes days with a CPU rate of around 3 %

EXPECTED RESULT
It should be much faster, there should be an option to disable CPU limitation to such a low percentage.
PROBLEM: If you don't request performance, CPU stays in low frequency and the whole process takes even longer. This behaviour is unbearable.

SOFTWARE/OS VERSIONS
Windows: Windows 10 21H1
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

ADDITIONAL INFORMATION
In contrary to other bug reports to this subject, I want to express that thew scanning takes too few system resources to do its job.
Comment 1 Maik Qualmann 2021-05-28 11:56:19 UTC
There is no CPU limit on the first scan.
Which database type are you using (MySQL / SQLite)
Where is the database located? (Local, network)
Where is the collection located? (Local SSD / HDD, USB, network)

The digiKam version used is no longer up-to-date.

Maik
Comment 2 Maik Qualmann 2021-08-09 19:32:09 UTC
Git commit 03f233383d94c20099afe07e75edb727694e5c3c by Maik Qualmann.
Committed on 09/08/2021 at 19:29.
Pushed by mqualmann into branch 'master'.

first step for a faster scan of network drives
We only run through a network collection once in
the first scan to determine the number of files.
The possible number of images and albums is now
taken from the database.
This can be a little less precise under certain
circumstances, we will test it.
Related: bug 405235, bug 431037, bug 265241

M  +1    -1    core/libs/database/collection/collectionscanner.h
M  +11   -13   core/libs/database/collection/collectionscanner_scan.cpp
M  +25   -4    core/libs/database/collection/collectionscanner_utils.cpp
M  +33   -0    core/libs/database/coredb/coredb.cpp
M  +5    -0    core/libs/database/coredb/coredb.h

https://invent.kde.org/graphics/digikam/commit/03f233383d94c20099afe07e75edb727694e5c3c
Comment 3 Maik Qualmann 2021-08-16 20:11:54 UTC
Git commit d14487fc0da4e26bc7ba2a82730b7a36d1c4cff6 by Maik Qualmann.
Committed on 16/08/2021 at 20:09.
Pushed by mqualmann into branch 'master'.

use album modification date to speed up the initial scan
Database update from V13 to V14 is carried out, the
column "modificationDate" is added to the Albums table.
Related: bug 405235, bug 431037, bug 265241

M  +14   -4    core/data/database/dbconfig.xml.cmake.in
M  +1    -1    core/libs/database/collection/collectionscanner.h
M  +5    -0    core/libs/database/collection/collectionscanner_p.cpp
M  +53   -16   core/libs/database/collection/collectionscanner_scan.cpp
M  +45   -0    core/libs/database/coredb/coredb.cpp
M  +19   -1    core/libs/database/coredb/coredb.h
M  +9    -1    core/libs/database/coredb/coredbschemaupdater.cpp

https://invent.kde.org/graphics/digikam/commit/d14487fc0da4e26bc7ba2a82730b7a36d1c4cff6
Comment 4 Maik Qualmann 2021-08-18 17:34:22 UTC
Git commit 9a443ffcae0f417d355a16e5040225f3a8e7075a by Maik Qualmann.
Committed on 18/08/2021 at 17:33.
Pushed by mqualmann into branch 'master'.

use album date cache to reduce file reading
Related: bug 405235, bug 431037, bug 265241

M  +2    -0    core/libs/database/collection/collectionscanner_p.h
M  +37   -3    core/libs/database/collection/collectionscanner_scan.cpp
M  +1    -1    core/libs/database/utils/scan/scancontroller_progress.cpp
M  +10   -2    core/libs/database/utils/scan/scancontroller_start.cpp

https://invent.kde.org/graphics/digikam/commit/9a443ffcae0f417d355a16e5040225f3a8e7075a
Comment 5 caulier.gilles 2022-01-09 15:20:47 UTC
Clara,

Please give us a fresh feedback using current 7.5.0 pre-release installer for Windows available here :

https://files.kde.org/digikam/

It will includes all last changes from Maik listed in this file.

Thanks in advance and happy new year

GilleS Caulier
Comment 6 caulier.gilles 2023-05-01 07:53:04 UTC
@clara

digiKam 8.0.0 is out. This entry still valid with this release ?

Best regards

Gilles Caulier
Comment 7 caulier.gilles 2023-10-15 10:10:37 UTC
@clara,


This problem still reproducible with the new digiKam 8.2.0 pre-release Windows
installer available at usual place:

https://files.kde.org/digikam/

This new bundle is based on last Qt framework 5.15.11 and KDE framework 5.110.

Thanks in advance

Gilles Caulier