First of all a happy new year and a big thank you to the Digikam developers for providing another Beta version :-) I started working with this recent and again feature- and bug-fix filled Beta-2. Unfortunately I found some issues with respect to performance, which makes Digikam nearly unusable on my Windows system. From my perception it seems related to database, therefore I chose this Component for this Bug. Please correct, if wrong. SUMMARY Digikam 7.2.0 beta-2 is very slow - when applying tags to a set of pictures or - refreshing an album folder or - importing new pictures (the progress of "finding new pictures in folder XYZ" after the import window was closed is felt to never finish) During each of such process the software is quite unresponsive and frequently appears to be frozen (showing a white application window). It is hardly possible to apply further actions on photos during processing the current task. This renders the software nearly unusable in such step of the workflow - or in other words, you really need paaaaaatience ;-) Examples: When applying a tag to pictures it takes some tenth of seconds until this action is done. Applying meta data to files is set to "lazy synchronisation". Refreshing a folder with 20 JPEG files and a sub-folder containing 26 RAW files really takes minutes to complete. EXPECTED RESULT I hope that you are able to duplicate this and make the application responsive again. I will be happy to provide you with any logs needed to debug this on your side. Just kindly tell me, which steps to do in which application configuration. SOFTWARE/OS VERSIONS Digikam: 7.2.0 Beta-2 Windows: Windows 10 Pro x64 (2004 build 19041.685) ADDITIONAL INFORMATION Network: Wi-Fi 5 / 802.11ac with a latency of 1..2ms in ping, so I assume that this won't be a bottle-neck Database: MySQL Server using a network hosted mariadb 10.5.4 Storage: NAS using SMB3 and allowing high data transfer rates (from my point of view, this shouldn't be the bottleneck either)
Happy new year. Previously, do you have used a stable release in the same database environment without time latency ? Gilles Caulier
Give us a log from DebugView. Don't forget to set the environment variable as described here: https://www.digikam.org/contribute/ Maik
@Gilles: I went from Beta-1 to Beta-2 and performance regarding tagging decreased remarkable. Performance when adding a new folder or refreshing an existing was never sooooo fine, also with the last stable release. So I added this to the request anyway. @Maik: I did some logging, please find the files attached containing some comments regarding the actions that I did. I also attached two screenshots of the progress bar, that keeps showing up for quite a long time after import (progressbar01.png) and even after import finished (progressbar02.png).
Created attachment 134442 [details] Logging
Created attachment 134443 [details] progressbar screenshot 01
Created attachment 134444 [details] progressbar screenshot 02
I don't see error messages or other malfunctions in the log. The scan times from the collection scanner make me suspect that the network drive (Z:\) is too slow. I cannot judge how fast the database is from the log. As a test you could include a collection with a UNC path (\\server\share). Otherwise your WiFi connection is too slow, check the signal strength and current connection speed. Remember that WiFI is always a bottleneck, no matter which version of WiFi you are using. If the database (loading thumbnails etc.) and the drive Z:\ is running via this WiFi connection, it becomes very tight. Please note that we have to open the files several times to read them in order to use Exiv2 to fetch the metadata, create a thumbnail etc. Maik
Hi Maik, thanks for your quick reply (as always ;-). I will do some testing on my side and come back with the results. Apart from WiFi, etc. I'm still wondering, why applying tags (not immediately syncing with the files, since I'm using lazy synchronization) to photos is significantly slower in Beta-2 than it was with Beta-1. Bye, Lars.
Sorry that it took some time to come back with my results from testing, but spare-time was quite limited during the last weeks ;-) I did some research and updated my mariadb installation from 10.5.4 (which indeed had some performance issues) to 10.5.8 (in which those issues were fixed according to the release notes). After the database update, I ran some performance tests with sysbench 1.0.18 (LuaJIT 2.1.0-beta3) against a mariadb test instance of 10.000 rows: As an average I got - 37 transactions per second - 747 queries per second I also checked my network performance with iperf3 between the Windows client and the Linux server running mariadb and the NAS storage. iperf3 showed me the following numbers: Ethernet (1GBit LAN) ==================== 1 stream -> 894 MBit/s 2 stream -> 918 MBit/s (gain +3%) WiFi (802.11ac, 2*2 MIMO) ========================= 1 stream -> 257 MBit/s 2 stream -> 375 MBit/s (gain +45%) 3 stream -> 406 MBit/s (gain +8%) 4 stream -> 423 MBit/s (gain +4%) My WiFi router is capable of 430MBit/s net data rate, when using AC 2*2 MIMO. I also did some more testing with 7.2.0 beta1 (WiFi+LAN) and 7.2.0 beta2 (LAN) to get some more performance numbers. My test procedure was as follows: - start Digikam with an empty (=no albums) database 1. set a network folder containing 18 raw (with sidecars) 18 JPG (without sidecars) 2. Having lazy sync set, assign two tags to 18 JPG pictures 3. Sync the files - quit the software - reset the database (drop and re-create) Please kindly have a look at the logs that I uploaded. The summary is as follows: 7.2.0-beta1 WIFI ================ 1. ends 704.81 (complete scan 595,426s) 2. assign tags 1st tag start: 781.98 (took 62,68s) 2nd tag start: 878.55 (took 56,4s) 3. start: 1082 (took 12,71s) 7.2.0-beta1 LAN =============== 1. ends 384.55 (complete scan 298,478s) 2. assign tags 1st tag start: 718.71 (took 46,18s) 2nd tag start: 865.27 (took 42,26s) 3. start: 985 (took 4,26s) 7.2.0-beta2 LAN =============== 1. ends 471.56 (complete scan 392,204s) 2. assing tags 1st tag start: 528.07 (took 54,7s) 2nd tag start: 632.21 (took 21s) 3. start: 670.52 (took 4,26s) That data given, I would like to ask if you think that the performance is normal / to be expected for that configuration of database, WiFi and LAN. Or if you see something that might be optimized to increase performance. Thanks in advance for all your work and time you put into this nice project and my call in special. Bye, Lars. P.S.: I didn't do a test using a unc path, since from my experience using the unc path is typically less performant under Windows than using a network drive.
Created attachment 135306 [details] 7.2.0 beta1 WiFi
Created attachment 135307 [details] 7.2.0 beta1 LAN
Created attachment 135308 [details] 7.2.0 beta2 LAN
Git commit 03f233383d94c20099afe07e75edb727694e5c3c by Maik Qualmann. Committed on 09/08/2021 at 19:29. Pushed by mqualmann into branch 'master'. first step for a faster scan of network drives We only run through a network collection once in the first scan to determine the number of files. The possible number of images and albums is now taken from the database. This can be a little less precise under certain circumstances, we will test it. Related: bug 405235, bug 437771, bug 265241 M +1 -1 core/libs/database/collection/collectionscanner.h M +11 -13 core/libs/database/collection/collectionscanner_scan.cpp M +25 -4 core/libs/database/collection/collectionscanner_utils.cpp M +33 -0 core/libs/database/coredb/coredb.cpp M +5 -0 core/libs/database/coredb/coredb.h https://invent.kde.org/graphics/digikam/commit/03f233383d94c20099afe07e75edb727694e5c3c
Git commit d14487fc0da4e26bc7ba2a82730b7a36d1c4cff6 by Maik Qualmann. Committed on 16/08/2021 at 20:09. Pushed by mqualmann into branch 'master'. use album modification date to speed up the initial scan Database update from V13 to V14 is carried out, the column "modificationDate" is added to the Albums table. Related: bug 405235, bug 437771, bug 265241 M +14 -4 core/data/database/dbconfig.xml.cmake.in M +1 -1 core/libs/database/collection/collectionscanner.h M +5 -0 core/libs/database/collection/collectionscanner_p.cpp M +53 -16 core/libs/database/collection/collectionscanner_scan.cpp M +45 -0 core/libs/database/coredb/coredb.cpp M +19 -1 core/libs/database/coredb/coredb.h M +9 -1 core/libs/database/coredb/coredbschemaupdater.cpp https://invent.kde.org/graphics/digikam/commit/d14487fc0da4e26bc7ba2a82730b7a36d1c4cff6
Git commit 9a443ffcae0f417d355a16e5040225f3a8e7075a by Maik Qualmann. Committed on 18/08/2021 at 17:33. Pushed by mqualmann into branch 'master'. use album date cache to reduce file reading Related: bug 405235, bug 437771, bug 265241 M +2 -0 core/libs/database/collection/collectionscanner_p.h M +37 -3 core/libs/database/collection/collectionscanner_scan.cpp M +1 -1 core/libs/database/utils/scan/scancontroller_progress.cpp M +10 -2 core/libs/database/utils/scan/scancontroller_start.cpp https://invent.kde.org/graphics/digikam/commit/9a443ffcae0f417d355a16e5040225f3a8e7075a
Hi, I find digikam 7.3 slow at scanning photo collection as well. I am using wifi, but I am sure Picasa would do this much quicker. Will do a test and post the result. My last scan was done after the NAS was mounted to a new ip-address, but using the same drive letter p:. Digikam did not understand that it is the same collection and I had to rescan. I that in the discussion it is mentioned that "we have to open the files several times to read them in order to use Exiv2 to fetch the metadata, create a thumbnail etc." Is that statement valid in my case as well where the thumbnails already exist? Should be mentioned that the thumbnails lives on my laptop. If still valid, does not digikam maintain some kind of marker/tag/identifier between the thumbnails and the original photos?
@LarsE digiKam 8.0.0 is out. This entry still valid with this release ? Best regards Gilles Caulier
Thanks for the reminder. Unfortunately I changed my data workflow in the meantime and all photos are accessed locally now. Therefore digikam is working very fast which puts much joy into my processing workflow now :-). I use the NAS solely for backup purposes and not for a direct access anymore. So I'm no longer of any help with respect to validation of any changes regarding this issue. I'm sorry. Perhaps @dajomu may help, since they also experienced a slow data access as mentioned in comment #16 Bye & thanks so much for all effort you and the dev team are putting into this really highly valuable software. I deeply appreciate that. Lars.
@dajomu, This problem still reproducible with the new digiKam 8.2.0 pre-release Windows installer available at usual place: https://files.kde.org/digikam/ This new bundle is based on last Qt framework 5.15.11 and KDE framework 5.110. Thanks in advance Gilles Caulier
@dajomu,, digiKam 8.2.0 for Windows is now compiled in native under VCPKG + MSVC toolchain. Installer is available here : https://files.kde.org/digikam/ Can you reproduce your dysfunctions with this version ? Thanks in advance Gilles Caulier