SUMMARY Adding a tag to many photos causes several minutes of HDD activity - already after the process of writing metadata is finished STEPS TO REPRODUCE 1. Select 1000 photos in the thumbnails view 2. Check a tag in the Captions/Tags view 3. Apply OBSERVED RESULT The tag is added to the photos, the process takes about 5-10 seconds for 1000 photos. After that there's still a lot of HDD activity, scrolling through the thumbnails is very slow. Task manager shows a CPU load of 16% (on a 4c/8t CPU) and around 80MB/s read activity for digikam. Write activity on mysql is around 1.5MB/s. To the user this seems like digikam is regenerating the thumbnails for those photos (not just adding a tag to them). This goes on for around 5-10 minutes after which digikam behaves normally. EXPECTED RESULT After the process of adding metadata, HDD activity should come to an end immediately. SOFTWARE/OS VERSIONS Linux/KDE Plasma: Debian Testing Kernel 5.9.0-5-amd64 (available in About System) KDE Plasma Version: 5.20.4 KDE Frameworks Version: 5.77.0 Qt Version: 5.15.2 ADDITIONAL INFORMATION Running on i7-4790K (4xc/8t), 32GB of RAM. digikam RAM-usage is < 1GB during the whole operation. mySQL-DB is on a separate drive (M2-SSD).
By experience, using a database on a USB removable media give a poor performance. I suspect that database update continue in background due to USB bottleneck... Gilles Caulier
The database is on a M2-SSD, not USB:-) The problem is that the photo-HDD (via 6GBit SATA) is so high and takes considerably longer than digikam writes metadata (at least considering the progress bars). As stated in the bug report, it seems the thumbnails are regenerated as the CPU load is exactly 1/8th of total - typically meaning that a single thread is fully saturated.
Yes, of course new thumbnails will be created. When you write metadata in images, the file size changes a little. The Collection Scanner must assume that the image data may have changed and creates a new thumbnail. So basically a normal and correct behavior of digiKam. What can we do? My idea would be that we update the new file size in the DB immediately for certain metdata operations that do not affect the thumbnail, image size, etc. Maik
Changing the file size wouldn't be enough and was a bad idea. The unique hash changes and with it the assignment in the thumbnail DB. Maik
I cannot reproduce the problem here. We have 2 progress bars, the first is the process that writes the tags to the DB (goes quickly). The second progress bar writes the tags into the images, and the thumnails are created immediately. I have no "lagging" hard drive activity here. Maybe it would be good to upload a debug log from the terminal. Do not forget to set the environment variable, as described here: https://www.digikam.org/contribute/ Maik
Created attachment 134649 [details] ZIP file of a session log 1) Redirected stderr to log file and started digikam 2) Took a set of 80 photos (all jpg) and added tag "test" to them 3) Waited for HDD light to go out (after around 1 min.) and clodsed digikam
Made a log file (just uploaded). I start to understand - the metadata is not only written to the sidecar xmp but also to the JPG? Can I turn this off somewhere? Metadata in sidecars to me seems to be enough - no need to also have them in the image files themselves.
Additional note: The total of those 80 files is 1.6GB, so it's 20MB per image. So maybe it needs quite big files (as for a JPG) to really notice the problem.
I can no longer confirm your debug log with the digiKam-7.2.0-RC. The problem may already have been resolved as we removed the pre-generation of thumbnails for example. You can test the digiKam-7.2.0-RC as an AppImage from here: https://files.kde.org/digikam/ It will perform a database update, so keep this in mind if you want to revert to an older version. Writing only in sidecars can be set in digiKam Settings-> Metadata-> Sidecars. Maik
Created attachment 134653 [details] Log file of 7.2.0RC session Maik, thanks a lot for your very valuable help. I have checked the setting in 7.1.0 and it says "Write to XMP sidecar only". Using md5sum I confirmed that the JPGs are not altered, the checksums stay the same. I downloaded and tried 7.2.0RC, effect is exactly the same. There is a lot of HDD activity (and digikam-UI-lag) after the two processes have finished. I have included the log-file. Is there anything more I can run on my machine that helps identifying a setting that causes this behaviour? Thanks, Elmar
@elmar digiKam 8.0.0 is out. This entry still valid with this release ? Best regards Gilles Caulier
I downloaded the DigiKam 8.0.0 Appimage but it does not run on a copy of my DigiKam DB saying Failed to update the database schema from version 12 to version 13. The cause could be a missing upgrade of the database to the current server version. Now start digiKam again to perform a required upgrade of the database Restarting digiKam gives me the same error message. I'm on Debian 11 which has digiKam 7.1.0 packaged with it.
Maik, I have a similar problem with the current MacOS package build today. Until now, sqlite database work as expected using previous build of 8.1.0 pre-release (one week ago i think). Now, digiKam refuse to start with this message : https://i.imgur.com/MJU8k7I.png I don't rebuild macports environment since a while. This is the same used to release 7.10.0 and 8.0.0. Gilles
With one more byte per character, we now reach the maximum length of LONGTEXT faster with a long path. This is bad... Maik
This is not correct, LONGTEXT can be 4GB, it exceeds the FOREIGN KEY. Maik
Git commit c6cccb739a4e470920c2b1f5972793d2f05455ca by Maik Qualmann. Committed on 04/05/2023 at 20:51. Pushed by mqualmann into branch 'master'. fix FOREIGN KEY error with update to 17 M +40 -0 core/data/database/dbconfig.xml.cmake.in https://invent.kde.org/graphics/digikam/commit/c6cccb739a4e470920c2b1f5972793d2f05455ca
Git commit c3f8fd9a6119a5fb504383e825ffbe55555f7778 by Maik Qualmann. Committed on 04/05/2023 at 21:08. Pushed by mqualmann into branch 'master'. fix FOREIGN KEY error with update to 4 M +40 -0 core/data/database/dbconfig.xml.cmake.in https://invent.kde.org/graphics/digikam/commit/c3f8fd9a6119a5fb504383e825ffbe55555f7778
Hi Maik, I just recompiled the 8.1.0 PKG, and problem still the same: digikam.databaseserver: Database Parameters: Type: "QMYSQL" DB Core Name: "digikam" DB Thumbs Name: "digikam" DB Face Name: "digikam" DB Similarity Name: "digikam" Connect Options: "UNIX_SOCKET=/Users/gilles/Library/Application Support/digikam/digikam/db_misc/mysql.socket" Host Name: "" Host Port: -1 WAL Mode: false Internal Server: true Internal Server Path: "/Users/gilles/Pictures" Internal Server Admin Cmd: "/Applications/digiKam.org/digikam.app/Contents/lib/mariadb/bin/mysqladmin" Internal Server Serv Cmd: "/Applications/digiKam.org/digikam.app/Contents/lib/mariadb/bin/mysqld" Internal Server Init Cmd: "/Applications/digiKam.org/digikam.app/Contents/lib/mariadb/bin/mysql_install_db" Username: "root" Password: "" digikam.databaseserver: Internal Server data path: "/Users/gilles/Pictures/.mysql.digikam/db_data" digikam.databaseserver: The mysql configuration was already up-to-date: "/Users/gilles/Library/Application Support/digikam/digikam/mysql.conf" digikam.databaseserver: Database server: "/Applications/digiKam.org/digikam.app/Contents/lib/mariadb/bin/mysqld" ("--defaults-file=/Users/gilles/Library/Application Support/digikam/digikam/mysql.conf", "--datadir=/Users/gilles/Pictures/.mysql.digikam/db_data", "--basedir=/Applications/digiKam.org/digikam.app/Contents//lib/mariadb/", "--socket=/Users/gilles/Library/Application Support/digikam/digikam/db_misc/mysql.socket") digikam.databaseserver: Internal database server started digikam.databaseserver: Running 0 seconds... digikam.dbengine: Loading SQL code from config file "/Applications/digiKam.org/digikam.app/Contents/Resources/digikam/database/dbconfig.xml" digikam.dbengine: Checking XML version ID => expected: 3 found: 3 digikam.coredb: Core database: running schema update digikam.coredb: Core database: have a structure version 16 digikam.coredb: Core database: makeUpdates 16 to 17 digikam.dbengine: Failure executing query: "" Error messages: "QMYSQL: Unable to execute query" "Specified key was too long; max key length is 767 bytes" "1071" 2 Bound values: () digikam.dbengine: Error while executing DBAction [ "UpdateSchemaFromV16ToV17" ] Statement [ "ALTER TABLE Albums MODIFY COLUMN relativePath LONGTEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL;" ] digikam.coredb: Core database: schema update to V 17 failed! digikam.coredb: Core database: cannot process schema initialization digikam.databaseserver: Running 30 seconds... digikam.general: Allowing a cache size of 400 MB digikam.databaseserver: Shutting down database server digikam.databaseserver: Internal database server stopped QThreadStorage: Thread 0x60000024c2a0 exited after QThreadStorage 9 destroyed Gilles
OK, problem is clear. Index storage is 767 bytes == 255 characters * 3 bytes = 765 bytes. If we switch to utf8mb4 we have 255 characters * 4 bytes = 1020 bytes. We could then only use an index of 191 characters, but this could then cause problems with long file paths, which are then no longer unique. I will completely undo this database update first. Maik
Git commit 56a9195583a9938a931abcfa7ea67742b9d14ce3 by Maik Qualmann. Committed on 05/05/2023 at 06:12. Pushed by mqualmann into branch 'master'. revert "perform database update to switch to utf8m" Related: bug 469310 M +4 -22 core/data/database/dbconfig.xml.cmake.in M +1 -9 core/libs/database/coredb/coredbschemaupdater.cpp M +1 -20 core/libs/database/thumbsdb/thumbsdbschemaupdater.cpp M +0 -1 core/libs/database/thumbsdb/thumbsdbschemaupdater.h https://invent.kde.org/graphics/digikam/commit/56a9195583a9938a931abcfa7ea67742b9d14ce3
Maik, Problem is now fixed on my macbook Pro with your last commit. @elmar Tomorrow I will rebuild the 8.1.0 pre-release bundles with last Maik fixes. I will inform you when the AppImage file will be ready to test on your computer. Best Gilles Caulier
@elmar digiKam 8.1.0 pre-release including last Maik fixes is available as AppImage bunde here: https://files.kde.org/digikam/digiKam-8.1.0-20230508T132004-x86-64.appimage.mirrorlist Gilles Caulier
@elmar, Any feedback here ? Gilles Caulier
@elmar, What's about this file using current 8.2.0 AppImage Linux bundle ? It's reproducible ? https://files.kde.org/digikam/ Thanks in advance Gilles Caulier