SUMMARY In the Albums View, when I have a lot of images with Comments, Geolocation Data and Tags and Press Cntrl-S in order to save the data to the image files, this randomly deletes some of theses information from the database and the image files. This happens since v 8.0. Switching back to v 7.10 helps. STEPS TO REPRODUCE 1. load an album with 100+ images and add Geolocation, Comments and Tags. 2. in the albums main view press CNTRL-S to save the meta data to the image files. OBSERVED RESULT Some of the files loose the meta data in the view and also from the image files. EXPECTED RESULT All meta data is written to the image files and remains in the database SOFTWARE/OS VERSIONS Windows: Windows 10 Pro 22H2 build 19045.4651 ADDITIONAL INFORMATION I noticed that behaviour for the first time when upgrading from v7.10 to v8.2. It was reproducible on v8.3 and is on v.8.4. Downgrading to v7.10 helps. As I am using the German local, some of the buttons, menues or windows might be called slightly different in English.
If you add metadata, such as GPS, tags or comments, these are already written into the images. What should "Ctrl+S" do? This is not a standard key in digikam. Maik
Ah, sorry, "Ctrl+S" is the Shortcut for "write meta data to files" from the Albums menue. It should save all meta data to the files. Am 19.07.2024 um 11:35 schrieb Maik Qualmann: > https://bugs.kde.org/show_bug.cgi?id=490484 > > Maik Qualmann <metzpinguin@gmail.com> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |metzpinguin@gmail.com > > --- Comment #1 from Maik Qualmann <metzpinguin@gmail.com> --- > If you add metadata, such as GPS, tags or comments, these are already written > into the images. What should "Ctrl+S" do? This is not a standard key in > digikam. > > Maik >
Please take a screenshot of your metadata settings. Are you using Sidecar? Maik
Created attachment 171774 [details] Metadata settings 1
Created attachment 171775 [details] Metadata settings 2
Created attachment 171776 [details] Metadata settings 3
Created attachment 171777 [details] Metadata settings 4
Created attachment 171778 [details] Metadata settings 5
Created attachment 171779 [details] Metadata settings 6
Hi. Could be the "size of XMP JPEG segment larger than 65535 bytes"? /Roberto
yes it is. This problem is already reported in bugzilla and Exiv2 shared library do not support yet this chunk size. Gilles Caulier
Ok, thanks, so is there a workaround for me, or do I better stay with v 7.10 until this gets fixed?
Why digiKam 7.10 ? The JPEG chunk size limitation exists since the while in Exiv2.
Because I have never encountered the problem on v7.10 working on the same images. Maybe it uses a differen version of exiv2?
yes sure, it will use an older Exiv2 version, but the chunk size problem as always existed in Exiv2. https://bugs.kde.org/show_bug.cgi?id=468830 https://github.com/Exiv2/exiv2/issues/1549 https://github.com/Exiv2/exiv2/issues/2150 Another possibility, is something wrong with the Exif makernotes specificity not yet managed by Exiv2. Makernotes are block in Exif where camera maker store private data not documented. Which camera generate your JPEG (corrupted). Can you share samples before and after corruption (use cloud web service please). And finaly, install DebugView from Microsoft and capture the digiKam log while corruption appear. Exiv2 can print instrutive messages here. Look this page for details : https://www.digikam.org/contribute/ Gilles Caulier
Ok, I'll try. This might take some time. The cameras I use are a Canon R7 and the one of my Xiaomi 11 Ultra. I'll observe the issue and come back with more details here, as soon as I have some.
We are currently mixing two problems here because Roberto mentioned the problem with the Exif segment size of 65535. This problem could be related to lazy synchronization (Verzögerter Metadaten-Abgleich verwenden). You should not simply write the metadata back, but use the write function in the status bar to synchronize the metadata. Maik
Created attachment 171875 [details] view of files in explorer with missing data
Created attachment 171876 [details] data still present in Albums view within digiKam
Created attachment 171877 [details] data back in the explorer view
Created attachment 171878 [details] other missing data in explorer
Created attachment 171879 [details] asian codeset in image description
Created attachment 171880 [details] Debug log v8.04
Created attachment 171881 [details] Debug log v7.10
So, here we go. Today I did the following: - I copied a folder with 619 images, that all have geolocations, desciptions and tags. - I started digiKam - I selected all images and removed the color tags and changed a star rating. digiKam said in the lower status bar of the Albums view that 619 files were waiting for update. - I did an "Albums - Write Meta data to Files". This took quite some time. - Looking at the files in the explorer, I see, that randomly descriptions got lost for 16 files. (see Problem_1.jpg "view of files in explorer with missing data" for an example) However, in the Albums view, the descriptions are still visible (se Problem_2.jpg "data still present in Albums view within digiKam" for an example, eg. the pic of the churche in the top left corner) The status bar still says 619 files waiting for update. - I tried saving the info from the church image "2024-07-27 14-45-23.jpg" to the file by doing an "Item - write meta data to file". This did not change anything in the file. - I quit digiKam, and it applied the outstanding changes of the meta data to the 619 files. Again, this took a long time. The thumbnails of the images in the Albums view disappeared for some of the images, before digiKam shut down. - Looking a the images in the explorer, I see, that the information for the files, where descriptions got lost, are back again (see Problem_3.jpg "data back in the explorer view"), but others have now lost tags or descriptions (see Problem_4.jpg "other missing data in explorer") or were written back in asian font set (see Problem_5.jpg "asian codeset in image description") I'll attach the debug log of the session (VIDAR_digikam804.LOG). I downgraded to digiKam 7.10 and repeated the scenario above - the writing the meta data to the files was significantly faster and none of the information, tags or ratings got lost, just as I expected. I'll also attach the debug log of this session (VIDAR_digiKam710.LOG).
Unfortunately, the logs were created without activating the internal debugging. The digiKam-8.4.0 shows the locked database problem, which should have been fixed. It is clear that it is slow and data is missing. What metadata action did you take exactly when the log was created? The digiKam-7.10 log shows a broken digiKam, many DLL files cannot be loaded, have you mixed different digiKam versions? Maik
Ah, I overlooked that and will repeat the procedure. The only changes to the meta data from the existing ones were, that I removed a color filter tag from approx 160 images and changed the star rating for one. When downgrading from 8.04 to 7.10 the installer does not ask to remove a previous installation, as it does when upgrading. Thus it is likely that there are files in the directories that came with the 8.0x installations. They do however not seem to do any harm to the 7.10 installation.
Please clean up the digiKam directory in "Program Files\digikam" manually. I cannot reproduce the locked database problem here, there is only the expected "zero" lock, after which we reset the database for the transaction. The problem cannot occur with digiKam-8.4.0. The occurrence of Asian characters in comments is also a problem of digiKam-7.x.x. Maik
Created attachment 171884 [details] Problem_6
Created attachment 171885 [details] Problem_7
Created attachment 171886 [details] Debug log v8.04 now really
2nd try: - I copied a folder with 619 images, that all have geolocations, desciptions and tags. - I upgraded to 8.04 again - I started digiKam and enabled internal debug logging - I selected all images and removed the color tags (CNRTL+ALT+1) and changed a single star rating. digiKam said in the lower status bar of the Albums view that 619 files were waiting for update. - I did an "Albums - Write Meta data to Files". - Looking at the files in the explorer, I see, that randomly descriptions got lost for 7 different files. (see Problem_6.jpg) Again, in the Albums view, the descriptions are still visible. The status bar still says 619 files waiting for update. - I quit digiKam, and it applied the outstanding changes of the meta data to the 619 files. - Looking a the images in the explorer, I see, that now 8 other files have lost the description and 3 were translated to asian (see Problem_7.jpg). I'll attach the debug log of the session (VIDAR_digikam804_debug.LOG).
Git commit f40c39b98ea66f8f323146f887f6e8cdd00a8446 by Maik Qualmann. Committed on 22/07/2024 at 10:47. Pushed by mqualmann into branch 'master'. same error handling as in query batch mode for a single query M +13 -0 core/libs/database/engine/dbenginebackend.cpp https://invent.kde.org/graphics/digikam/-/commit/f40c39b98ea66f8f323146f887f6e8cdd00a8446
Created attachment 171887 [details] Debug log v8.04 after reinstall
- I uninstalled digiKam 8.4 via the Windows Control Panel (Systemsteuerung) - I verfied, that there is no subdirectoy digikam in C:\Program Files - In reinstalled digiKam. Btw., I got the binary from https://mirrors.xtom.de/kde/ - I verified, that now there is a subdirectoy digikam in C:\Program Files - I started digiKam, had lunch and repeated the steps from the previous comments, with exactly the same result. - I do get the database lock, as you can see from the debug.log (VIDAR_digikam804_debug_2.LOG)
Hi. Does enabling database WAL mode change anything? I had a similar bug when 8.3.0 was launched and part of the solution was enabling WAL... /Roberto On 2024-07-22 08:10, Mitch Rauth wrote: > https://bugs.kde.org/show_bug.cgi?id=490484 > > --- Comment #35 from Mitch Rauth <mitch@rauth.at> --- > - I uninstalled digiKam 8.4 via the Windows Control Panel (Systemsteuerung) > - I verfied, that there is no subdirectoy digikam in C:\Program Files > - In reinstalled digiKam. Btw., I got the binary from > https://mirrors.xtom.de/kde/ > - I verified, that now there is a subdirectoy digikam in C:\Program Files > - I started digiKam, had lunch and repeated the steps from the previous > comments, with exactly the same result. > - I do get the database lock, as you can see from the debug.log > (VIDAR_digikam804_debug_2.LOG) > /Roberto
The WAL mode improves the writing and reading behavior of the database, but the problem can still occur. Only the patch from Comment 33 will fix it. I hope Gilles still has time to create a Windows bundle, otherwise it will take some time. Maik
Hi all, Yes, this evening i will update the bundles. Gilles
Repeating the steps with WAL mode enabled did not delete any data anymore, although there were still database locked messages in the debug log. Thanks for all the help. I'll wait for the new binary. Mitch
Please test the digiKam-8.5.0 pre-release to see if the problem can still be reproduced. https://files.kde.org/digikam/ Maik
Created attachment 171898 [details] Problem_8
Created attachment 171899 [details] Debug log v8.05
I downloaded and installed digiKam-8.5.0-20240722T151954-Qt6-Win64.exe, upgrading from the 8.4 version - I started digiKam, disabler WAL mode, restarted. - I repeated the tests from above. - When doing a "Albums - write meta data to files", this took quite long, and I saw a lot of database locked messages in the debug window. I also observed, that the progress bar stayed at 2% and leap jumped to 100% at the end. The files in the explorer view were fine and not corrupted. - When I quit digiKam and it applied the changes to the files, this went smoother and the progress bar showed a normal behavior. However, when I checked the files in the explorer, I saw again, that some hat lost tags and descriptions (see Problem_8.jpg) I attached the debug log v8.05
One of the main problems is that you have 12 CPU cores. There is simply no more slot available for the database to store the data. I think we will generally limit the number of tasks on these multi-core processors. Maik
Git commit 3e387a3d137e97d730422b30ca398e69829382ec by Maik Qualmann. Committed on 22/07/2024 at 19:35. Pushed by mqualmann into branch 'master'. protect 2 more SQL functions for endless lock M +24 -2 core/libs/database/engine/dbenginebackend.cpp https://invent.kde.org/graphics/digikam/-/commit/3e387a3d137e97d730422b30ca398e69829382ec
Git commit 128d9c9fc73015309128f0863d7768e9e5fd3e0c by Maik Qualmann. Committed on 22/07/2024 at 20:29. Pushed by mqualmann into branch 'master'. limits the maximum number of threads used M +8 -1 core/libs/threads/actionthreadbase.cpp https://invent.kde.org/graphics/digikam/-/commit/128d9c9fc73015309128f0863d7768e9e5fd3e0c
@Gilles, if possible another Windows bundle update to get the latest changes. Best Maik
It will be done this morning Gilles
@Mitch Rauth, can you please test our latest digiKam-8.5.0 to see if there is any change? Maik
Hm, the latest build I see at https://files.kde.org/digikam/ is from 2004-07-23 16:34. Is this the right one?
hi Mitch yes it is ...
Created attachment 172000 [details] Debug log v8.05_0723 Unfortunately the problem is still reproducible. Doing a "Albums - Write Meta Data to Images" shows a progress bar at 3% until done, and the meta data in the files is not corrupted afterwards. The subsequent CNTRL+Q shows a normal progress bar, a lot of locks in the debug log, and the files are corrupted at the end.
Hi Maik, Note to 8.5.0 : Exiv2 is now the last stable 0.28.3 (compared to 8.4.0 where Exiv2 is 0.28.2). In case of the problem is relevant of metadata engine... Gilles
From the log : 00040637 08:51:26 [9592] digikam.dbengine: WARNING !!! Transaction count is -1 when destroying database!!! 00040638 08:51:26 [9592] digikam.dbengine: WARNING !!! Transaction count is -1 when destroying database!!! 00040639 08:51:26 [9592] digikam.dbengine: WARNING !!! Transaction count is -1 when destroying database!!! strange...
Something lock the database a lot : ... 00039533 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 2000 00039534 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 1750 00039535 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 2000 00039536 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 2000 00039537 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 7250 00039538 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 2000 00039539 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 2000 00039540 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 1750 00039541 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 2000 00039542 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 2250 00039543 08:51:23 [9592] digikam.dbengine: Database is locked. Waited 2250 ...
This one is more instructive and come plenty of time: 00038943 08:51:09 [9592] digikam.dbengine: Database is locked. Waited 5500 00038944 08:51:09 [9592] digikam.dbengine: Database is locked. Waited 10000 00038945 08:51:09 [9592] digikam.dbengine: Detected locked database file. There is an active transaction. Waited but giving up now. 00038946 08:51:09 [9592] digikam.dbengine: Failure executing query: 00038947 08:51:09 [9592] "SELECT orientation FROM ImageInformation WHERE imageid=:0;" 00038948 08:51:09 [9592] Error messages: "Der Datensatz konnte nicht abgeholt werden" "database table is locked: ImageInformation" "262" 1 00038949 08:51:09 [9592] Bound values: QList(QVariant(qlonglong, 132321)) 00038950 08:51:09 [9592] digikam.dbengine: Database is locked. Waited 250 00038951 08:51:09 [9592] digikam.dbengine: Database is locked. Waited 250 I suspect an anti-virus perturbing in the background when digiKam try to update the sqlite database... Where did you store the database files for your collections ? In local storage, a removable one, in your home directory ? Gilles
Hi Gilles, I am using a small Intel NUC PC with one single 1 TB SSD, running an ordinary windows 10, which has it's on board virus defender. The database is at: [9592] DB Core Name: "C:/Users/mitch/Pictures/DigiKam/digikam4.db" [9592] DB Thumbs Name: "C:/Users/mitch/Pictures/DigiKam/thumbnails-digikam.db" [9592] DB Face Name: "C:/Users/mitch/Pictures/DigiKam/recognition.db" [9592] DB Similarity Name: "C:/Users/mitch/Pictures/DigiKam/similarity.db" Thus local storage in my home directory.
Ok, nothing special here. My VM is very similar one running WIN10. About the internal SSD can your install smartmontools package and check the state of the media using smartctl ? https://en.wikipedia.org/wiki/Smartmontools This utility will read all the electronics device information and status and report dysfunction if any. To process, all is very well explained here : https://linuxconfig.org/how-to-check-an-hard-drive-health-from-the-command-line-using-smartctl Even if a SSD do not includes mechanics, depending of the quality of the device, it can be fail to work for multiple technical reasons. Best Gilles Caulier
We have often received this type of malfunction in a similar way (rotating images, for example). It is always the machines with > 8 CPU cores. 12 CPU cores are reported, but it can probably be 6 CPU cores, is that correct? Maik
Gilles, I'll try and check on the weekend. Maik, it is a nuc10fnh with 4 physical cores. Mitch
Git commit b911d13ae2834d3746ff608512b57a21ffd5c5a8 by Maik Qualmann. Committed on 26/07/2024 at 10:32. Pushed by mqualmann into branch 'master'. remove query reset for single queries, as it has no effect. M +0 -36 core/libs/database/engine/dbenginebackend.cpp https://invent.kde.org/graphics/digikam/-/commit/b911d13ae2834d3746ff608512b57a21ffd5c5a8
Git commit 8b54673a92b63a0a322d66bef0946ab1e2204d58 by Maik Qualmann. Committed on 26/07/2024 at 10:35. Pushed by mqualmann into branch 'master'. disable SQLite shared cache for a test as it is deprecated and no longer recommended. M +2 -2 core/libs/database/engine/dbenginebackend.cpp https://invent.kde.org/graphics/digikam/-/commit/8b54673a92b63a0a322d66bef0946ab1e2204d58
I do not know if this is important, but if I run the same test with v7.10 I do not have any problems. Thus it appears to me, that something changed to the major version 8.x, that introduced the issue.
The problem of the endless locked database has been around for a very long time, far ahead of digiKam-7.X.X. It depends on how the threads run together and access the database. Maik
Hi Gilles, here are the results from the smartmontools. Looks quite normal to me. C:\Program Files\smartmontools\bin>smartctl.exe -a /dev/sda smartctl 7.4 2023-08-01 r5530 [x86_64-w64-mingw32-w10-22H2] (sf-7.4-1) Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: TEAM T253X6001T Serial Number: TPBF2110010011300722 LU WWN Device Id: 0 000000 000000000 Firmware Version: SBFMJ1.3 User Capacity: 1.024.209.543.168 bytes [1,02 TB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches TRIM Command: Available Device is: Not in smartctl database 7.3/5528 ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sat Jul 27 16:52:19 2024 MS SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (65535) seconds. Offline data collection capabilities: (0x79) SMART execute Offline immediate. No Auto Offline data collection support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 30) minutes. Conveyance self-test routine recommended polling time: ( 6) minutes. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 13755 12 Power_Cycle_Count 0x0012 100 100 000 Old_age Always - 405 168 Unknown_Attribute 0x0012 100 100 000 Old_age Always - 0 170 Unknown_Attribute 0x0003 081 081 010 Pre-fail Always - 283 173 Unknown_Attribute 0x0012 100 100 000 Old_age Always - 4128863 192 Power-Off_Retract_Count 0x0012 100 100 000 Old_age Always - 15 194 Temperature_Celsius 0x0023 067 067 000 Pre-fail Always - 33 (Min/Max 33/33) 218 Unknown_Attribute 0x000b 100 100 050 Pre-fail Always - 1 231 Unknown_SSD_Attribute 0x0013 100 100 000 Pre-fail Always - 93 241 Total_LBAs_Written 0x0012 100 100 000 Old_age Always - 13237 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Conveyance offline Completed without error 00% 13755 - # 2 Extended offline Completed without error 00% 13754 - # 3 Extended offline Completed without error 00% 13750 - SMART Selective self-test log data structure revision number 0 Note: revision number not 1 implies that no selective self-test has ever been run SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. The above only provides legacy SMART information - try 'smartctl -x' for more C:\Program Files\smartmontools\bin>
Created attachment 172041 [details] real Debug log v7.10 Hi Maik, I downgraded to 7.10 again and repeated the test. You are right, there are 3 database lock messages in the debug log as well, but they only waited 0 ms. I have attached the 7.10 debug log. Mitch
yes the smartmontools reports sounds safe with your device...
Can you please test again with the current digiKam version? Maik
The difference to digiKam-7.x.x is that we now read the metadata again after writing it, which leads to a double load on the database. You can tell by the "Scanning took" entries that are now present. There may have been a reason for this, I have to check whether it is actually necessary. Maik
Git commit 341ffaa20f5d37775169d2b86f729c0014bb2fc7 by Maik Qualmann. Committed on 28/07/2024 at 08:57. Pushed by mqualmann into branch 'master'. do not rescan item after writing metadata Related: bug 490821 M +2 -2 core/libs/database/engine/dbenginebackend.cpp M +3 -14 core/libs/database/utils/ifaces/itemgps.cpp M +3 -20 core/libs/fileactionmanager/fileworkeriface.cpp M +1 -2 core/libs/properties/captions/itemdescedittab.cpp M +1 -2 core/libs/tags/engine/tagmodificationhelper.cpp M +1 -8 core/libs/threads/actionthreadbase.cpp M +3 -7 core/utilities/facemanagement/database/faceutils.cpp M +1 -2 core/utilities/imageeditor/main/imagewindow.cpp M +1 -3 core/utilities/maintenance/tools/autotags/autotagsassignmenttask.cpp M +1 -3 core/utilities/maintenance/tools/imgqsort/imagequalitytask.cpp M +1 -3 core/utilities/maintenance/tools/metaremover/metadataremovetask.cpp M +2 -5 core/utilities/maintenance/tools/metasync/metadatasynctask.cpp M +1 -3 core/utilities/queuemanager/manager/actionthread.cpp https://invent.kde.org/graphics/digikam/-/commit/341ffaa20f5d37775169d2b86f729c0014bb2fc7
Git commit 15101d696baf5491abb60be9013fadf3b5dd0b3c by Maik Qualmann. Committed on 28/07/2024 at 09:03. Pushed by mqualmann into branch 'master'. Revert "do not rescan item after writing metadata" Related: bug 490821 M +2 -2 core/libs/database/engine/dbenginebackend.cpp M +14 -3 core/libs/database/utils/ifaces/itemgps.cpp M +20 -3 core/libs/fileactionmanager/fileworkeriface.cpp M +2 -1 core/libs/properties/captions/itemdescedittab.cpp M +2 -1 core/libs/tags/engine/tagmodificationhelper.cpp M +8 -1 core/libs/threads/actionthreadbase.cpp M +7 -3 core/utilities/facemanagement/database/faceutils.cpp M +2 -1 core/utilities/imageeditor/main/imagewindow.cpp M +3 -1 core/utilities/maintenance/tools/autotags/autotagsassignmenttask.cpp M +3 -1 core/utilities/maintenance/tools/imgqsort/imagequalitytask.cpp M +3 -1 core/utilities/maintenance/tools/metaremover/metadataremovetask.cpp M +5 -2 core/utilities/maintenance/tools/metasync/metadatasynctask.cpp M +3 -1 core/utilities/queuemanager/manager/actionthread.cpp https://invent.kde.org/graphics/digikam/-/commit/15101d696baf5491abb60be9013fadf3b5dd0b3c
Created attachment 172068 [details] Debug log v8.05_0727 Hi Maik, I rerun the test with the digiKam-8.5.0-20240727T092720-Qt6-Win64.exe binary, and the meta data are NOT corrupted anymore. I see database locked messages in the debug.log, much more than with 7.10, but all with 0 ms and "no giving up" anymore. Yeah. One change to the previous build that I notived are Errors of the kind: 00005565 13:55:39 [25672] "DELETE FROM ImageTags WHERE imageID=:0 AND tagid=:1;" 00005566 13:55:39 [25672] Error messages: "Die Anzahl der Parameter ist falsch" "" "" 2 where in the 0723 build it says: 00014795 08:47:13 [9592] "DELETE FROM ImagePositions WHERE imageid=:0;" 00014796 08:47:13 [9592] Error messages: "Der Datensatz konnte nicht abgeholt werden" "database table is locked" "262" 1 I do not know, if this is important, as it has no visible impact to my workflow. I attach the current debug.log. Whatever you have done, this looks much better for me. Thanks a lot. Mitch
(In reply to Maik Qualmann from comment #71) > Git commit 15101d696baf5491abb60be9013fadf3b5dd0b3c by Maik Qualmann. > Committed on 28/07/2024 at 09:03. > Pushed by mqualmann into branch 'master'. > > Revert "do not rescan item after writing metadata" > Related: bug 490821 > > M +2 -2 core/libs/database/engine/dbenginebackend.cpp > M +14 -3 core/libs/database/utils/ifaces/itemgps.cpp > M +20 -3 core/libs/fileactionmanager/fileworkeriface.cpp > M +2 -1 core/libs/properties/captions/itemdescedittab.cpp > M +2 -1 core/libs/tags/engine/tagmodificationhelper.cpp > M +8 -1 core/libs/threads/actionthreadbase.cpp > M +7 -3 core/utilities/facemanagement/database/faceutils.cpp > M +2 -1 core/utilities/imageeditor/main/imagewindow.cpp > M +3 -1 > core/utilities/maintenance/tools/autotags/autotagsassignmenttask.cpp > M +3 -1 core/utilities/maintenance/tools/imgqsort/imagequalitytask.cpp > M +3 -1 > core/utilities/maintenance/tools/metaremover/metadataremovetask.cpp > M +5 -2 core/utilities/maintenance/tools/metasync/metadatasynctask.cpp > M +3 -1 core/utilities/queuemanager/manager/actionthread.cpp > > https://invent.kde.org/graphics/digikam/-/commit/ > 15101d696baf5491abb60be9013fadf3b5dd0b3c So, rescanning item after writing metadata *is* necessary? Could you elaborate a bit why? There is a metadata setting "Rescan file when files are modified" which I thought would control rescanning or not... /Roberto
There are different depths of scanning, the option "Rescan file when files are modified" will do a full scan as if the file is unknown to capture all changes. Without this option, the scan will not be as deep, just file date and size or similar. When metadata is written, we need to rescan the file as the file size changes and the unique file hash needs to be updated in the database. Otherwise, this process would only take place at the next start when the initial scan is active. In the meantime, the database would not be in sync with the files, e.g. an incorrect file size would be displayed in the GUI, etc. Maik
Git commit a7ec93df612cdb1a921e40e52794e81364820510 by Maik Qualmann. Committed on 28/07/2024 at 19:02. Pushed by mqualmann into branch 'master'. fix prepare a copied query and handle bind values correctly Under Qt5 the bind values were a QMap, in Qt6 it is a QVariantList. If we add a QVariantList we get a list in the list. To change this behavior we add the bind values one by one. Related: bug 490821 M +11 -11 core/libs/database/coredb/coredb.cpp M +40 -5 core/libs/database/engine/dbenginebackend.cpp M +8 -0 core/libs/database/engine/dbenginesqlquery.cpp M +1 -0 core/libs/database/engine/dbenginesqlquery.h M +1 -8 core/libs/threads/actionthreadbase.cpp https://invent.kde.org/graphics/digikam/-/commit/a7ec93df612cdb1a921e40e52794e81364820510
A new digiKam-8.5.0 pre-release version with the latest changes is available. Please test whether the problem can still be reproduced. https://files.kde.org/digikam/ Maik
Hi Maik, I retested, and could not reproduce the problem. However, I noticed, that the process took longer than with the previous build 20240724. In the debug log I sas more "database is locked. Waited 0" and a bunch of new "Retray to preare query" messages. I'll attach the debug file. Thus, prolem solved, performance is a bit degraded, but data do not get corrupted anymore. Thanks. One thing I do not really understand is, why digiKam says "No pending reconcillation of meta data" (if this is the correct translation of the info in the lower status bar of the main window), but when I quit digiKam it is reconcilling the meta data that were just written to the files before. But that does not hurt.
Created attachment 172929 [details] debug log v8.5.0 20240822
The multiple blocked database accesses in a row come from your many CPU threads, they all come across a database that is still busy at the same time and have to resend their request. But that doesn't really take much time. Using WAL mode optimizes the SQLite database even more. I wrote before that you write the metadata twice if you write the metadata manually beforehand and then use lazy synchronization. In that case, your manual writing of the metadata is unnecessary. We can think about deleting the waiting image IDs from the lazy synchronization waiting list if a manual writing of the metadata was forced. Thanks for the logs and feedback to solve the problem. Maik
You can try the l
Mitch, To test, you can try the last 8.5.0 build at https://files.kde.org/digikam/ Best GilleS Caulier
Git commit 148a70d2e1a7cdfb2948bbc27966624cfa8aa0f8 by Maik Qualmann. Committed on 25/08/2024 at 09:50. Pushed by mqualmann into branch 'master'. remove items from lazy sync when the metadata is written M +17 -0 core/libs/fileactionmanager/metadatahubmngr.cpp M +1 -0 core/libs/fileactionmanager/metadatahubmngr.h M +6 -0 core/utilities/maintenance/tools/metasync/metadatasynctask.cpp https://invent.kde.org/graphics/digikam/-/commit/148a70d2e1a7cdfb2948bbc27966624cfa8aa0f8
Git commit 2bf19295c307b0f4af53b9443fed4a2620771219 by Maik Qualmann. Committed on 25/08/2024 at 11:03. Pushed by mqualmann into branch 'master'. use signal/slot connection to remove pending lazy sync items M +8 -0 core/app/views/stack/itemiconview_albums.cpp M +8 -0 core/app/views/stack/itemiconview_items.cpp M +1 -0 core/app/views/stack/itemiconview_p.h M +1 -2 core/libs/fileactionmanager/metadatahubmngr.cpp M +1 -1 core/libs/fileactionmanager/metadatahubmngr.h M +3 -0 core/utilities/maintenance/manager/maintenancethread.cpp M +5 -0 core/utilities/maintenance/manager/maintenancethread.h M +3 -0 core/utilities/maintenance/tools/metasync/metadatasynchronizer.cpp M +4 -0 core/utilities/maintenance/tools/metasync/metadatasynchronizer.h M +2 -7 core/utilities/maintenance/tools/metasync/metadatasynctask.cpp M +1 -0 core/utilities/maintenance/tools/metasync/metadatasynctask.h https://invent.kde.org/graphics/digikam/-/commit/2bf19295c307b0f4af53b9443fed4a2620771219