SUMMARY *** After scaning/identifying faces on multiple pictures (many selected at same time), when trying to add the same tag(s) to all selected, face rectangles/other tags are gone in multiple images (not all). The same issue occurs for pictures which had people identified months ago... *** STEPS TO REPRODUCE 1. Select multiple images (I tested with 6 and the issue already showed up) with faces on them and/or other tags 2. Add/remove same tag on multiple images 3. Check that some images have been erased of their face rectangles/other tags OBSERVED RESULT Up to 8.2, it was perfectly possible to select multiple pictures to add/remove tags w/o any undesired effects. This is repeatable. EXPECTED RESULT From 8.3, faces/tags are randomly removed when you try to add/remove tags from multiple images at same time. I have tried with 3 different 8.3 "weekly snapshots" and the issue is veryfiable in all of them. SOFTWARE/OS VERSIONS Windows: 10.0.19045.4046 macOS: Linux/KDE Plasma: (available in About System) KDE Plasma Version: KDE Frameworks Version: Qt Version: ADDITIONAL INFORMATION Not sure if it is related, but up to 8.2, I could remove tags in the Tags tab w/o affecting any other tags/faces rectangles. Now, the same issue described above will ocurr if you remove tags from the Tags tab.
Moving back to 8.2 resolves the issue
1. Which database backend (SQLite/MySQL)? 2. Metadata is written into the images? 3. Is ExifTool used for writing? 4. Are sidecars used? Maik
1. Which database backend (SQLite/MySQL)? SQLlite 2. Metadata is written into the images? Yes. Checked with exiftool. And reading metadata from the image after the issue yields nothing as expected. 3. Is ExifTool used for writing? No. 4. Are sidecars used? No. /Roberto > On 21 Feb 2024, at 11:29, Maik Qualmann <bugzilla_noreply@kde.org> wrote: > > https://bugs.kde.org/show_bug.cgi?id=481630 > > Maik Qualmann <metzpinguin@gmail.com> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |metzpinguin@gmail.com > > --- Comment #2 from Maik Qualmann <metzpinguin@gmail.com> --- > 1. Which database backend (SQLite/MySQL)? > 2. Metadata is written into the images? > 3. Is ExifTool used for writing? > 4. Are sidecars used? > > Maik > > -- > You are receiving this mail because: > You reported the bug.
There are actually no changes in the relevant code parts compared to 8.2.0 that could cause this problem. Can you narrow it down to specific images or a specific camera/image format? Maybe a debug log from the terminal would be helpful if the problem occurs, as described here for macOS: https://www.digikam.org/contribute/ Maik
I will prepare to do that with latest weekly 8.3 snapshot. But... Running tests now in 2 albuns with half a dozen pictures (PNG and JPG) I could not prove metadata is being written to files. The face rectangles are still missing in some images after changing tags but reading the metadata recreates them. When I first stumbled on the issue I had changed tags in hundreds of files. So it can be that both occurs for I have checked a few images with exiftool and tags were missing on the files. Will need to test on larger albums with the nasty effect that I loose faces/tags that I need to either save somehow in beforehand or recreate manually :( Thanks for now. /Roberto On 2024-02-21 12:13, Maik Qualmann wrote: > https://bugs.kde.org/show_bug.cgi?id=481630 > > --- Comment #4 from Maik Qualmann <metzpinguin@gmail.com> --- > There are actually no changes in the relevant code parts compared to 8.2.0 that > could cause this problem. Can you narrow it down to specific images or a > specific camera/image format? Maybe a debug log from the terminal would be > helpful if the problem occurs, as described here for macOS: > > https://www.digikam.org/contribute/ > > Maik >
Was able to reproduce the error as described in the bug with an album of 27 pictures. It is a mix of situations: some pics you can retrieve faces by reading metadata, others you don't. I believe tags will disappear or not depending on your tags structure. This time, tags didn't disappear. After fixing all faces/tags I wrote metadata to all files, and to my surprise, faces disapeared in some pics and reading metadata didn't help. My settings are to write face tags incl face areas to file. All JPG coming from iPhone SE 3rd Gen (in this case). But sure affect other cameras since I had the problem with Canon camera images at least. Will work to get a debug log. /Roberto On 2024-02-21 12:57, Beto wrote: > I will prepare to do that with latest weekly 8.3 snapshot. > > But... > > Running tests now in 2 albuns with half a dozen pictures (PNG and JPG) I > could not prove metadata is being written to files. The face rectangles > are still missing in some images after changing tags but reading the > metadata recreates them. > > When I first stumbled on the issue I had changed tags in hundreds of > files. So it can be that both occurs for I have checked a few images > with exiftool and tags were missing on the files. > > Will need to test on larger albums with the nasty effect that I loose > faces/tags that I need to either save somehow in beforehand or recreate > manually :( > > Thanks for now. > > /Roberto > > On 2024-02-21 12:13, Maik Qualmann wrote: >> https://bugs.kde.org/show_bug.cgi?id=481630 >> >> --- Comment #4 from Maik Qualmann <metzpinguin@gmail.com> --- >> There are actually no changes in the relevant code parts compared to >> 8.2.0 that >> could cause this problem. Can you narrow it down to specific images or a >> specific camera/image format? Maybe a debug log from the terminal >> would be >> helpful if the problem occurs, as described here for macOS: >> >> https://www.digikam.org/contribute/ >> >> Maik >> >
Created attachment 165976 [details] 8.3 log file Attached log file. FYI, log entries 58-79 produced by multiple attempts to read metadata from 2 pics missing the face rectangle it had before. No success. It took me 4 passes to fix everything for faces continued to disappear every time I came back to check, some possible to read from file, others don't. This time some tags were also erased... /Roberto On 2024-02-21 13:20, Beto wrote: > Was able to reproduce the error as described in the bug with an album of > 27 pictures. It is a mix of situations: some pics you can retrieve faces > by reading metadata, others you don't. I believe tags will disappear or > not depending on your tags structure. This time, tags didn't disappear. > > After fixing all faces/tags I wrote metadata to all files, and to my > surprise, faces disapeared in some pics and reading metadata didn't > help. My settings are to write face tags incl face areas to file. > > All JPG coming from iPhone SE 3rd Gen (in this case). But sure affect > other cameras since I had the problem with Canon camera images at least. > > Will work to get a debug log. > > /Roberto > > On 2024-02-21 12:57, Beto wrote: >> I will prepare to do that with latest weekly 8.3 snapshot. >> >> But... >> >> Running tests now in 2 albuns with half a dozen pictures (PNG and JPG) >> I could not prove metadata is being written to files. The face >> rectangles are still missing in some images after changing tags but >> reading the metadata recreates them. >> >> When I first stumbled on the issue I had changed tags in hundreds of >> files. So it can be that both occurs for I have checked a few images >> with exiftool and tags were missing on the files. >> >> Will need to test on larger albums with the nasty effect that I loose >> faces/tags that I need to either save somehow in beforehand or >> recreate manually :( >> >> Thanks for now. >> >> /Roberto >> >> On 2024-02-21 12:13, Maik Qualmann wrote: >>> https://bugs.kde.org/show_bug.cgi?id=481630 >>> >>> --- Comment #4 from Maik Qualmann <metzpinguin@gmail.com> --- >>> There are actually no changes in the relevant code parts compared to >>> 8.2.0 that >>> could cause this problem. Can you narrow it down to specific images or a >>> specific camera/image format? Maybe a debug log from the terminal >>> would be >>> helpful if the problem occurs, as described here for macOS: >>> >>> https://www.digikam.org/contribute/ >>> >>> Maik >>> >>
You don't have debug mode enabled, but warnings still appear. The problem is clear: Cannot save metadata with Exiv2 backend: (Error # 38 : "Size of XMP JPEG segment is larger than 65535 bytes" This also occurs with digiKam-8.2.0. This is an Exiv2 issue of not supporting metadata entries longer than 65535 bytes. See also Bug 468830. It would not help to activate writing with ExifTool at the moment, since we create the metadata container with Exiv2. It would be good to have a sample image in question so that we can see which entry exceeds 65,535 bytes in order to perhaps make a workaround. Unfortunately, the problem has been occurring more frequently lately because current cameras sometimes write large binary blobs into the metadata and thus come close to the 65535 limit. Maik
Rolled back to 8.2, with debug mode enabled. Big log this time (I edited the slightest possible to safegard people names and specifics). The message Cannot save metadata with Exiv2 backend: (Error # 38 : "Size of XMP JPEG segment is larger than 65535 bytes" is found a few times in this new log but seems to indicate metadata can't be saved to file. I assume will still be in digikam db for any purpose... And it occurs also for pictures I am *not* having the issue with (like file names ending in _0613 and _1849). So, it is a bit confusing this could be the cause of all issues I am having with faces lost in the database (since for most images reading metadata fixes faces and tags). On the other hand, I see a lot of Error messages: "Unable to fetch row" "database table is locked" "262" 1 messages in the mew log. Could that be part of the issue? You don't find them in the 8.3 log I sent before. May be because I didn't restart digikam to enable the debug mode enabled... but don't seem they belong normally.
Created attachment 165983 [details] 8.2 log file
The locked database issue is definitely the cause of tags disappearing or not being saved. It is normal for a SQLite database to be locked during a write operation, in this case we simply wait until it is available again, up to 10 seconds. Only there are no wait debug messages. May have changed the locking setting in current SQLite versions, I'll investigate. Maik
Gilles, the problem is related to compiling with MSVC, we need to set "SQLITE_DEFAULT_LOCKING_MODE=1" (Exclusive) as the compile option for SQLite. The default is "Normal". We have to see what compile options are still set in normal Linux distributions for SQLite. We could also send a PRAGMA statement to the database if we don't want to make it compile dependent. Maik
Gilles, ignore my Comment 12. Maik
I seen (:=)))
I can reproduce the lost transactions problem here with a SQLite database under stress. The problem may be related to Qt6. I see another SQLite error code that we need to catch, the SQLITE_LOCKED_SHAREDCACHE (262). This could still be related to WAL mode. I'll debug it tonight. Maik
Git commit a3f9448aa09a074fdb039c8864bca04b4c791052 by Maik Qualmann. Committed on 22/02/2024 at 17:52. Pushed by mqualmann into branch 'master'. add check for SQLITE_LOCKED_SHAREDCACHE to SQLite database FIXED-IN: 8.3.0 M +1 -1 NEWS M +6 -3 core/libs/database/engine/dbenginebackend.cpp https://invent.kde.org/graphics/digikam/-/commit/a3f9448aa09a074fdb039c8864bca04b4c791052
A note, you should also activate WAL mode in the digiKam settings under Database. Maik
FYI, enabling WAL and testing with the same set of pictures, the error didn't happen with all faces and tags preserved across all images when I do the changes.
I mean even in 8.2. Sorry.
Git commit 27aa2d0c26e25268253c98aa9ad6ebbea39d3d78 by Maik Qualmann. Committed on 22/02/2024 at 19:49. Pushed by mqualmann into branch 'master'. For WAL mode we need another error code SQLite no longer recommends using shared cache, but instead using WAL mode. We only activate the shared cache if WAL mode has not been activated. M +7 -2 core/libs/database/engine/dbenginebackend.cpp https://invent.kde.org/graphics/digikam/-/commit/27aa2d0c26e25268253c98aa9ad6ebbea39d3d78