SUMMARY Sometimes when I remove tags from a given file, they don't actually seem to get updated. I uncheck the tag, click Apply, and the file modification date does get updated (showing that digiKam actually did "touch" the file). However, if I diff the file against a backup, no binary changes have taken place, and if I "Reread Metadata From File", digiKam reveals that the tag was actually never removed from the file. Easiest to illustrate with a screencast: https://jiij.cc/snaps/2022-12-22_21.50.36.mp4 STEPS TO REPRODUCE 1. Settings->Configure digiKam->Metadata->Behavior->Check all the checkboxes under "Write This Information to the Metadata" (all other settings on that dialog are as default). 2. Navigate to a photo that has a tag. Note its modification date. 3. Make a backup copy of that photo somewhere 4. On the right of digiKam, go to Captions->Tags 5. Untick the tag, then click "Apply" at the bottom 6. Note that the file modification date has been updated, as expected. 7. Binary diff the photo against its backup. They are identical - the tag was not actually removed. 8. Item->Reread Metadata From File. The tag now reappears in the digiKam UI. SOFTWARE/OS VERSIONS Windows: 7
We need a DebugView log of the process. Download and launch DebugView from Microsoft. Activate internal debugging in the digiKam settings under Miscellaneous-> System and start digiKam again. Run the tag operation and post the DebugView log. Whether the modification date changes depends on the metadata settings. You enabled it to update back to the previous one. Maik
> Whether the modification date changes depends on the metadata settings. Yeah, that was desired - was just mentioning it to illustrate that digiKam had actually done something to the file (even though no binary changes took place). > You enabled it to update back to the previous one. Not sure what you mean by this...? > We need a DebugView log [11176] digikam.metaengine: Loading metadata with "Exiv2" backend from "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" [11176] digikam.general: Writing tags [11176] digikam.general: Delete all keywords [11176] digikam.metaengine: "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" ==> New Iptc Keywords: () [11176] digikam.metaengine: MetaEngine::metadataWritingMode 0 [11176] digikam.metaengine: Will write Metadata to file "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" [11176] digikam.metaengine: wroteComment: true [11176] digikam.metaengine: wroteEXIF: true [11176] digikam.metaengine: wroteIPTC: true [11176] digikam.metaengine: wroteXMP: true [11176] digikam.metaengine: Metadata for file "Mich visit 10-2010 003-2.JPG" written to file. [11176] digikam.metaengine: Loading metadata with "Exiv2" backend from "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" [11176] digikam.dimg: "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" : "JPEG" file identified [11176] digikam.database: Scanning took 6 ms [11176] digikam.database: Finishing took 7 ms [11176] digikam.metaengine: Loading metadata with "Exiv2" backend from "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" [11176] digikam.general: Trying to get thumbnail from "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" ( "image" ) [11176] digikam.general: Trying to get thumbnail with Exiv2 for "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" [11176] digikam.metaengine: Loading metadata with "Exiv2" backend from "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" [11176] digikam.general: Trying to get thumbnail from "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" ( "image" ) [11176] digikam.general: Trying to get thumbnail with Exiv2 for "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" [11176] digikam.general: Trying to get thumbnail with DImg preview for "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" [11176] digikam.general: Trying to get thumbnail with DImg preview for "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" [11176] digikam.dimg: "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" : "JPEG" file identified [11176] digikam.dimg: "D:/_Pics+Vids_/Pics/0-(82-94) - Early Childhood/Pops & Grandma Misc/Mich visit 10-2010 003-2.JPG" : "JPEG" file identified [11176] digikam.general: Using 8 CPU core to run threads [11176] digikam.general: Using 8 CPU core to run threads [11176] digikam.general: Action Thread run 1 new jobs [11176] digikam.general: Action Thread run 1 new jobs [11176] digikam.general: One job is done [11176] digikam.general: Cancel Main Thread [11176] digikam.general: One job is done [11176] digikam.general: Cancel Main Thread
According to the log, the metadata was written without errors. If the tag was written with Windows programs, there is a possibility that the tag is written in Microsoft XP* metadata. Exiv2 can currently only read this metadata area, but not change it. That would explain the tag reappearing. Can you email me the image? Maik
Is the output of exiftool enough? Let me know if not & I'll send the picture itself, but exiftool's output is below. I do see mention of "XP Keywords." However, if there's a known issue that a particular type of keyword will not be properly written, shouldn't the software raise an error/warning, rather than appearing to succeed but just silently failing? I spent quite some time going through & cleaning up tags before (much later) uncovering that all that time had actually been wasted, and nothing was actually being updated. I found it completely by chance when I diffed an image for other reasons, then re-read from metadata and suddenly every single tag I'd just removed was back. So apparently I hadn't actually done anything. ``` ExifTool Version Number : 12.51 File Name : Mich visit 10-2010 003-2.JPG Directory : . File Size : 969 kB File Modification Date/Time : 2022:12:22 22:57:20-08:00 File Access Date/Time : 2022:12:23 08:38:53-08:00 File Creation Date/Time : 2022:12:17 19:44:13-08:00 File Permissions : -rw-rw-rw- File Type : JPEG File Type Extension : jpg MIME Type : image/jpeg Exif Byte Order : Big-endian (Motorola, MM) Make : SONY Camera Model Name : DSC-W70 Orientation : Horizontal (normal) X Resolution : 72 Y Resolution : 72 Resolution Unit : inches Software : Adobe Photoshop CS5 Windows Modify Date : 2010:10:27 09:21:26 Y Cb Cr Positioning : Co-sited XP Keywords : Mich visit 10-2010 PrintIM Version : 0300 Exposure Time : 1/50 F Number : 5.2 Exposure Program : Program AE ISO : 160 Exif Version : 0221 Date/Time Original : 2010:09:23 17:56:36 Create Date : 2010:09:23 17:56:36 Components Configuration : Y, Cb, Cr, - Compressed Bits Per Pixel : 8 Exposure Compensation : +1.7 Max Aperture Value : 2.8 Metering Mode : Multi-segment Light Source : Unknown Flash : On, Return detected Focal Length : 18.9 mm Flashpix Version : 0100 Color Space : sRGB Exif Image Width : 2304 Exif Image Height : 3072 Interoperability Version : 0100 File Source : Digital Camera Scene Type : Directly photographed Custom Rendered : Normal Exposure Mode : Manual White Balance : Auto Scene Capture Type : Standard Contrast : High Saturation : High Sharpness : Normal Padding : (Binary data 2060 bytes, use -b option to extract) Offset Schema : 4200 Compression : JPEG (old-style) Thumbnail Offset : 5030 Thumbnail Length : 6889 XMP Toolkit : XMP Core 4.4.0-Exiv2 Creator Tool : Microsoft Windows Photo Gallery 6.0.6001.18000 Metadata Date : 2010:10:27 09:21:26-07:00 Format : image/jpeg Color Mode : RGB ICC Profile Name : sRGB IEC61966-2.1 Instance ID : xmp.iid:0B6D863FE6E1DF11964DF8DAF7BB095B Document ID : xmp.did:0B6D863FE6E1DF11964DF8DAF7BB095B Original Document ID : xmp.did:0B6D863FE6E1DF11964DF8DAF7BB095B Description : History Action : saved History Instance ID : xmp.iid:0B6D863FE6E1DF11964DF8DAF7BB095B History When : 2010:10:27 09:21:26-07:00 History Software Agent : Adobe Photoshop CS5 Windows History Changed : / Profile CMM Type : Linotronic Profile Version : 2.1.0 Profile Class : Display Device Profile Color Space Data : RGB Profile Connection Space : XYZ Profile Date Time : 1998:02:09 06:49:00 Profile File Signature : acsp Primary Platform : Microsoft Corporation CMM Flags : Not Embedded, Independent Device Manufacturer : Hewlett-Packard Device Model : sRGB Device Attributes : Reflective, Glossy, Positive, Color Rendering Intent : Perceptual Connection Space Illuminant : 0.9642 1 0.82491 Profile Creator : Hewlett-Packard Profile ID : 0 Profile Copyright : Copyright (c) 1998 Hewlett-Packard Company Profile Description : sRGB IEC61966-2.1 Media White Point : 0.95045 1 1.08905 Media Black Point : 0 0 0 Red Matrix Column : 0.43607 0.22249 0.01392 Green Matrix Column : 0.38515 0.71687 0.09708 Blue Matrix Column : 0.14307 0.06061 0.7141 Device Mfg Desc : IEC http://www.iec.ch Device Model Desc : IEC 61966-2.1 Default RGB colour space - sRGB Viewing Cond Desc : Reference Viewing Condition in IEC61966-2.1 Viewing Cond Illuminant : 19.6445 20.3718 16.8089 Viewing Cond Surround : 3.92889 4.07439 3.36179 Viewing Cond Illuminant Type : D50 Luminance : 76.03647 80 87.12462 Measurement Observer : CIE 1931 Measurement Backing : 0 0 0 Measurement Geometry : Unknown Measurement Flare : 0.999% Measurement Illuminant : D65 Technology : Cathode Ray Tube Display Red Tone Reproduction Curve : (Binary data 2060 bytes, use -b option to extract) Green Tone Reproduction Curve : (Binary data 2060 bytes, use -b option to extract) Blue Tone Reproduction Curve : (Binary data 2060 bytes, use -b option to extract) Current IPTC Digest : 4ae40b964f5554a91a2fbc346024a33b Coded Character Set : UTF8 Application Record Version : 0 Date Created : 2010:09:23 Time Created : 17:56:36+00:00 IPTC Digest : 6575fb6f71ab3939cf0ef36f4930ae6b Displayed Units X : inches Displayed Units Y : inches Print Style : Centered Print Position : 0 0 Print Scale : 1 Global Angle : 30 Global Altitude : 30 URL List : Slices Group Name : Mich visit 10-2010 003 Num Slices : 1 Pixel Aspect Ratio : 1 Photoshop Thumbnail : (Binary data 6889 bytes, use -b option to extract) Has Real Merged Data : Yes Writer Name : Adobe Photoshop Reader Name : Adobe Photoshop CS5 Photoshop Quality : 6 Photoshop Format : Standard Warning : IPTCDigest is not current. XMP may be out of sync DCT Encode Version : 100 APP14 Flags 0 : Encoded with Blend=1 downsampling APP14 Flags 1 : (none) Color Transform : YCbCr Image Width : 2304 Image Height : 3072 Encoding Process : Baseline DCT, Huffman coding Bits Per Sample : 8 Color Components : 3 Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2) Aperture : 5.2 Image Size : 2304x3072 Megapixels : 7.1 Shutter Speed : 1/50 Thumbnail Image : (Binary data 6889 bytes, use -b option to extract) Date/Time Created : 2010:09:23 17:56:36+00:00 Focal Length : 18.9 mm Light Value : 9.7 ```
Exactly, XP Keywords is the problem. I can look up the relevant bug reports for you. Just so much, because of the encoding, this tag has historically only been read-only in Exiv2. It also doesn't help to enable writing with ExifTool in digiKam-8.0.0 because we create the container with Exiv2. digiKam uses the C++ program Exiv2 internally. What options do you have? 1. You can remove XP Keywords using batch processing and the Remove Metadata Tool, there is an extra entry for Exif in the combo box. 2. You can disable the reading of XP Keywords in the advanced metadata settings, but the value of XP keywords will still remain in the image metadata. 3. We find another solution... Maik
Those options all work as a one-off fix for a particular images/images (or for users who somehow become aware of the issue), but more broadly, the software should really provide some warning. Ie just a simple check to see if that tag is there, and if so, the confirmation toast should let the user know that their changes may have not been applied. Otherwise, anytime someone copies a bunch of photos from a friend/relative/etc, and attempts to just work with tags normally, it's not at all obvious that half the actions taken were actually ignored. If or until there's a true fix, why not just show: "Warning: 12 of the images written contained XP metadata, so your changes may not have applied. Please see (link to this issue, or to the most relevant)."
Giving a warning every time isn't really practical. On the one hand we don't get an error message from Exiv2 in this case. On the other hand, we would have to read all the metadata of the images to determine which ones contain the XP* tags, which would lead to a severe loss of performance, especially on network drives. We can actually write the XP* tags with ExifTool in the same step as the modified metadata container. In a first patch we will generally remove the XP* tags when writing new tags. Maik
Git commit d71f6605893d5b1b801ebc425756114711c5fc97 by Maik Qualmann. Committed on 25/12/2022 at 10:20. Pushed by mqualmann into branch 'master'. add support for XPTitle and remove XP* metadata when writing Related: bug 422242, bug 421464, bug 450522 M +36 -1 core/libs/metadataengine/dmetadata/dmetadata_comments.cpp M +24 -10 core/libs/metadataengine/dmetadata/dmetadata_tags.cpp M +13 -5 core/libs/metadataengine/dmetadata/dmetadatasettingscontainer.cpp https://invent.kde.org/graphics/digikam/commit/d71f6605893d5b1b801ebc425756114711c5fc97
Git commit 0e93b5344bf740ad0db1ca85532a66f1f472146a by Maik Qualmann. Committed on 25/12/2022 at 13:40. Pushed by mqualmann into branch 'master'. add write support for XP* metadata when Exiftool writing is enabled By default, the legacy XP* metadata is disabled in a new config. Related: bug 422242, bug 421464, bug 450522 M +8 -5 NEWS M +22 -2 core/libs/metadataengine/dmetadata/dmetadata_comments.cpp M +13 -1 core/libs/metadataengine/dmetadata/dmetadata_tags.cpp M +3 -0 core/libs/metadataengine/dmetadata/dmetadatasettingscontainer.cpp https://invent.kde.org/graphics/digikam/commit/0e93b5344bf740ad0db1ca85532a66f1f472146a
(In reply to Maik Qualmann from comment #7) > Giving a warning every time isn't really practical. So currently the software has that toast UI, i.e. "Updating database - process is done." I was thinking a similar pop-up when applying from database to file, i.e. "Writing files from database - xxx items may not have been successful." Something like that. > On the one hand we don't get an error message from Exiv2 in this case. But the digiKam catalog already knows the files have that particular metadata tag: https://jiij.cc/snaps/2022-12-25_09.17.56.png. So it shouldn't need an explicit error for Exiv2, right? Just knowing that "we're writing tags to a file" and "that file has the XP* tag" is enough to infer that "this file may not be written correctly." > On the other hand, we would have to read all the metadata of the images to determine which ones contain the XP* tags, which would lead to a severe loss of performance, especially on network drives. Why would it need to read all the metadata from files again? The catalog clearly already knows the metadata is there (that's how it knew that tag existed in the first place, and made it available to be unchecked in the "Captions" pane). So no need to re-read from disk again...? > We can actually write the XP* tags with ExifTool in the same step as the modified metadata container. In a first patch we will generally remove the XP* tags when writing new tags. > add write support for XP* metadata when Exiftool writing is enabled > By default, the legacy XP* metadata is disabled in a new config. So to clarify, to avoid the issue, I need to explicitly enable "Settings->Metadata->Delegate to ExifTool all backend operations to write metadata to files"? If so, any other ramification/side effect, besides this bug not occurring?
We already have requests to provide a user viewable log in digiKam. I'm sure it will come in the future. The metadata viewer only displays the entry of a single selected image. In the case of a multi-selection or maintenance tool process, we would have to examine the images for this beforehand. Writing an empty tags list now correctly removes XPKeywords. If you don't have write with ExifTool enabled, XPKeywords will be removed in all cases. If you have enabled writing with ExifTool, XPKeywords will be written correctly. My research resulted in a bug report for a Windows photo program, either they also use Exiv2 in the background, since XPKeywords is also always deleted. Also the author of ExifTool writes that one should avoid the XP* metadata. Maik
(In reply to Maik Qualmann from comment #11) > The metadata viewer only displays the entry of a single selected image. In the case of a multi-selection or maintenance tool process, we would have to examine the images for this beforehand. Ohh I see - so that's literally directly viewing the metadata in the file, not in the catalog (and the catalog only stores a more generalized subset of meta - not knowing that "tags" came from XP specifically). > Writing an empty tags list now correctly removes XPKeywords. If you don't have write with ExifTool enabled, XPKeywords will be removed in all cases. If you have enabled writing with ExifTool, XPKeywords will be written correctly. Gotcha, great! So no settings changes needed. > My research resulted in a bug report for a Windows photo program, either they also use Exiv2 in the background, since XPKeywords is also always deleted. Also the author of ExifTool writes that one should avoid the XP* metadata. Yeah, I mean I wouldn't use it myself. But since one doesn't have control over what others might have done to photos before they were received, I just wanted to make sure it was possible to reliably work with tags, more generally, even if someone else has already used XP tags. Sounds like this fix now does it :) Sorry, quick semi-related question: I'm using v8.0.0 (per our other discussion related to sharing the db between win+linux), but it looks like the weekly builds are for 7.10, not 8.0 (https://files.kde.org/digikam/). Is there somewhere with weekly builds for 8.0, that contain all the latest patches?
Normally, Gilles creates weekly snapshots. However, he is very busy at the moment, restructuring and porting the digiKam documentation. I don't think he'll be anywhere near the build machines over the holidays either. A bit patience... Maik
(In reply to Maik Qualmann from comment #13) > Normally, Gilles creates weekly snapshots. However, he is very busy at the > moment, restructuring and porting the digiKam documentation. I don't think > he'll be anywhere near the build machines over the holidays either. A bit > patience... > > Maik Ah OK, so I guess they'll appear at that same place when available then? Just wasn't sure if there was somewhere else for 8.x snapshots.
Hi Maik and Merry Christmas, Yes, i'm busy bu i advance well https://docs.digikam.org/en/index.html I hope to finalize online doc in few days... I will rebuild the Windows bundle tomorrow morning, no problem. Gilles
(In reply to caulier.gilles from comment #15) > Hi Maik and Merry Christmas, > > Yes, i'm busy bu i advance well > > https://docs.digikam.org/en/index.html > > I hope to finalize online doc in few days... > > I will rebuild the Windows bundle tomorrow morning, no problem. > > Gilles No rush, take your time! I was just curious where to find them, more generally - sounds like I did have the right spot :)
> No rush, take your time! I was just curious where to find them, more > generally - sounds like I did have the right spot :) Hi Metal450, It is in the same place as 7.10, only in the unstable branch: https://files.kde.org/digikam/unstable/
Aha, perfect, thanks!