Bug 448729

Summary: Metadata is not written back to file when size of XMP JPEG segment is larger than 65535 bytes - Use ExifTool instead Exiv2
Product: [Applications] digikam Reporter: José Oliver-Didier <jose_oliver>
Component: Metadata-ExifToolAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: normal CC: caulier.gilles
Priority: NOR    
Version: 7.5.0   
Target Milestone: ---   
Platform: Microsoft Windows   
OS: Microsoft Windows   
Latest Commit: Version Fixed In: 8.0.0
Sentry Crash Report:
Attachments: Test Image File
DebugView output log

Description José Oliver-Didier 2022-01-18 17:22:35 UTC
Created attachment 145623 [details]
Test Image File

SUMMARY
- After editing metadata in Digikam, metadata is not written back to the file.

STEPS TO REPRODUCE
1. Open attached image file test1.jpg in Digikam 
2. Edit Metadata - Add some Captions
3.  Apply and write metadata to file.

OBSERVED RESULT
- No error and metada is NOT written back to file.

EXPECTED RESULT
- Metadata should be written back to file.


SOFTWARE/OS VERSIONS
- Windows 11 PRO
- Digikam 7.5.0 (Release)

ADDITIONAL INFORMATION
- After running Debug View I encountered the following  error message:
digikam.metaengine: Cannot save metadata with Exiv2 backend:  (Error # 37 :  "Size of XMP JPEG segment is larger than 65535 bytes"
Comment 1 José Oliver-Didier 2022-01-18 17:23:38 UTC
Created attachment 145624 [details]
DebugView output log
Comment 2 caulier.gilles 2022-01-18 17:38:41 UTC
Hi,

This problem is know and is relevant to a limitation of Exiv2 library used in background to play with metadata

See the UPSTREAM bug here :

https://github.com/Exiv2/exiv2/issues/1549

We don't maintain Exiv2 code. Please ask details and plan to Exiv2 team.

Best

Gilles Caulier
Comment 3 José Oliver-Didier 2022-01-18 19:24:35 UTC
I managed to add the metadata using Exiftool via the command line.  The original XMP size (after performing an xmp output) for the file was 63K, close to the 65K threashold. After modifying the file the XMP size increased to 129K. Digikam seems to read the file with the large XMP ok, the problem seems to be on write operations. To accomodate this cases, could Digikam use exiftool for these cases instead?
Comment 4 caulier.gilles 2022-01-18 19:40:11 UTC
Yes, digiKam in the future will use ExifTool with specific cases not supported by Exiv2. We have already few reports about these topics.

The code is not yet finalized and needs regression tests. 

Gilles Caulier
Comment 5 José Oliver-Didier 2022-01-18 21:00:26 UTC
Excellent. Good to know. Seems like the exiv2 issue has not seen much activity in a while. 

Apparently it was resolved in tvisitor.cpp (https://github.com/Exiv2/exiv2/issues/396)
Comment 6 caulier.gilles 2022-01-18 21:07:18 UTC
If it have been solved why this issue exist ?
Comment 7 José Oliver-Didier 2022-01-18 22:16:59 UTC
Good question. From https://github.com/Exiv2/exiv2/issues/396 : 

"tvisitor.cpp which is a stripped down version of Exiv2. It handles JPEG>64k (and BigTiff and ISOBMFF and lots of other stuff not in Exiv2)."

Seems that Exiv2 has not resolved it yet.
Comment 8 caulier.gilles 2022-04-07 04:59:05 UTC
Git commit 3a672f38cda7738356f19707a6a634ca91c269e6 by Gilles Caulier.
Committed on 07/04/2022 at 04:45.
Pushed by cgilles into branch 'master'.

Write metadata with ExifTool backend if Exiv2 fails to process, for exemple with RAW files or Video files.
Tested successfuly with a MP4 video to insert a comment.
For RAW and other king of files as MPO or JPEG with more than 64kB of Exif segement, Exiv2 must be bannned to uses in cases of writting operations as files can be corrupted.
This require more regression tests to validate ExifTool uses in place of Exiv2.
Related: bug 406540, bug 416516, bug 237504, bug 384092, bug 264210, bug 326408, bug 421464, bug 134486, bug 309341, bug 219856, bug 377622, bug 325458, bug 170693, bug 338075, bug 436876, bug 366348

M  +1    -0    core/libs/metadataengine/dmetadata/dmetadata.h
M  +43   -0    core/libs/metadataengine/dmetadata/dmetadata_exiftool.cpp
M  +49   -3    core/libs/metadataengine/dmetadata/dmetadata_fileio.cpp
M  +2    -2    core/libs/metadataengine/exiftool/exiftoolparser.h

https://invent.kde.org/graphics/digikam/commit/3a672f38cda7738356f19707a6a634ca91c269e6
Comment 9 caulier.gilles 2022-04-10 03:51:32 UTC
Git commit f490d0c7b62454a4935754be87e84985f0296115 by Gilles Caulier.
Committed on 10/04/2022 at 03:42.
Pushed by cgilles into branch 'master'.

ExifTool Support:

Add new option to write metadata to files using ExifTool nackend instead Exiv2.
This option replace all calls to Exiv2 to patch files by ExifTool as well.
This will prevent to corrupt files due to lack of stability support from Exiv2.
This includes large XMP section in JPEG and MPO files.
For RAW and DNG files 2 options still here but are depends of ExifTool availability.
We will never write at all to DNG or RAW files with Exiv2.
This new option to replace Exiv2 by ExifTool is turned off by default for the moment,
but if no side-effects is discovered in the future before official 8.0.0 release,
this option will be turned on by default to be safe in all cases.
Related: bug 325458

FIXED-IN: 8.0.0

M  +10   -8    NEWS
M  +1    -0    core/libs/metadataengine/dmetadata/dmetadata.cpp
M  +0    -63   core/libs/metadataengine/dmetadata/dmetadata_exiftool.cpp
M  +2    -48   core/libs/metadataengine/dmetadata/dmetadata_fileio.cpp
M  +14   -4    core/libs/metadataengine/engine/metaengine.cpp
M  +13   -4    core/libs/metadataengine/engine/metaengine.h
M  +171  -30   core/libs/metadataengine/engine/metaengine_p.cpp
M  +9    -2    core/libs/metadataengine/engine/metaengine_p.h
M  +1    -1    core/libs/metadataengine/engine/metaenginesettings.cpp
M  +65   -59   core/libs/metadataengine/engine/metaenginesettingscontainer.cpp
M  +1    -0    core/libs/metadataengine/engine/metaenginesettingscontainer.h
M  +36   -31   core/libs/widgets/metadata/exiftool/exiftoolconfpanel.cpp
M  +8    -0    core/libs/widgets/metadata/exiftool/exiftoolconfpanel.h
M  +3    -0    core/showfoto/setup/showfotosetupmetadata.cpp
M  +16   -30   core/utilities/setup/metadata/setupmetadata.cpp
M  +2    -1    core/utilities/setup/metadata/setupmetadata.h
M  +48   -17   core/utilities/setup/metadata/setupmetadata_behavior.cpp
M  +5    -1    core/utilities/setup/metadata/setupmetadata_p.cpp
M  +4    -0    core/utilities/setup/metadata/setupmetadata_p.h

https://invent.kde.org/graphics/digikam/commit/f490d0c7b62454a4935754be87e84985f0296115