I take a RAW file from my Canon 400D - it doesn't matter if it's the CR2 file or if it's been converted to a DNG. I add a tag, title, and caption in Digikam. Then I edit the file in RawTherapee. Then I load up the newly created JPEG in Digikam. It only can read the caption. The title and tag do not show in Digikam. Looking with exiftool confirms that the data has been properly written to the JPEG. However, when I look at the file using exiv2 - which Digikam uses, I get the following warnings: exiv2 -p a _MG_3694.jpg Warning: Ignoring IPTC information encoded in the Exif data. Warning: Ignoring XMP information encoded in the Exif data. Which I feel is responsible for the problem. Reproducible: Always Steps to Reproduce: 1. Tag RAW files in Digikam 2. Edit in RawTherapee 3. Produce a JPEG 4. The JPEG is missing tags. Actual Results: No tags (or title) on resulting JPEGs Expected Results: Tags and titles on resulting JPEGs. You can see the CR2 here: https://www.dropbox.com/s/jvqiinpsaapzxf4/_MG_3694.CR2?dl=0 You can see the resulting JPEG here; https://www.dropbox.com/s/ip503k3tom9ciew/_MG_3694.jpg?dl=0
Oh yeah, the version of exiv2 on Fedora is 0.24-4, but I had to install that separately from Digikam so I imagine that you guys have exiv2 embedded and the version I have on my computer doesn't matter.
>I take a RAW file from my Canon 400D - it doesn't matter if it's the CR2 file or if it's been converted to a DNG. I add a tag, title, and caption in Digikam. >Then I edit the file in RawTherapee. Then I load up the newly created JPEG in Digikam. It only can read the caption. => Caption is stored in JPEG comment section, outside XMP and IPTC >The title and tag do not show in Digikam. Looking with exiftool confirms that the data has been properly written to the JPEG. Stored in XMP >However, when I look at the file using exiv2 - which Digikam uses, I get the following warnings: >exiv2 -p a _MG_3694.jpg >Warning: Ignoring IPTC information encoded in the Exif data. >Warning: Ignoring XMP information encoded in the Exif data. Sound like a corruption or somthing like that performed by RawTherapee. Exiv2 do not like it. >Which I feel is responsible for the problem. It's clear. Metadata have been corrupted by RawTherapee. There is nothing to do in digiKam/libkexiv2 interface 1/ Report this problem to Exiv2 bugzilla. Explain well all operation done in RawTherapee to reproduce the problem. Provide original images, images tagged in digiKam, and images fixed in RawTherapee. 2/ When problem is identified by Exiv2 team, the question will be : there is something to fix in Exiv2 ? If not report this problem to RawTherapee team. Gilles Caulier
>Oh yeah, the version of exiv2 on Fedora is 0.24-4, but I had to install that separately from >Digikam so I imagine that you guys have exiv2 embedded and the version I have on my >computer doesn't matter. No. digiKam do not include Exiv2 library code. It use system shared library as well. Gilles Caulier
Gilles, what does "If possible, write Metadata to RAW files (experimental)" do? In the samples, the XMP has landed up inside of Exif. That is the experimental feature?
Well, Raw metadata writing mode delegate Raw writing support to Exiv2 as it's done with others file format. Raw is considerated by people as read only image (as negative photo). So the option is disabled by default. The option is annotated experimental because not all format are supported by Exiv2 (and also tested). Typically, only TIFF/EP file based are supported. I think we must remove this option in Exiv2, since XMP sidecar do the stuff instead, if end user turn on right option in this settings panel. Note : It's libkexiv2 which wrap these workflow settings with Exiv2 in background. Gilles
Just so you know, this also happens with DNG file, which is considered "safe" for writing metadata to without the need for XMP sidecar. Also, very important to keep from having to keep track of two different files when backing up, etc
The XMP is inside of Exif ApplicationNotes. Is that valid according to the specs? According to exiftool: | 16) ApplicationNotes (SubDirectory) --> | + [XMP directory, 4868 bytes] | | XMPToolkit = XMP Core 4.4.0-Exiv2 | | Software = digiKam-4.9.0 | | DateTime = 2015-05-13T17:08:52 | | CreatorTool = digiKam-4.9.0 | | CreateDate = 2015-05-13T17:08:52 | | MetadataDate = 2015-05-13T17:08:52 | | ModifyDate = 2015-05-13T17:08:52 | | DateTimeOriginal = 2015-05-13T17:08:52 | | DateCreated = 2015-05-13T17:08:52 | | Urgency = 0 | | PickLabel = 0 | | ColorLabel = 0 | | ImageDescription = Leaves and Scarlett | | UserComment = Leaves and Scarlett | | TagsList = Scarlett | | CaptionsAuthorNames = | | CaptionsDateTimeStamps = 2015-05-13T20:33:02 | | RegionList = | | RegionInfoRegions = | | LastKeywordXMP = Scarlett | | HierarchicalSubject = Scarlett | | Title = Scarlett Playing with Leaves | | Description = Leaves and Scarlett | | Subject = Scarlett
The XMP in which file ? JPEG generated by RawTherapee ? If yes, the Exif spec permit to do it but it's not recommended with JPEG (and in general), as JPEG Exif chunk is limited to 64Kb. JPEG IPTC and XMP chunk are also limited in size to 64Kb, but as separated than Exif. To resume : Exif = 64Kb IPTC = 64Kb XMP = 64Kb If you host IPTC and XMP to Exif chunk as well, there is a risk to overload this chunk quickly and to break JPEG structure. This is why, if i remember well, Exiv2 drop this Exif entries. Note : i know that Adode has proposed a way to extend each JPEG chunk to use more than one 64Kb blocks, but i'm not sure if this is already standardized. In all case, RawTherapee must store IPTC and XMP in dedicated chunks. Gillesq
No, the XMP is in the cr2 file generated by digikam.
Ok, it looks like what exiftool calls ApplicationNotes, exiv2 calls Exif.Image.XMLPacket.
Does that mean exiv2 needs a translation between the two so it's able to read it?
To comment #11 : no, if i remember, there is already a file in Exiv2 bugzilla about to ignore ITPC and XMP from Exif tags Gilles
Alan, Just a terminology adjustement : I used JPEG "chunk" previously, where in JPEG we use JPEG "section". Chunk is used for PNG container. Gilles
Exiv2 issues more and less relevant : http://dev.exiv2.org/issues/699 http://dev.exiv2.org/issues/533
(In reply to Gilles Caulier from comment #12) > To comment #11 : no, if i remember, there is already a file in Exiv2 > bugzilla about to ignore ITPC and XMP from Exif tags > > Gilles So at this point it's still an exiv2 but, then?
(In reply to Eric Mesa from comment #15) > (In reply to Gilles Caulier from comment #12) > > To comment #11 : no, if i remember, there is already a file in Exiv2 > > bugzilla about to ignore ITPC and XMP from Exif tags > > > > Gilles > > So at this point it's still an exiv2 but, then? That should say "bug" not "but", couldn't find how to edit my comment.
Not sure if it's an Exiv2 bug. But in all case, in digiKam, there is nothing to do... Gilles
(In reply to Gilles Caulier from comment #17) > Not sure if it's an Exiv2 bug. But in all case, in digiKam, there is nothing > to do... > > Gilles Gotcha. Thanks. At this point it appears either something in Exiv2 or something in RawTherapee - I guess whoever's right depends on who's adhering to the spec the most, I imagine. Worst case scenario I could always write a bash script to read the data out of the DNGs with Exiftool and into the JPEGs with Exiv2 until it's resolved.
1st: It has nothing to do with RawTherapee as the XMP data was written to the Exif section of the CR2 image by libkexiv2 via digikam. RawTherapee carried over that data unchanged to the JPG. 2nd: This is why it is labelled experimental. For CR2 images, the correct workflow is to allow digikam to only write to XMP sidecar files. Then once you have converted to JPG, import the XMP metadata from the sidecar file. 3rd: There has been some work on improving RAW file support in exiv2, but it is not yet complete. This includes reading from Exif.Image.XMLPacket.
(In reply to Alan Pater from comment #19) > 1st: It has nothing to do with RawTherapee as the XMP data was written to > the Exif section of the CR2 image by libkexiv2 via digikam. RawTherapee > carried over that data unchanged to the JPG. > > 2nd: This is why it is labelled experimental. For CR2 images, the correct > workflow is to allow digikam to only write to XMP sidecar files. Then once > you have converted to JPG, import the XMP metadata from the sidecar file. > > 3rd: There has been some work on improving RAW file support in exiv2, but it > is not yet complete. This includes reading from Exif.Image.XMLPacket. I wouldn't mind changing it to XMP sideca (or maybe sidecar+file?), but does the same workflow hold for DNG files? Because I know that I had the same thing happen with a DNG file as with a CR2 file. (In other words, the data was not read in exiv2 after the JPEG was made). The reason I ask is that, at this point with how good support is across the board for camera raw files, the only real reason for me to convert to DNGs is to take advantage of being able to store metadata within the file rather than in a sidecar file. Otherwise, RawTherapee, Digikam, etc can all read the CR2 file and it works just fine for archival purposes - my descendents will most likely be able to find a converter and/or the most important files will most likely have been made into JPEGs anyway. So, should I be able to save this data to the DNG and expect the resultant JPEG to be read by Exiv2? That said, I understand this is all FLOSS software done by volunteers. I'm not trying to be entitled or whatever. I'm just trying to understand the anticipated workflow as well as bugs to be filed/versions to wait for to expect the functionality.
DNG support XMP directly, which is one of it's advantages.
(In reply to Alan Pater from comment #21) > DNG support XMP directly, which is one of it's advantages. Sure, but it's still having the same issues as the CR2 file. At least in this context.
Alan, > 1st: It has nothing to do with RawTherapee as the XMP data was written to > the Exif section of the CR2 image by libkexiv2 via digikam. RawTherapee > carried over that data unchanged to the JPG. no. digiKam or libkexiv2 do not force to play directly with Exif.Image.XMLPacket. I just grep in code to see if Exif.Image.XMLPacket is used somewhere, and it appear only in digiKam tiff loader for image editor. This have nothing about this entry. Typically, digiKam as to Exiv2 (through libkexiv2) to fill Exif, IPTC and XMP. Exiv2 must decide where tags must be handle (in this case at Exif, or Xmp section) The CR2 Raw file can not store XMP value to a dedicated sub container (section, chunk) as it's do with JPEG or PNG. After all CR2 are TIFF, and there is not size limitation as JPEG to host data. Gilles
(In reply to Gilles Caulier from comment #23) > Alan, > > > 1st: It has nothing to do with RawTherapee as the XMP data was written to > > the Exif section of the CR2 image by libkexiv2 via digikam. RawTherapee > > carried over that data unchanged to the JPG. > > no. digiKam or libkexiv2 do not force to play directly with > Exif.Image.XMLPacket. I just grep in code to see if Exif.Image.XMLPacket is > used somewhere, and it appear only in digiKam tiff loader for image editor. > This have nothing about this entry. > > Typically, digiKam as to Exiv2 (through libkexiv2) to fill Exif, IPTC and > XMP. Exiv2 must decide where tags must be handle (in this case at Exif, or > Xmp section) > > The CR2 Raw file can not store XMP value to a dedicated sub container > (section, chunk) as it's do with JPEG or PNG. After all CR2 are TIFF, and > there is not size limitation as JPEG to host data. > > Gilles Yes, but in this example the CR2 is a bit of a red herring as it's not working on DNG either and that's supposed to work fine with XMP.
DNG file with same issues: https://www.dropbox.com/s/pmz8oneual5pq82/Pots.dng?dl=0 JPEG: https://www.dropbox.com/s/l8rw3zpjgf599f7/Pots.jpg?dl=0
Technically, there is not difference between CR2 and DNG : both are tiff/EP based files. So same dysfunction... Gilles Caulier
I am sure it is digikam writing the Exif XMLPacket. here is how I tested and you can repeat this yourself to confirm. Download a CR2 file: http://www.rawsamples.ch/raws/canon/400d/RAW_CANON_400D_ARGB.CR2 Use exiv2 to check if XMLpacket exists in the file: $ exiv2 -g XMLPacket RAW_CANON_400D_ARGB.CR2 Confirmed. XMLPacket does not exist in the CR2 image. In digikam, add a Title and a Tag. Go to Image: Write metadata to image. $ exiv2 -g XMLPacket RAW_CANON_400D_ARGB.CR2 Exif.Image.XMLPacket Byte 4326 (Binary value suppressed)
Try the same test with DNG, but without the experimental writing metadata to Raw feature- digikam will save the Title and Tag in a normal XMP section. Convert the file to JPG in RawTherapee. The resulting JPG does not have normal XMP section. RawTherapee has removed it.
It is indeed exiv2 which automatically adds Exif.Image.XMLPacket to the file without any explicit instruction from digikam/libkexiv2. ~$ exiv2 -g XMLPacket raw.exiv2.cr2 ~$ exiv2 -M'set Xmp.dc.title EXIV TITLE' raw.exiv2.cr2 ~$ exiv2 -g XMLPacket raw.exiv2.cr2 Exif.Image.XMLPacket Byte 2495 (Binary value suppressed)
Alan, Exactly. As i said previously, digiKam as to Exiv2 to store XMP data to CR2. Exiv2 choose the right place to do it. For CR2, as it's TIFF/EP file based, the way to store XMP in tiff is to use XMLPacket from Exif. There is no separated section/chunk in tiff. After all, tiff structure is a tree of tags. This way is the same for DNG. So this is normal until now. When you convert RAW tiff based with XMLPacket to JPG, the fast way is to move XMLPacket to dedicated section. RawTherapee must do it, but it let XMP to Exif tags. Later when Exiv2 will read JPG file, it found XMP in XMLPacket instead XMP section. This is ignored because XMLPacket is deprecated than XMP section for JPEG. Why ? because for JPEG, as Exif section is limited to 64Kb, XMP must go to XMP section instead to bloat Exif section to prevent to break JPEG structure. A section more than 64Kb give a corrupted file, excepted if more than one section are chained to host Exif data (The Adobe way). But i'm not sure that Exiv2 support it well, as all other metadata framework (excepted Adobe tools of course). I don't know how RawTherapee work, but for me the metadata workflow is wrong.