Bug 211758

Like previous bug from you : take a look of difference between metadata of both TIFF files using ExifTool.

Gilles Caulier

Can you please provide Bild_M2c_061h.tif when it was still ok, before it was saved in digiKam?

How did you create Bild_M2c_061i.tif? Which version of exiv2 was used to save it (if it was saved using digiKam at all)?

Andreas

Hi Andreas,

> Can you please provide Bild_M2c_061h.tif when it was still ok, before it was
> saved in digiKam?
>
It is quiet large (120MB). Today my upload speed is really bad. So I'll provide a link tomorrow.

> How did you create Bild_M2c_061i.tif? Which version of exiv2 was used to save
> it (if it was saved using digiKam at all)?
> 
For both images I used Version 1.0.0-beta6 (rev.: 1039950) with exiv2 0.18.2 and libtiff 3.8.2. Before saving I scaled the image down to reduce the size, but the odd result was the same when I just saved it. The difference was only that for Bild_M2c_061h.tif the checkbox "Compress TIFF file" was ticked and for Bild_M2c_061i.tif not.

Guenther

(In reply to comment #3)
> It is quiet large (120MB). Today my upload speed is really bad. So I'll
> provide a link tomorrow.
> 
http://www.gmx.de/mc/gHBpcdORx2xt9u16rDvJm2ZFSzMngl

This link is valid until Nov. 26th.

Guenther

*** Bug 212072 has been marked as a duplicate of this bug. ***

Output from tiffinfo with 061h.tif from above:
TIFFReadDirectory: Warning, Bild_M2c_061i.tif: invalid TIFF directory; tags are not sorted in ascending order.
TIFFReadDirectory: Warning, Bild_M2c_061i.tif: unknown field with tag 11 (0xb) encountered.
TIFFReadCustomDirectory: Warning, Bild_M2c_061i.tif: unknown field with tag 40961 (0xa001) encountered.
TIFFReadDirectory: Warning, Bild_M2c_061i.tif: unknown field with tag 513 (0x201) encountered.
TIFFReadDirectory: Warning, Bild_M2c_061i.tif: unknown field with tag 514 (0x202) encountered.
MissingRequired: Bild_M2c_061i.tif: TIFF directory is missing required "ImageLength" field.

This is bug #211066, only this time affecting image data!

Yep, the same bug in Ubuntu 9.10. These tiffs (8 or 16 bits)) are generated by builtin Batch Raw Converter.

yes, i can reproduce there this dysfunction, using ne HDR creation tool when RAW are imported as HDR. Internally, RAW are converted to TIFF file as post processed before import.

TIFF are loadable but image contents is broken, as if R and B color are switched... or definition of component.

This is only reproducible when libkexiv2/Exiv2 is called to restore metadata from original RAW to TIFF.

And this i also reproducible using RAW converter create TIFF from RAW file. The tiff writter code is the same than HDR Creator tool (code is shared in libkipiplugins).

If i comment libkexif call, TIFF file generated is fine... This line in fact :

http://lxr.kde.org/source/extragear/graphics/kipi-plugins/common/libkipiplugins/kpwriteimage.cpp#763

The tiff writter use libtiff of course. 

I have used Exiv2 0.18.2 to make tests. I will try current code from svn...

I make some test image for investiguations. upload under progress...

Gilles Caulier

Andreas,

Test files generated with Exiv2 0.18.2 are there :

http://digikam3rdparty.free.fr/bugs/212072

-rw-r--r-- 1 gilles gilles 30686734 2009-12-05 23:43 MINOLTA-DYNAX7D-04-2.tif
-rw-r--r-- 1 gilles gilles   446816 2009-12-06 15:24 MINOLTA-DYNAX7D-04-2.txt
-rw-r--r-- 1 gilles gilles  9190688 2006-03-24 00:00 MINOLTA-DYNAX7D-04.MRW
-rw-r--r-- 1 gilles gilles    14230 2009-12-06 15:34 MINOLTA-DYNAX7D-04.MRW.txt
-rw-r--r-- 1 gilles gilles 30716928 2009-12-05 23:22 MINOLTA-DYNAX7D-04.tif
-rw-r--r-- 1 gilles gilles   360619 2009-12-06 15:24 MINOLTA-DYNAX7D-04.txt

- MINOLTA-DYNAX7D-04.MRW is the original RAW file
- MINOLTA-DYNAX7D-04.tif is the 1th TIF file converted from RAW file. This one is corrupted by libkexiv2/Exiv2 call.
- MINOLTA-DYNAX7D-04-2.tif is the 2nd TIF file converted from RAW file without to call libkexiv2/Exiv2 to restore metadata. In fact only libtiff is used in this case.

Look the message given by ImageMagick commanad line tool to try to see this image :

[gilles@localhost HORIZONTAL]$ display MINOLTA-DYNAX7D-04.tif
display: MINOLTA-DYNAX7D-04.tif: invalid TIFF directory; tags are not sorted in ascending order. `TIFFReadDirectory' @ tiff.c/TIFFWarnings/703.
display: MINOLTA-DYNAX7D-04.tif: unknown field with tag 11 (0xb) encountered. `TIFFReadDirectory' @ tiff.c/TIFFWarnings/703.
display: MINOLTA-DYNAX7D-04.tif: unknown field with tag 50341 (0xc4a5) encountered. `TIFFReadDirectory' @ tiff.c/TIFFWarnings/703.
display: MINOLTA-DYNAX7D-04.tif: TIFF directory is missing required "ImageLength" field. `MissingRequired' @ tiff.c/TIFFErrors/493.

Note : message are generated by libtiff of course...

Gilles

Andreas,

Don't forget this code from libkexiv2 to trying to preserve TIFF image structure.

http://lxr.kde.org/source/KDE/kdegraphics/libs/libkexiv2/libkexiv2/kexiv2.cpp#391

This code is inspired from UFRAW when metadata are backported from RAW to TIFF.

http://ufraw.cvs.sourceforge.net/viewvc/ufraw/ufraw/ufraw_exiv2.cc?revision=1.53&view=markup

...see at line 285...

Gilles Caulier

*** Bug 211066 has been marked as a duplicate of this bug. ***

In exiv2's action.cpp, there is a method 
bool isExifTag(const Exiv2::Exifdatum& ed) (line 1797) to determine if a tag from ExivData is an Exif tag in TIFF-like files. It's used from metacopy(...) (line 1703), specifically for TIFF files (1736-1747).

Thanks Marcel for the tip.

Andreas, Exiv2 0.18.99 do not help here. Same problem. Tiff file generated are broken

Gilles Caulier

Marcel,

If i fell all tags listed in isExifTag() as untouched tags from KExiv2::save method, the tiff file is completly broken : 200kb intead 29Mb...

I'm completly lost. I'm pretty sure that Exiv2 touch something worng there. But what ?

Gilles

I tried to copy verbatim the code from metacopy() to libkexiv2, but still, gimp is complaining.

SVN commit 1059510 by cgilles:

disable temporaly tiff metadata using Exiv2.
CCBUGS: 211758


 M  +3 -1      kpwriteimage.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1059510

SVN commit 1059511 by cgilles:

idem in digiKam TIFFF writter
CCBUGS: 211758


 M  +2 -1      tiffloader.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1059511

(In reply to comment #15)
> I tried to copy verbatim the code from metacopy() to libkexiv2, but still, gimp
> is complaining.

Marcel, what does it complain about exactly? Can you show me the resulting picture and the modified code?

Andreas

Andreas, 

Have you take a look to my TIFF files shared on #9 ?

Gilles

Created attachment 38900 [details]
Metadata extracted from NEF with exiv2

Andreas, find attached metadata from a NEF file extracted with exiv2.
Then I have taken a TIFF image (http://digikam3rdparty.free.fr/TEST_IMAGES/TIFF/Solar_Spectrum.tiff) and inserted the metadata with "exiv2 in".
Afterwards, GIMP gives the typical error messages:

/home/marcel/Solar_Spectrum.tiff: invalid TIFF directory; tags are not sorted in ascending order
/home/marcel/Solar_Spectrum.tiff: wrong data type 7 for "Photoshop"; tag ignored
/home/marcel/Solar_Spectrum.tiff: wrong data type 7 for "Photoshop"; tag ignored

/home/marcel/Solar_Spectrum.tiff: unknown field with tag 11 (0xb) encountered
/home/marcel/Solar_Spectrum.tiff: unknown field with tag 18246 (0x4746) encountered
/home/marcel/Solar_Spectrum.tiff: unknown field with tag 18249 (0x4749) encountered
/home/marcel/Solar_Spectrum.tiff: unknown field with tag 11 (0xb) encountered
/home/marcel/Solar_Spectrum.tiff: unknown field with tag 18246 (0x4746) encountered
/home/marcel/Solar_Spectrum.tiff: unknown field with tag 18249 (0x4749) encountered
TIFF-Bild: /home/marcel/Solar_Spectrum.tiff: wrong data type 7 for "Photoshop"; tag ignored

The former three in a dialog, the latter on the console.
The picture itself can be opened by gimp, but there are other example images in some of the four bug reports merged here that are more severely broken.
So keep in mind there could be two distinct problems, but we should assume it's one problem and fix these warnings.

Marcel,

Thanks for the reproducer. Some preliminary analysis, below are the tags which cause the warnings, if I delete all four of them Gimp is happy. It looks like the complaint about tags not sorted in ascending order is a side-effect. I think libtiff simply doesn't know the first three and for the last one it expects a different type ('byte' instead of 'undefined' - which makes no difference - but I don't remember where I have 'undefined' from, will investigate).

Andreas


Unknown field
0x000b Exif.Image.ProcessingSoftware                Ascii      34  digiKam-0.10.0-rc1 (rev.: 891281)
0x4746 Exif.Image.Rating                            SLong       1  0
0x4749 Exif.Image.RatingPercent                     SLong       1  0

Photoshop tag
0x8649 Exif.Image.ImageResources                    Undefined 160  (Binary value suppressed)

Andreas, as an "emergency" measurement to work around this for our 1.0 release, should we just remove these four tags when saving a tiff? Or should we rather keep metadata saving disabled?

Marcel,

I had already disabled tiff metadata saving in digiKam and kipi-plugins...

Gilles Caulier

Marcel, Gilles,

For the first three of these tags, there is nothing wrong if digiKam writes these [1].
For the ImageResources tag, exiv2 seems to use the wrong type, although I have yet to find the specification. It's easy to fix in exiv2 if necessary. This tag may contain information that the users want to keep (written by Photoshop), it would be interesting to know if Photoshop still reads it with the changed type.

So these warnings are relatively harmless. They do not explain the corrupted images that some users have encountered.

-ahu.


[1] TIFF 6.0 Specification, Section 7: "Other fields. TIFF readers must be prepared to encounter fields other than those required in TIFF files. TIFF writers are allowed to write optional fields such as Make, Model, and DateTime, and TIFF readers may use such fields if they exist. TIFF readers must not, however, refuse to read the file if such optional fields do not exist. TIFF readers must also be prepared to encounter and ignore private fields not described in the TIFF specification."

> For the ImageResources tag, exiv2 seems to use the wrong type, although I have
> yet to find the specification. It's easy to fix in exiv2 if necessary.

Changed to BYTE: http://dev.exiv2.org/issues/show/661

(In reply to comment #25)
> > For the ImageResources tag, exiv2 seems to use the wrong type, although I have
> > yet to find the specification. It's easy to fix in exiv2 if necessary.
> 
> Changed to BYTE: http://dev.exiv2.org/issues/show/661

This last bugfix will solve the problem in opening digikam TIFFs with The GIMP?
Is this only available in SVN? On my Kubuntu Karmic I have exiv2-0.18.2-1, may I just compile the SVN source and install it or are there compatibility issues?

Thanks

No, this bug is not solved.

Repeated testcase from comment #20: Extracted metadata from a NEF and inserting this into Solar_spectrum.tiff, using the exiv2 command line tool, current exiv2 SVN.

Error message from gimp:
invalid TIFF directory; tags are not sorted in ascending order

Console output:
/home/marcel/Solar_Spectrum.tiff: unknown field with tag 11 (0xb) encountered
/home/marcel/Solar_Spectrum.tiff: unknown field with tag 18246 (0x4746) encountered
/home/marcel/Solar_Spectrum.tiff: unknown field with tag 18249 (0x4749) encountered

Marcel,

The issues mentioned in comment #27 are not due to exiv2 - they can easily be reproduced with any other tool.

As for tag 0xb, try the following:

1 - Take a (minimal) TIFF image.tif
2 - Add the tag with any program that can do this properly, e.g., with exiftool -ProcessingSoftware=exiftool image.tif
3 - Open in GIMP

And you'll get that "invalid TIFF directory; tags are not sorted in ascending order" error. The error itself is wrong - wherever it comes from. The image is perfectly ok, and the tags are correctly sorted in ascending order.

As for the warnings for the other two tags - 0x4746 and 0x4749 - refer to quote from the TIFF specs in comment #24. A TIFF-reader needs to be able to deal with unknown tags. Issuing a warning is fine I guess but the warning can be ignored, the image is fine.

It would cripple exiv2 if I'd disallow writing non-standard TIFF tags. Instead if you decide they should not occur in a TIFF image, digiKam needs to suppress them.

Andreas

Next is the error 'TIFF directory is missing required "ImageLength" field.'

The root cause for that message is the JPEG thumbnail which is included in these digiKam generated TIFFs as a second "page" (IFD1). Presumably these tags were copied from the Exif metadata of a JPEG with an Exif thumbnail.

This can be fixed by deleting all Exif.Thumbnail.* tags (the 2nd "page") from the TIFF image (or not writing them in the first place). That can be done by digiKam or exiv2, not sure yet if it should be in exiv2 (as part of http://dev.exiv2.org/issues/show/668, will find out when it comes to TIFF-like RAW images).

Finally, the "emboss effect":

Apparently 'Adobe Deflate' compression requires the 'Predictor' field, which seems to get removed in the libkexiv2/Exiv2 call. The broken images can be fixed by adding the tag again with a Short value 2, e.g., 

exiv2-0.19 -M'set Exif.Image.0x013d Short 2' MINOLTA-DYNAX7D-04.tif

using the sample from Gilles in comment #9.

This issue is fixed in the current Exiv2 trunk revision - Exiv2 now differentiates between image tags and metadata, and does not modify any image tags (http://dev.exiv2.org/issues/show/668). And the 'Predictor' tag is an image tag.

(Note that also means the command above won't work with the current exiv2 from trunk - it will ignore the command because it attempts to set an image tag.)

Gilles, following the steps from comment #9 you should not be able to reproduce this emboss effect anymore using exiv2 from the trunk. Can you confirm that?

Andreas

To Andreas, 

- comment #28 : "invalid TIFF directory; tags are not sorted in ascending
order" error come from libtiff. gimp and digiKam delegate all tiff loading code to libtiff.

- comment #30 : I will check it tomorrow morning.

Gilles

Thanks Andreas for having a look into this. It's an important bug for us.

Two questions:

- do you "recommend" that we remove Exif.Thumbnail.* when saving a TIFF with our TIFF loader? In that case, we'll do that. I dont know better than you.
- is there a workaround for the Predictor tag in our TIFF loader for older exiv2 versions? Or is the field removed so early that we can't know it's been there?

I'm still a bit unsure about the Gimp warnings from libtiff, but I can't easily know where they come from either.

Re comment #32: 

> - do you "recommend" that we remove Exif.Thumbnail.* when saving a TIFF with our TIFF loader?

Yes, that way it will also work with older versions of exiv2.

> - is there a workaround for the Predictor tag in our TIFF loader

If the TIFF is written as an 'Adobe deflated' compressed TIFF, i.e., the TIFF image written by libtiff has an Exif.Image.Compression tag with a Short value 8 then digiKam can add (or prevent deletion of) the Predictor tag. It seems to have a Short value of 2 (but I've only determined that empirically with just a few TIFFs, so not sure if it's really always 2 - I'm not into TIFF image data compression algorithms).

More general, when older versions of exiv2 are in use, digiKam could prevent modifications to the image tags of TIFF images itself, similar to what exiv2 is doing in Exiv2::Internal::TiffHeader::isImageTag() - http://dev.exiv2.org/repositories/entry/exiv2/trunk/src/tiffimage.cpp#L1643 (this is the newer version, IIRC you once implemented something like this based on the previous code)

I believe this issue is also related to bug #183171 and may go away once that bug is fixed.

Andreas

SVN commit 1113706 by mwiesweg:

Remove all Exif.Thumbnail.* tags when writing TIFF.

CCBUG: 211758

 M  +10 -1     tiffloader.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1113706

183171 should now be fixed.
To test if this actually fixes the problem with the Predictor tag I need to use an older version from exiv2 trunk? Do you know an SVN revision to downgrade to?

> 183171 should now be fixed.

Yeah!

> To test if this actually fixes the problem with the Predictor tag I need to use
> an older version from exiv2 trunk? Do you know an SVN revision to downgrade to?

Try exiv2 r2036 (just before the 1st commit for #668)

-ahu.

This want mean that TIFF writter drop Exif thumbnail creation ? 

It's will be generated by Exiv2 after libtiff code instead ?

I think Exif thumbnail creation is important there, to follow properly TIFF/EP paper from ISO.

Or i forget something there ?

Gilles Caulier

Gilles: I dont know about this. I let Andreas answer this question.

Andreas: I'm not sure how to test this. Perhaps the original reporter here can give a hint if the problem is fixed with latest versions.

With Digikam 1.2.0 (SVN of 03 April 2010) I can open TIFFs with The Gimp without any problem (Gimp was complaining about TIFF tags - the bug was merged with the current one). The Gimp issue can be marked as solved.

Thank you.

In 1.2.0 we dont write the metadata to the created TIFF to workaround this issue.
So it wasn't fixed there, but the feature disabled.

Re comment #37: Thumbnails

The standards regarding this are conflicting. There are at least four different specifications to consider:

TIFF/EP and DNG
---------------
Recommend thumbnail in IFD0, main image in a sub-IFD of IFD0.
Chained IFDs are used for multiple non-identical images (each of which using this sub-IFD tree structure)

Adobe PageMaker TIFF Technical Notes
------------------------------------
(also referred from the Photoshop Technical notes)
Recommend main image in IFD0, thumbnail in a sub-IFD of IFD0.
Chained IFDs are used as above.

Original TIFF
-------------
Only provides for chaining, the sub-IFD tag doesn't exist in the original TIFF specification. It doesn't mention thumbnails, but the most obvious place to store one is in a chained IFD, as specified in the Exif standard.

Exif
----
IFD0 contains only metadata, no image data. The Exif thumbnail is stored in a chained IFD1. In reality, the Makernote structure of many cameras contain further thumbnails.

---

The thumbnail that Marcel disabled was in a chained IFD1 and not recognized by GIMP as such - GIMP looks at a chained IFD as a second non-identical image ("page") within the TIFF file, as recommended by the TIFF/EP and DNG standards.

Exiv2 only has support to read thumbnails which already exist in a TIFF image. It doesn't generate thumbnails from the main image (e.g., after that was modified). I'm not familiar with libtiff, but that would be the next place to look for such a feature it seems: 

Does libtiff have a feature to generate a thumbnail from the main image and store both main image and thumbnail in the appropriate places within the TIFF image?

Andreas

> Andreas: I'm not sure how to test this. Perhaps the original reporter here can
> give a hint if the problem is fixed with latest versions.

See Gilles' comment #9. I suspect converting a RAW file to a compressed TIFF is a way to reproduce the problem.

Andreas,

>Does libtiff have a feature to generate a thumbnail from the main image and
>store both main image and thumbnail in the appropriate places within the TIFF
>image?

I think no. libtiff is really low level API. You need to do it by hand, as i do with code in digiKam tiff writer.

Gilles

So, how to create "a chained IFD as a second non-identical image ("page") within the TIFF file"? Assuming I have the preview image data generated, do I add it with exiv2 or with libtiff?

Marcel,

That "chained IFD" is what digiKam generates today. I think it's not the correct way to create a thumbnail. Instead I suggest to create the thumbnail either (1) in IFD0 according to TIFF/EP standard or (2) in a sub-IFD as suggested in the Adobe PageMaker TIFF Technical Note.

Using libtiff, see the file tools/thumbnail.c for an example. I believe it creates a thumbnail according to method (2).

With Exiv2, conceptually you do the same, i.e., add the relevant tags, but I don't have a good sample. I don't mind writing one, it's a good use-case. Will post it here.

Andreas

Created attachment 42899 [details]
Sample program: Add a (dummy) JPEG thumbnail to a TIFF image with Exiv2

The attached sample adds a JPEG thumbnail in the first sub-IFD of a TIFF image. It only adds the most basic tags, a real application may want to add a few more (eg, X/YResolution, ResolutionUnit). All of the applications I tried are happy with a TIFF image modified with this sample, but none actually seems to use the thumbnail.

Andreas

Thanks a lot Andreas.

So what I will do with TIFFs is:

- use ExifThumb::erase() to remove all IFD1 entries
- use your code example to add an IFD0 thumb.

With all other image formats except for TIFF, we add a thumb with ExifThumb::setJpegThumbnail.
For all images including TIFF, we set a (larger) preview in Iptc.Application2.Preview.

Is that ok?

SVN commit 1124097 by mwiesweg:

Use dedicated methods from libkexiv2 to set a TIFF thumbnail.
Reenable metadata writing to TIFF for testing (not all problems solved).

Provide original image size for JPEG and RAW from DImg.
For RAWs, size is read with Exiv2.

CCBUG: 211758

 M  +21 -0     dimg/dimg.cpp  
 M  +4 -0      dimg/dimg.h  
 M  +2 -0      dimg/loaders/jpegloader.cpp  
 M  +3 -0      dimg/loaders/pgfloader.cpp  
 M  +2 -7      dimg/loaders/tiffloader.cpp  
 M  +11 -2     threadimageio/previewtask.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1124097

I have no followed the suggestions to add a thumbnail to a TIFF file, and Gimp does no longer show to images for the TIFF.

However, the "not sorted in ascending order" problem is still unsolved!
As a testcase, I used the MRW file from Gilles from comment #9. Importing from RAW, saving as TIFF, and Gimp shows an error message about tag sort order.

Andreas, after some considerations and a bit reading the docs, I have these ideas:

As soon as I remove the 0xb tag, all user-visible GIMP warnings are gone.
0x000b is officially defined as a TIFF TAG, the GPS tag GPSDOP. Libtiff may expect such a tag in the GPS IFD, and is then right in its warning about the wrong order.
As I understand, this tag was created by ACDsee and not by any official standard?

In any case, I suggest we remove this tag when writing a TIFF. This should finally solve the remaining problems here.

Marcel, yes not writing the 0x0b tag is a workaround to prevent the libtiff error. However the issue is clearly a small libtiff bug - it wrongly says the tags are not sorted (just remembered I was actually looking for a place to file this one weekend, but got sidetracked somehow...) The rule is simply that tags within an IFD must be sorted. A tag with a given number in one directory is not in any way related to another tag with the same number in a different directory.

Andreas

SVN commit 1129105 by mwiesweg:

Remove the ProcessingSoftware tag (0xb) when writing a TIFF file, because
it makes libtiff generate a warning which is then user-visible in Gimp.

Now finally, the Gimp is happy again with our TIFF files.

BUG: 211758

 M  +2 -1      NEWS  
 M  +2 -0      libs/dimg/loaders/tiffloader.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1129105