Hi,

I have already reproduce this problem in the past. I have suspected a bug in Exiv2 library used by digikam to manage metadata. But not sure...

Can you give me the Exiv2 release installed in your computer ? 0.10.0 ?

Here, to develop digiKam, i'm using current Exiv2 implementation from svn.

Gilles Caulier

I currently have exiv2-0.10.tar.gz   loaded .

I just loaded exiv2 from cvs, made clean and recompiled.
No Joy.  Same results.

Am Montag, 10. Juli 2006 16:37 schrieb J.Westveer:
[bugs.kde.org quoted mail]
I have the same problem, Gilles, with the latest exiv2 from svn, and the 
problem existed at east for the last 5 revisions of exiv2.

Gerhard

This is what exifprobe says for bad.jpg, note the "INVALID JPEG TAG":

@0x000614f=24911   :    <JPEG_APP13> 0xffed length 44828, Photoshop 3.0 - unknown format (not dumped: use -A)
-0x001106c=69740   :    </JPEG_APP13>
@0x001106d=69741   :    <TAG_0xb060> INVALID JPEG TAG
-0x04389e6=4426214 :  END OF FILE
@000000000=0       :  Start of JPEG (UNKNOWN JPEG compression) primary image [0x0] length 0 (CORRUPTED) (no image)

With the standalone exiv2 utility I can modify and write good.jpg (bot Exif and IPTC metadata) without problems. 

Further, good.jpg (unmodified) starts with 

00000000  ff d8 ff e1 d1 b1 45 78  69 66 00 00 4d 4d 00 2a  |ÿØÿáѱExif..MM.*|
00000010  00 00 00 08 00 0c 01 0e  00 02 00 00 00 1e 00 00  |................|

but bad.jpg starts with:

00000000  ff d8 ff e0 00 10 4a 46  49 46 00 01 01 00 00 01  |ÿØÿà..JFIF......|
00000010  00 01 00 00 ff e1 61 39  45 78 69 66 00 00 4d 4d  |....ÿáa9Exif..MM|

Exiv2 does not insert a JFIF segment, so this image was not written with Exiv2 (only).

Finally, with "dd skip=30 bs=1 if=bad.jpg of=bad.tif count=24881" I can extract the Exif data from bad.jpg and parse it with exiv2 -pt bad.tif and it looks ok.

Andreas

This is what exifprobe says for bad.jpg, note the "INVALID JPEG TAG":

@0x000614f=24911   :    <JPEG_APP13> 0xffed length 44828, Photoshop 3.0 - unknown format (not dumped: use -A)
-0x001106c=69740   :    </JPEG_APP13>
@0x001106d=69741   :    <TAG_0xb060> INVALID JPEG TAG
-0x04389e6=4426214 :  END OF FILE
@000000000=0       :  Start of JPEG (UNKNOWN JPEG compression) primary image [0x0] length 0 (CORRUPTED) (no image)

With the standalone exiv2 utility I can modify and write good.jpg (bot Exif and IPTC metadata) without problems. 

Further, good.jpg (unmodified) starts with 

00000000  ff d8 ff e1 d1 b1 45 78  69 66 00 00 4d 4d 00 2a  |ÿØÿáѱExif..MM.*|
00000010  00 00 00 08 00 0c 01 0e  00 02 00 00 00 1e 00 00  |................|

but bad.jpg starts with:

00000000  ff d8 ff e0 00 10 4a 46  49 46 00 01 01 00 00 01  |ÿØÿà..JFIF......|
00000010  00 01 00 00 ff e1 61 39  45 78 69 66 00 00 4d 4d  |....ÿáa9Exif..MM|

Exiv2 does not insert a JFIF segment, so this image was not written with Exiv2 (only).

Finally, with "dd skip=30 bs=1 if=bad.jpg of=bad.tif count=24881" I can extract the Exif data from bad.jpg and parse it with exiv2 -pt bad.tif and it looks ok.

Andreas

Andreas, 

I i understand what you mean, is this a bug in digiKam JPEGLoader implementation... right ?

Gilles

Gilles,

This was just the information gathered from playing around with exiv2, without using digikam. 
The question to further investigate is where does the INVALID TAG come from? 
How does digikam write the IPTC segment and what does it do right after that?

Regards,
Andreas

Andreas,

digiKam work like this :

- In editor, all image data/metadata are stored in memory using a DImg class instance. 

http://websvn.kde.org/trunk/extragear/graphics/digikam/libs/dimg/dimgprivate.h?rev=545648&view=auto

- To save into a new JPEG file, a JEPGLoader::save() is called :

http://websvn.kde.org/trunk/extragear/graphics/digikam/libs/dimg/loaders/jpegloader.cpp?rev=557529&view=auto

JPEG image loader is derivated from generic DimgLoader loader class (a friend of DImg) :

http://websvn.kde.org/trunk/extragear/graphics/digikam/libs/dimg/dimgloader.cpp?rev=557998&view=auto

- The JPEG Loader is simple and do not store the metadata in the new file. It create the the new and store the image data and the icc profile. The file is closed and the DImgLoader::saveMetadata() method is called to store Exif and IPTC (look at end of JPEGLoader::save() method.

- In fact the metadata are stored using Exiv2, by the famous DMetadata class (:=))). Do you remember ?

http://websvn.kde.org/trunk/extragear/graphics/digikam/libs/dmetadata/dmetadata.cpp?rev=556687&view=auto

I suspect a problem in DImgLoader::saveMetadata() calling DMetadata methods. Your viewpoint ?

Gilles

For me, this appeared once I switched from Exiv2 0.9.1 to 0.10.  This would imply to me that it might be a change in how Exiv2 presents or writes some of the data (when integrated into digikam).

Duncan report is interressing. I cannot test here to back on Exiv2 0.9.1. I'm not i'm home and my internet connection/time is limited during some day. 

Somebody can confirm this point ?

Duncan, can you confirm this problem is relevant of the target JPEG file size ?

Gilles

If I start with a 2.3 MB JPEG source, resize to 50% and re-save to a 385KB JPEG, the resulting JPEG is intact.

If I start with a 8.8 MB RAW file, and save to PNG, I get a 27 MB file with no tags at all (EXIF, Makernote or ITPC), but it's fine other than Exiv2 not liking it (digikam: Cannot load metadata using Exiv2 (/home/dhill/Images/2006/2006-07-10/z.png: The file contains data of an unknown image type))

If I start with a 8.8 MB RAW file, and save to JPEG with no changes, I get a 781 KB JPEG that's intact.

If I start with a -different- 8.8 MB RAW file, and save to JPEG with no changes, I get a corrupt JPEG.  If I resize to 50% I get a corrupt JPEG.

The correlation between corrupt and not corrupt appears to be the 'density' of colour data in an image.  The image that produces good JPEGs when saved is just a bowl of food.  The image that produces bad JPEGs when saved is a big yellow truck.

Resources to help test:
http://www.cricalix.net/~dhill/digikam/good_src.raw
http://www.cricalix.net/~dhill/digikam/good_src.jpg
http://www.cricalix.net/~dhill/digikam/bad_src.raw
http://www.cricalix.net/~dhill/digikam/bad_src.jpg

If I downgrade to Exiv2 0.9.1 using svn (by checking out the appropriate tag), and use the 'bad_src.raw' to generate a JPEG, I get a good JPEG.

0.10 is revision 846.
0.9.1 is revision 694.

I'll see if I can do a quick binary walk of the revisions to find where the problem kicks in (though anything related to compiling c++ is not quick, even on a Xeon).

Another data point - it's related to the embedding of ICC data into the JPEG.

Digikam          : 561569
Exiv2            : trunk 846
Source image     : bad_src.raw

Colour management: off
RAW to JPEG      : Intact, no corruption.
Colour management: on
RAW to JPEG      : Corrupted.

Digikam          : 561577
Exiv2            : trunk 750
Source image     : bad_src.raw 
 
Colour management: off 
RAW to JPEG      : Intact, no corruption. 
Colour management: on 
RAW to JPEG      : Corrupted. 

I have tried some brute-force debugging on this one, using the "good.jpg" from above, which reproducably produces a broken JPEG here.

First, removing image->setIptcData in dmetaloader.cpp, line 145 would fix the bug.
Then (good guess) I removed meta.setImagePreview in dimginterface.cpp, line 518. So it seems it is related to the embedded image preview, at least reproducably on my machine with the given file.

ok. So it is not a "Size" related problem.  Now that you point it out, this file
was read in with digikam (svn) because it has IPTC data in it. If I would have just mounted and copied the file, and not modifyed it, it would NOT have had IPTC data.

So we have a jpg, that was created by loading with digikam, that CAN be read and printed by digicam and other programs, but for some reason it CAN NOT be saved, because something is wrong with the (good.jpg) file.

This does change the description of the bug, but a problem still exists if you load a file with digikam, and then later can not modify/re-save it.

Thoughts?

I think is a bug is Exiv2. Here, only when the IPTC image preview is larger than 65536 bytes the JPEG gets corrupted. As a (hack|workaround|test)?
changing line 1517 in
 
libs/dmetadata/dmetadata.cpp to:

preview.save(previewFile.name(), "JPEG", 50);

gives me always a smaller than 64K preview, and no corrupt JPEG.

And, BTW about comment #16 from J.Westveer: my original files doesn't have
any IPTC data.

Florencio has a good point with his second statement: The size of a JPEG segment is limited to 64k (because the size of the segment is stored in a 16 bit number). So saving a >64k Thumbnail in an APP13 segment is definitely not possible.

Now, as far as Exiv2 is concerned, it should allow to set IPTC data which is larger than 64k (the limit for the Preview field is 256000 bytes according to the specs) and it should allow to copy that IPTC data data buffer in a binary format. However, it should NOT allow to write an IPTC block larger than 64k to a JPEG image (using Image::writeMetadata). I'll check if it follows this specs later but does digikam actually use writeMetadata or does it write the binary block obtained from Exiv2 using its own logic? (I know, it's probably the famous DMetadata class again :) but I can't check it out right now)

Andreas

I tried comment #17 from Florencio Guimaraes, with "JPEG", 50.
This did not work with my test file (good.jpg).  However "JPEG", 20 did work,
and the image is once again editable/saveable.

Ok, this is a bug in Exiv2. In the (old) version of dmetaloader.cpp that I have here, digikam uses (Jpeg)Image::writeMetadata and that function does not check the size of the segment (around line 500 of jpegimage.cpp); the result is a corrupted JPEG image as reported.
The fix in Exiv2 will probably simply be to throw an exception and digikam will need to either proactively check the size of the IPTC data or handle the exception.

Unfortunately, I don't have time to fix this immediately (Gilles please feel free to do so if you have more time), I can only do it next week.

Andreas

Created attachment 16982 [details]
my local-temporary-fix-workaround

This way, we get the best quality given the 64K limit to preview.
Additionally, get rid off a KTempFile.

diff patch works here.  tnx.

jjw

Andreas,

I'm not at home this week (hollidays). I just read my mail, and others internet stuff.

Next week, with Marcel, we will prepare the 0.9.0-beta1 release. I will disable IPTC preview record with JPEG target file (and JPEG target only because it work fine with PNG and TIFF) for this release until an Exiv2 issue will be find.

Friendly

Gilles

*** Bug 130728 has been marked as a duplicate of this bug. ***

As I understand this, there is a bug in libexiv2 because it does not check the size, but that's a minor problem, it is digikam's responsibility to check the size of the preview. Even if libexiv2 is fixed, digikam still needs to make sure that if the image is a JPEG, the preview may not be larger than 64k.

DMetadata does not know the image format, so the limit needs to be passed to the setImagePreview() method. This method is called from DImgInterface, which knows the file format.

For other formats, PNG, TIFF, the size is not limited?

Marcel,

To have checked libPNG, i have not seen any size restriction with text chuck used to save Exif and IPTC byte-array.

About tiff file, the spectification do not said about any restrictions (look the url links annoted in Digikam::DImg::TiffLoader), but i'm not sure... Andreas, can you confirm this point ?

Gilles

SVN commit 562917 by cgilles:

digikam from trunk : disabled temporally the IPTC preview tag recoring in JPEG target file duing an uncompatibility between the IPTC preview tag size (256K max) and the JPEG segment size used to store IPTC byte array (64K).

A solution need to be found in Exiv2 to prevent this problem!

CCMAIL: digikam-devel@kde.org, ahuggel@gmx.net
CCBUGS: 130525

 M  +13 -2     dimginterface.cpp  


--- trunk/extragear/graphics/digikam/utilities/imageeditor/canvas/dimginterface.cpp #562916:562917
@@ -514,9 +514,20 @@
     meta.setIptc(d->image.getIptc());
 
     // Update Iptc preview.
-    QImage preview = d->image.smoothScale(800, 600, QSize::ScaleMin).copyQImage();
-    meta.setImagePreview(preview);
 
+    // TODO: see B.K.O #130525. The a JPEG segment is limited to 64K. If IPTC byte array 
+    // bigger than 64K duing of image preview tag size, the target JPEG image will be
+    // broken. Note that IPTC image preview tag is limited to 256K!!!
+    // Temp. solution to disable IPTC preview record in JPEG file until a right solution 
+    // will be found into Exiv2.
+    // Note : There is no limitation with TIFF and PNG about IPTC byte array size.
+
+    if ( !mimeType.upper() == QString("JPG") || !mimeType.upper() == QString("JPEG") ) 
+    {
+        QImage preview = d->image.smoothScale(800, 600, QSize::ScaleMin).copyQImage();
+        meta.setImagePreview(preview);
+    }
+
     // Update Exif thumbnail.
     QImage thumb = preview.smoothScale(160, 120, QImage::ScaleMin);
     meta.setExifThumbnail(thumb);

Created attachment 17010 [details]
patch to adapt preview size

Inspired by the patch from Florencio, this patch adapts the preview file size
if the image is a JPEG and if the 64k limit is exceeded.

It trades off size and/or quality and makes five attempts at reaching the
limit. The exact parameters may be tuned, I am currently unsure which preview
sizes and which quality factors are still acceptable.
A quality of 0 is not acceptable btw, I tried that and the whole image is a
block artefact. I would prefer to downsize the preview then a bit.

Even if libexiv2 is fixed in the future, it will simply throw an exception, I
think we still need to do something like this if we really want to embed a
preview.

Created attachment 17011 [details]
this version compiles

Marcel,

This is my suggestions :

- using a jpeg compression ratio value under 50 give a lot of artifact.
- current preview size is 800x600. A size of 640*480 is also right to render on the screen (it used by Minolta camera). 
- using a preview size under 640*480 is not adapted. To optimize in all case, i recommend to use 640*480 by default instead 800x600 with JPEG files.
- unforget that IPTC is used also to store other metadata like comments, tags, rating, and others stuff. We need some place for that in iptc array. And this is the most important than preview.

The iptc spec is available here :

http://www.iptc.org/std/IIM/4.1/specification/IIMV4.1.pdf

Look page 45 and 47.

Gilles

SVN commit 563680 by cgilles:

digikam from trunk : missing to disable temporally IPTC preview record during download from camera to prevent broken JPEG image if autorotation is used.

Marcel, we need to unforget to fix this part too when a right solution will be found.

CCBUGS: 130525


 M  +7 -1      exifrotate.cpp  


--- trunk/extragear/graphics/digikam/libs/jpegutils/exifrotate.cpp #563679:563680
@@ -230,7 +230,13 @@
 
             // Update the image preview.
             QImage preview = img.scale(800, 600, QImage::ScaleMin);
-            metaData.setImagePreview(preview);
+            
+            // TODO: see B.K.O #130525. The a JPEG segment is limited to 64K. If IPTC byte array 
+            // bigger than 64K duing of image preview tag size, the target JPEG image will be 
+            // broken. Note that IPTC image preview tag is limited to 256K!!! 
+            // Temp. solution to disable IPTC preview record in JPEG file until a right solution 
+            // will be found into Exiv2. 
+            // metaData.setImagePreview(preview);
 
             // Update the image thumbnail.
             QImage thumb = preview.scale(160, 120, QImage::ScaleMin);

This is fixed in Exiv2 rev 847. Exiv2 now throws Exiv2::Error(37) if any JPEG APP segment is >64k but it allows to work with metadata which is larger.
http://dev.robotbattle.com/cmtinfo_svn.php?r=10&v=847

Please check the use of Exiv2::Photoshop::locateIptcIrb and locateIrb in digikam: the signatures of these members were fixed to use uint32_t from uint16_t

#26 : I also can't find any evidence that the size is limited in TIFF format.

Since IPTC is first packed into a Photoshop IRB (specified by Adobe) before it is written to the JPEG APP segment, I am wondering what Adobe applications do in this situation. The available Adobe specs are not clear.

So if anyone can test with Photoshop or similar Adobe app, I'd be interested to hear what happens if you try to save IPTC data >64k to a JPEG image, in particular is it split into two APP13 segments?

-ahu.

Is this possible to split into two APP13 segments if size of IPTC is > than 64K ? 

Note : ImageMagick remove completly the IPTC segment if the size is > 64K. To test, i have generated a PNG file with an IPTC profile embeded (85Ko) and i have converted it to JPEG. ImageMagick said that IPTC segemnt exit but do not put it in target JPEG file...

Gilles

Marcel,

I think than i havethe ultimate solution : using JPEG2000 instead JPEG like preview image format embeded.

If you read the IPTC spec, the preview support JPEG2000 format. Since we have a native  JPEG2000 encoder in digiKam core, we can do it easily. Advantage :

- Better quality.
- Better compression ratio.
- And the most important : we can specify to JPEG2000 codec the target image data size in bytes instead the image quality. The quality settings is automaticly computed by the codec. This way is not possible with JPEG codec. Like this we can limit preview data size to 10Kb for example. We will have 54Kb to store the rest of IPTC data. I think it will always be right.

To decode JPEG2000 preview image with Qt is not a problem. Since KDE 3.3, JPEG2000 decoder (not encoder) is provided by KDE API. Normally, we will change nothing in the preview image loader implementation.

What do you think about ?

Gilles

It's in the IPTC spec, so its standard-compliant, easy to code, and better quality for less size. That sounds great!

Marcel,

I have implemented the JPEG2000 Preview in my computer. It work fine, the quality and the size of the preview image are better (small and good), but there is a major problem with this solution :

It tooooooo long to decode JPEG2000 image preview with jasper library.

This is want mean than JPEG format is the only solution that we can use for preview. JPEG decoder is really more optimized than JPEG2000.

I will review again your patch to include it in libkexiv2. Perhaps we need to use a dedicaced and optimized JPEG codec to render a small and good JPEG preview file (instead to use the simple QT codec)

All suggestion welcome for that (link to implementation, libjpeg options optimization etc...)

Gilles

Marcel,

Since we use the fast JPEG image decoder part witch extract a reduced image from the whole image, i think than embbeding perview in JPEG is not necessary.

The JPEG preview mode work very well like this in digiKam. It's fast and suitable to browse/slide an image collection.

What do you think about ?

Gilles

I agree, loading JPEGs is very fast here.

Oh, the 64k limit is only a problem for JPEGs? Then this bug could finally be considered solved?

Yes Marcel, 64Kb size is only for JPEG sections witch have this limitation. PNG, TIFF don't have this problem.

If you save a PNG/TIFF image from a picture with editor, the IPTC preview is created and stored as well without problem. This preview image is used in Preview mode to render picture on screen.

Gilles