Bug 204437

Which libkdcraw and libraw version you use ? Go to Help/Components Info for details

Gilles Caulier

Pasted info window:

digiKam version 1.0.0-beta3
Eliminación de mosaico parallelizada: Sí
Exiv2 puede guardarse en JP2: Sí
Exiv2 puede guardarse en Jpeg: Sí
Exiv2 puede guardarse en PNG: Sí
Exiv2 puede guardarse en TIFF: Sí
Exiv2 se puede escribir a Pgf: No
Exiv2 soporta  metadatos XMP: Sí
LibCImg: 130
LibExiv2: 0.18.1
LibJPEG: 62
LibJasper: 1.900.1
LibKDE: 4.3.00 (KDE 4.3.0)
LibKExiv2: 0.6.0
LibKdcraw: 0.5.0
LibLCMS: 118
LibPGF: 6.09.24
LibPNG: 1.2.37
LibQt: 4.5.2
LibRaw: 0.7.2
LibTIFF: LIBTIFF, Version 3.8.2 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Widget Marble: 0.8
LibGphoto2: 2.4.6
LibKipi: 0.4.0

I haven't found any similar previous bug report about this, but i notice this extrange behavior with kipi-plugins 0.20 (ubuntu jaunty packages).

Goose.

Ok, try to update to Exiv2 0.18.2. libkexiv2/ kipi-plugins, and digiKam need to be recompiled.

For libraw, to have a last version 0.8.0, you must use libkdcraw from trunk.

Anyway, use kipi-plugins 0.5.0, not 0.2.0

Gilles Caulier

Ok, i can reproduce the problem with last libraw too.

./raw2dng.shell olympus.orf 
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Loading RAW data from  "olympus.orf"
<unknown program name>(11093)/ KDcrawIface::KDcrawPriv::progressCallback: LibRaw progress:  Reading metadata  pass  0  of  2
<unknown program name>(11093)/ KDcrawIface::KDcrawPriv::progressCallback: LibRaw progress:  Reading metadata  pass  1  of  2
<unknown program name>(11093)/ KDcrawIface::KDcrawPriv::progressCallback: LibRaw progress:  Reading RAW data  pass  0  of  2
<unknown program name>(11093)/ KDcrawIface::KDcrawPriv::progressCallback: LibRaw progress:  Reading RAW data  pass  1  of  2
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Raw data loaded:                                    
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Data Size:      20832000  bytes                            
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Date:           "2009-07-26T20:16:46"                      
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Make:           "OLYMPUS"                                  
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Model:          "E-420"                                    
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Size:           3720 x 2800                                
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Orientation:    0                                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Top margin:     0                                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Left margin:    0                                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Filter:         "RGGBRGGBRGGBRGGB"                         
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Colors:         3                                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Black:          68                                         
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- White:          4055                                       
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- CAM->XYZ:                                                  
<unknown program name>(11093)/ DNGIface::DNGWriter::convert:                     "0.8746  -0.2425  -0.1095"                 
<unknown program name>(11093)/ DNGIface::DNGWriter::convert:                     "-0.7594  1.5612  0.2073"                  
<unknown program name>(11093)/ DNGIface::DNGWriter::convert:                     "-0.1780  0.2309  0.7416"                  
<unknown program name>(11093)/ DNGIface::DNGWriter::convert:                     "0.0000  0.0000  0.0000"                   
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Formating RAW data to memory                        
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG memory allocation and initialization            
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG IFD structure creation                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG Negative structure creation                     
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Updating metadata to DNG Negative                   
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Backup Makernote ( 651928  bytes)                   
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Build DNG Negative                                  
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG preview image creation                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG thumbnail creation                              
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Creating DNG file  "olympus.dng"                    
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Backup meta-data using Exiv2                        
<unknown program name>(11093)/ KExiv2Iface::KExiv2::save: File Extension:  "dng"  is supported for writing mode             
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG conversion complete...

Alex,

ORF file come from E-420 model. What do you think about. 

picture converted is green. sound like an inverted color chanel somewhere.

"RGGBRGGBRGGBRGGB" is right for you as color filter ?

Gilles Caulier

I'm preparing my build environment with virtualbox, so i can test everything as i get info.
By the way, what annoys me is not having this issue with all my orf files, but only some of them, and i cannot isolate the problem.

Should i upload more *.orf files with and without this issue?

Thanks,
Goose.

dcraw_emu converts this file without problems, so pixel order is right.

I've tested at work (windows xp workstation):
-opening DNG file with Camera Raw (adobe photoshop cs3) results in file working properly... no black/green...
-Opening DNG file with rawtherapee 2.4 fails exactly like in my ubuntu installation.
-Opening DNG with ufraw windows v0.15, wrong too...

Original orf works as expected with every software, but not the resulting dng converted with kipi.

In general, DNG file contains two sets of metadata:
 - EXIF (MakerNotes) block copied from original RAW
 - DNG-specific metadata (color matrices, black level and so on)

I guess,  Adobe uses DNG-specific block, but other tools may use EXIF block on already processed data.

Some progress :

If i comment this line :

http://lxr.kde.org/source/extragear/graphics/kipi-plugins/dngconverter/dngwriter/dngwriter.cpp#816

Now jpeg preview image embeded in DNG is fine. RAW data color still wrong.

If i uncomment this line, JPEG preview has the same side effect than RAW color. This is really strange because JPEG preview is computed with DNG sdk using common RAW image data extracted from ORF file, and stored in DNG in dedicated format.

So, there is problem with Exiv2 here, but not only... Andreas ?

Gilles Caulier

Created attachment 36294 [details]
this is the jpeg preview image extracted from DNG and displayed as well when Exiv2 metadata backpup is commented.

I remember the singularity of ORF makernotes : some pointers are link to Exif information, outside makernotes area. I will try to comment makernotes backup in DNG converter

Gilles Caulier

Created attachment 36295 [details]
I found the problem : it's makernote backup !!!

Andreas, what's your tip here ? I'm using Exiv2 0.18.1 on this computer. Exiv2 from trunk can help me better with ORF makernotes ?

Gilles Caulier

Note : lines commented about makernotes backup is from 

http://lxr.kde.org/source/extragear/graphics/kipi-plugins/dngconverter/dngwriter/dngwriter.cpp#816

to

http://lxr.kde.org/source/extragear/graphics/kipi-plugins/dngconverter/dngwriter/dngwriter.cpp#641

Gilles Caulier

Andreas,

Important point : If i lets makernotes backup code commented and if i uncomments Exiv2 metadata backup code, DNG file generated is small and corrupted.

There is definitively something wrong with Exiv2 here i think. Your viewpoint ?

Gilles Caulier

Gilles,

Re comment #13: I don't think 0.18.2 or trunk is different in this area. But using trunk for the analysis is better anyway.

Re comment #15: Is that the same issue as discussed here: http://dev.exiv2.org/boards/3/topics/show/201 i.e., at least the "necessary" image tags need to be written anyway.

Check if this has something to do with the whitebalance: load the "corrupted"
image in ufraw and change the whitebalance setting from "automatic" (which means it reads it from the metadata) to something else. Does it make a difference?

Andreas

I've tested with various tools, playing with a lot of settings (including white balance, exposure, etc) and result is the same. Is not like the image got "green tint"... better take a look the image here:

http://www.flasherodepalo.com/digikam/kipi_050_dng.jpg

Looks like when you overexpose or underexpose a photo, and there's no way to recover image data with exposure controls... (sorry if i cannot write better explanation, my english is really limited).

Ok, finished testing in virtualbox with latest svn, resulting the same error.

Built within virtualized ubuntu karmic 9.10.
Pasted components info from showfoto:

showFoto version 1.0.0-beta4 (rev.: 1013570)
Exiv2 can write to Jp2: Yes
Exiv2 can write to Jpeg: Yes
Exiv2 can write to Pgf: No
Exiv2 can write to Png: Yes
Exiv2 can write to Tiff: Yes
Exiv2 supports XMP metadata: Yes
LibCImg: 130
LibExiv2: 0.18.2
LibJPEG: 62
LibJasper: 1.900.1
LibKDE: 4.3.00 (KDE 4.3.0)
LibKExiv2: 1.0.0
LibKdcraw: 1.0.0
LibLCMS: 118
LibPGF: 6.09.24
LibPNG: 1.2.37
LibQt: 4.5.2
LibRaw: 0.8.0-Beta5
LibTIFF: LIBTIFF, Version 3.8.2 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Parallelized demosaicing: Yes


If there is anything i can try to help, test, build....

Gustavo,

There is nothing to test anymore with current code. bug is with markernote backup. just look my previous comments. You can comment lines in source code as described and test in your computer. It's simple to do...

Gilles Caulier

Sure it is, but the point of bug tracking is being able to find the guilty!
Just kidding ;)
BTW i don't think is dngconverter fault... i've make these 2 tests:

1-Using dcraw 8.86 with the converted dng, dcraw claims about not getting correct white balance BUT resulting TIFF is working OK (not white balanced but not Black/Green).

2.-With Adobe DNG Converter the resulting .dng is working right(as i said previously) BUT: if then I use exiftool to write the EXIF from the ORF to the DNG... the modified DNG is Black/Green again!!!

So... i guess is not dngconverter "feature"... just DNG feature when you inject it some type of exif metadata?

Where should it be informed/bugtracked/cryed?

Thanks again,
Goose.

Yes, the problem is in Adobe DNG spec about how to handle markernotes. Here Andreas or Alex can explain better than mine.

This problem is also specific to ORF makernote format which link data outside makernotes area, as Exif for ex. 

I remember athread with Andreas about this subject about to make a wrapper in Exiv2 to fix the problem. I think nothing is done currently in this way.

Gilles Caulier

The way i see it: DNG Spec is not made to have all the exif data embeded "the same way" as original raw files... and maybe this is the reason they save some (don't know how much of it) exif infos into XMP space...

Should kipi-dngconverter acts same way Adobe converter does, and store all exif info found in original raw (same info displayed in digikam) into XMP space?

Hi, i found the guilty!!!
Try the following to see if anyone can reproduce it:
-Convert original ORF file with Kipi-dngconverter
-Assure result is bad (black/green picture in every raw developer)
-Delete "Exif.OlympusIp.BlackLevel" tag with exiv2
(exiv2 -M"del Exif.OlympusIp.BlackLevel" file_kipi.dng)
-Test resulting file with digikam, lightzone, rawtherapee: no black/green thing.
-Make an export from both orf and modified dng to png and compared both... looks identical.

So... maybe a bug within exiv2 interpretation of Exif.OlympusIp.BlackLevel?

Good work, Gustavo! But don't jump to a conclusion just yet.

OlympusIp is a Sub-IFD, that means there is nothing in there for Exiv2 to interpret. Check with another tool to see if the values differ.

As Alex already said in comment #9, is the Exif data in the makernote still valid for the converted image data? In particular the black level value?

Andreas

In other words: which Exif data values does libraw/dcraw use and do they need to be converted too when the image data is converted?

-ahu.

There is serveral possible cases for DNG converter:

  - exact copy RAW data from RAW file to DNG file without any processing. In this case, camera metadata is better than DNG ones, so camera Exif should be preserved.
  - some RAW processing (black level subtraction, raw tone curves and so on). In this case black level should be set in DNG metadata, but corresponding data fields in camera metadata should be zeroed to avoid mis-interpretation for already processed data.
  - and, of course, demosaicing of RAW into full RGB bitmap.

I have not take look of Gilles' DNG converter sources (and will be unable to do it in nearest 2-3 days), so cannot say anything more definitive about this case yet.

@Andreas, i'm unable to catch you, sorry (maybe my english limitations :P ). I don't know if libraw/dcraw/other raw developers use that sub-ifd, so don't know why is causing so much trouble.
I'm programmer (well, web programmer though) and casual photographer, so i really don't know so much about exif "internals"... i only did some tests...
The only thing i know: removing Exif.OlympusIp.BlackLevel means that tag no longer appears (tested many exif/metadata "visualizers") but raw developers works as supposed to do...
 

@Alex, don't bother so much with Gilles DNGConverter... i did same tests using Adobe DNG Converter, injecting original orf exif data to it with exiv2 and the problem persisted. Then removed the infamous Exif.OlympusIp.BlackLevel and gone right as with dngconverter.

Same tests using exiftool instead exiv2, same results.

So maybe is missinterpretation from raw development softwares confused finding olympus specific exif into a file which should not have it?

I'd love using dng cause full xmp write support from Digikam (and so interoperability between other softwares) but maybe i'd better stop dreaming of that?

So, if the image data is processed for the DNG, it may be necessary to also adjust the (Exif) metadata, rather than just copy it.

The tests that Gustavo did with different image converters and different metadata tools indicate that libraw and exiv2 by themselves are ok, just not synchronized in this case.

-ahu.

@Andreas:
In general, it is not possible to adjust metadata in all cases.

AFAIK, dcraw (and all derived programs/libraries) takes precedence of camera metadata over DNG metadata in some cases. I've seen this with canon files converted to DNG with adobe converter.
LibRaw do it too to preserve dcraw compatibility (over correctness).

There is two possible ways to deal with it
 1) It is possible to turn off all RAW preprocessing in LibRaw in Gilles' DNG converter, so raw data in DNG will be same as in source RAW
 2) It is possible to change LibRaw to prefer DNG metadata over camera ones, so all LibRaw-users (digiKam and so) will be happy.

I will take closer look into the problem on weekend. Sorry, no spare time on work-week.

Look there : 

http://lxr.kde.org/source/extragear/graphics/kipi-plugins/dngconverter/dngwriter/dngwriter.cpp#398

This code use DNG sdk. If i do not set this value, DNG file is not suitable (black hole). Perhaps it's a special case with ORF and DNG tags is uncompatible with Olympus makernote where a duplicate tag is set ?

Gilles

Gilles, 

the problem is not with DNG black level, it is correct (as we can see if we delete Olympus black level data from DNG). The problem is with Olympus black level data if this data are present in DNG file.

There is a third possibility, afaics:

3) Delete Exif.OlympusIp.BlackLevel before writing the metadata to the DNG image

(Or set the value to 0, if that makes more sense. And apply the same treatment to other tags as needed). These values are apparently related to the raw image data and since that has been processed, they are no longer valid. The Exif data which makes sense for the converted image remains intact.

Conceptually it is similar to what we do when we read metadata from a TIFF file and write it to a JPEG - some tags are filtered.

Out of curiosity, what's the rationale for processing the image data only to some extent, as it is apparently done now, if it still requires specialized RAW converters (dcraw) to display the image later? The resulting DNG seems to still contain very much proprietary image data in this case. Why not convert the image data to an open format? Or am I missing something? 

Andreas

Andreas,

I guess, that Olympus is not the only problematic camera. There is some logical problems with other cameras too. So, deleting Oly black level data is only partial solution. It is better to find more generic one.

Maybe this make some sense...
After taking a  look to Adobe DNG Spec (http://www.adobe.com/products/dng/pdfs/dng_spec_1_3_0_0.pdf, page 22) I can see it uses BlackLevel 

(and other 3 related tags) in a different way than usual RAW files ... so the problem is a little bigger.

Maybe dcraw/libraw/etc use algorithms to demosaic dng files based in adobe specification, but in this case (don't know others) the value from 

BlackLevel is not suited to those calculations cause is a "different" blacklevel (in spec pdf note a "mapping raw values to linear reference 

values") cause is taken directly from camera's raw.


Possible "global" solutions as far as i'm an ignorant :)?

1.-DNG Should not have exif data from original raw files than interfere with the Adobe specs, so every tag in exif metadata from raw files 

should be converted u ommited if it make unconsistency with oficial spec (as noted by Alex in comment #31).

2.-Every raw developer (dcraw/libraw/etc) should adapt to not use those "duplicated" exif data when dealing with DNG files.

Drawbacks:
-In the first case, dngconverter and/or exiv2 should adapt/filter exif values according to official dng spec.
-In the second case... too many software develovers should be concerned about this issue... and maybe it isn't the correct methos cause we're breaking the official dng specs... but who knows.

Notes:
-I've tested zeroing instead deleting the problematic "Olympus BlackLevel" tag, resulting in "NO black/green" picture but not as good colour 

as when deleting the key (maybe related to Alex comment #33?)

-@Andreas, there are lot of discussion with DNG usage, take a look to Barry Pearson "Defend DNG to the max" page at http://barrypearson.co.uk/articles/dng/index.htm. In many cases, photographers wants to retain their digital negatives as "intact" as they can. In my case also want to be able to tag these negatives, fill xmp metadata, comments, ratings, etc, AND store those valuable informations within the file itself so i can see/use it in other softwares/workstations. With ORF i can only set ratings/comments/tags, no way to modify anything else AND the rating/comments/tags are stored into digikam database... so i can't see/use it outside Digikam. That's my personal point about using DNG, but there ar more out there :)

P.S.-One IMPORTANT thing i've noted (and posted at the begining of the bug file): I have "tons" of .orf files which converted to dng WITHOUT the problem mentioned here... but there are a bunch with the problem. Why sometimes right, sometimes wrong? Well, i've guessed why:
When my camera takes a picture and the resulting ORF has a BlackLevel value of (0,0,0,0), the converted DNG (with both DNG BlackLevel tag and Olympus BlackLevel tag having the same value) works allright.
But the problematic raw files has a BlackLevel of (68,68,68,68) (in both BlackLevel tags, dng and olympus)... Obviously, if i set both to (0,0,0,0) the image displays without the nasty black/green, but the color balance/white balance/black balance is not accurate.

Gustavo, the LibRaw's black level (used in Gilles' dng converter) is really right value for DNG specs. LibRaw linearizes RAW (where appropriate, e.g. Sony RAWs or PhaseOne ones) on the reading stage, so black level subtraction is the logical next step in terms of DNG processing model.

The only problem is camera metadata precedence over DNG metadata. This is 'feature' (or logical bug?) of dcraw and all dcraw-derived tools, including LibRaw. I know about this feature of LibRaw (I've see some problems with DNGs converted from Canon RAWs) but have choose to keep dcraw compatibility instead of fixing it.

I'll take closer look into this problem on weekend. May be, there is easy way to fix it, at least for particular cameras (e.g. Oly)

Alex, i know libraw could be patched to fix it, i'me very confident about you all developers ;)... but that only fixes software using libraw... so dcraw based programs won't be aware of the problem. (i know bug could be filed against them).

And the other point mentioned in my previous comment... why is this "precedence" affecting only photographs with blacklevel different to (0,0,0,0)? Why that "precedence" affects the image with blacklevel of (68,68,68,68) if BOTH the conflictive tags has the SAME value(68,68,68,68)?

There is something i cannot get into...

Once again, thanks for all your effort.

Andreas,

To simulate #23, I have just set this code after DNG file creation :

// See B.K.O #204437 : remove Olympus black level makernote tag.
meta.load(dngFilePath);
meta.setWriteRawFiles(true);
bool b = meta.removeExifTag("Exif.OlympusIp.BlackLevel", false);
kDebug() << "Exif.OlympusIp.BlackLevel found = " << b;
meta.applyChanges();

... tag is found but never removed. Tag in makernote still there.

Code to remove Exif tag is similar that actions.cpp code from Exiv2 :

http://lxr.kde.org/source/KDE/kdegraphics/libs/libkexiv2/libkexiv2/kexiv2exif.cpp#303

Note : line #318 is removed here.

Gilles Caulier

Andreas,

I just created a test program to test makernote tag erase fonction. As expected, it doesn't work, and i don't know why.

Program is there :

http://websvn.kde.org/trunk/KDE/kdegraphics/libs/libkexiv2/test/erasetag.cpp?view=markup

Executing test program give results below :

./erasetag.shell olympus.dng
main: Exif.OlympusIp.BlackLevel found =  true
main: Exif.OlympusIp.BlackLevel removed =  true
KExiv2Iface::KExiv2::save: File Extension:  "dng"  is supported for writing mode
KExiv2Iface::KExiv2::save: Exif is supported for writing mode
KExiv2Iface::KExiv2::save: Exif is updated

You can see that Olympus makernote tag is removed in memory, but after when exiv2::writeMetadata() is called in libkexiv2::save() method

http://lxr.kde.org/source/KDE/kdegraphics/libs/libkexiv2/libkexiv2/kexiv2.cpp#283

...DNG file still untouched (same size, same timestamp)

Note: if i set debug statement in this method, i can see that Exif data is set to Exiv2::Image instance and exiv2::writeMetadata() is called properly.

Testing with Exiv2 0.18.2 command line tool using 

exiv2 -M"del Exif.OlympusIp.BlackLevel" olympus.dng

...work fine. Where is the problem ?

Gilles Caulier

Gilles,

Looking at the code, I can't see the problem right now (it's 2am here, I'm obviously overlooking something). Suggest to add debug output to Exiv2::Image::writeMetadata to see if it's called in the first place and if the tag is there or not. Or step through KExiv2::save.

Andreas

Gilles,

The problem is here:

http://lxr.kde.org/source/KDE/kdegraphics/libs/libkexiv2/libkexiv2/kexiv2.cpp#368

The loop copies all tags from d->exifMetadata except for those that are in the untouchedTags list. But this loop doesn't deal with tags that have been removed from d->exifMetadata, they are not deleted from exif.

Andreas

Thanks Andreas, Code from libkexiv2 fixed...

Gustavo,

DNGConverter is fixed now in svn. Please test following these instructions :

- checkout libkexiv2 from svn trunk (1.0.0). Compile and install
- checkout kipi-plugins from svn trunk (0.7.0). Compile and install
- try again with you ORF file. Now it' loaded fine under showfoto.

Please confirm. I will backport libkexiv2 fix from trunk to KDE 4.3 and KDE 4.2 branches later. Thanks in advance

SVN commit 1020041 by cgilles:

fix makernotes after DNG creation. This is require to ORF files
This fix require libkexiv2 for KDE 4.4 for this moment. I'm waiting 
confirmation from tester to backport fix to KDE 4.3. and KDE 4.2 branches
CCBUGS: 204437


 M  +7 -9      dngwriter.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1020041

I can confirm the resolution of the original problem :)

Well, i had another problem with dngconverter creating 2 temporal files and not removing the dummy one and not renaming the hidden ".kipi********" file. But this is maybe related to my mix of ubuntu packages and manual compiled ones in my test environment.

Tested with a bunch of raw files and everything works as expected.
Despite being or not "the best solution" (because of removing exif data), it should serve to close this bug.