Bug 204437 - DNGConverter turns olympus ORF unusable
Summary: DNGConverter turns olympus ORF unusable
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Plugin-Bqm-DngConverter (show other bugs)
Version: unspecified
Platform: Ubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-08-19 19:14 UTC by Gustavo Claramunt
Modified: 2018-03-23 11:26 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 0.7.0


Attachments
this is the jpeg preview image extracted from DNG and displayed as well when Exiv2 metadata backpup is commented. (687.28 KB, image/png)
2009-08-20 12:12 UTC, caulier.gilles
Details
I found the problem : it's makernote backup !!! (791.94 KB, image/png)
2009-08-20 12:24 UTC, caulier.gilles
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Gustavo Claramunt 2009-08-19 19:14:22 UTC
Version:            (using KDE 4.3.0)
OS:                Linux
Installed from:    Ubuntu Packages

Hi,
After testing DNGConverter with ubuntu 9.04 (kipi 0.20), ubuntu 9.10 (kipi
0.50) and windows digikam (kipi 0.20 only) i'm completely lost about this...

My olympus *.orf files (e-420) get converted flawlessly retaining all my
valuable exif metadata including makernotes and all...

But there's strange behaviour sometimes when opening those converted orf to
dng with almost every application (tested digikam image editor, rawtherapee,
lightzone...) the developed image is full black with some lighter areas
containint green colours.

I've already tested same file conversion with adobe dng converter and
doesn't show this problem (although adobe converter doesn't retain my
makernotes with focusing data, etc).

As i said, not happening with every *.orf file, but i cannot guess why. I've
searched all over the net last 2 months but... nothing to clarify me the
problem.

Testing files uploaded with original orf, converted dng with latest "adobe converter" and converted dng with kipi plugins 0.50 here:
http://www.flasherodepalo.com/digikam/
Thanks,
Goose.
Comment 1 caulier.gilles 2009-08-19 19:47:47 UTC
Which libkdcraw and libraw version you use ? Go to Help/Components Info for details

Gilles Caulier
Comment 2 Gustavo Claramunt 2009-08-19 22:22:02 UTC
Pasted info window:

digiKam version 1.0.0-beta3
Eliminación de mosaico parallelizada: Sí
Exiv2 puede guardarse en JP2: Sí
Exiv2 puede guardarse en Jpeg: Sí
Exiv2 puede guardarse en PNG: Sí
Exiv2 puede guardarse en TIFF: Sí
Exiv2 se puede escribir a Pgf: No
Exiv2 soporta  metadatos XMP: Sí
LibCImg: 130
LibExiv2: 0.18.1
LibJPEG: 62
LibJasper: 1.900.1
LibKDE: 4.3.00 (KDE 4.3.0)
LibKExiv2: 0.6.0
LibKdcraw: 0.5.0
LibLCMS: 118
LibPGF: 6.09.24
LibPNG: 1.2.37
LibQt: 4.5.2
LibRaw: 0.7.2
LibTIFF: LIBTIFF, Version 3.8.2 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Widget Marble: 0.8
LibGphoto2: 2.4.6
LibKipi: 0.4.0

I haven't found any similar previous bug report about this, but i notice this extrange behavior with kipi-plugins 0.20 (ubuntu jaunty packages).

Goose.
Comment 3 caulier.gilles 2009-08-19 22:26:26 UTC
Ok, try to update to Exiv2 0.18.2. libkexiv2/ kipi-plugins, and digiKam need to be recompiled.

For libraw, to have a last version 0.8.0, you must use libkdcraw from trunk.

Anyway, use kipi-plugins 0.5.0, not 0.2.0

Gilles Caulier
Comment 4 caulier.gilles 2009-08-20 11:13:46 UTC
Ok, i can reproduce the problem with last libraw too.

./raw2dng.shell olympus.orf 
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Loading RAW data from  "olympus.orf"
<unknown program name>(11093)/ KDcrawIface::KDcrawPriv::progressCallback: LibRaw progress:  Reading metadata  pass  0  of  2
<unknown program name>(11093)/ KDcrawIface::KDcrawPriv::progressCallback: LibRaw progress:  Reading metadata  pass  1  of  2
<unknown program name>(11093)/ KDcrawIface::KDcrawPriv::progressCallback: LibRaw progress:  Reading RAW data  pass  0  of  2
<unknown program name>(11093)/ KDcrawIface::KDcrawPriv::progressCallback: LibRaw progress:  Reading RAW data  pass  1  of  2
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Raw data loaded:                                    
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Data Size:      20832000  bytes                            
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Date:           "2009-07-26T20:16:46"                      
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Make:           "OLYMPUS"                                  
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Model:          "E-420"                                    
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Size:           3720 x 2800                                
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Orientation:    0                                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Top margin:     0                                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Left margin:    0                                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Filter:         "RGGBRGGBRGGBRGGB"                         
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Colors:         3                                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- Black:          68                                         
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- White:          4055                                       
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: --- CAM->XYZ:                                                  
<unknown program name>(11093)/ DNGIface::DNGWriter::convert:                     "0.8746  -0.2425  -0.1095"                 
<unknown program name>(11093)/ DNGIface::DNGWriter::convert:                     "-0.7594  1.5612  0.2073"                  
<unknown program name>(11093)/ DNGIface::DNGWriter::convert:                     "-0.1780  0.2309  0.7416"                  
<unknown program name>(11093)/ DNGIface::DNGWriter::convert:                     "0.0000  0.0000  0.0000"                   
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Formating RAW data to memory                        
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG memory allocation and initialization            
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG IFD structure creation                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG Negative structure creation                     
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Updating metadata to DNG Negative                   
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Backup Makernote ( 651928  bytes)                   
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Build DNG Negative                                  
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG preview image creation                          
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG thumbnail creation                              
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Creating DNG file  "olympus.dng"                    
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: Backup meta-data using Exiv2                        
<unknown program name>(11093)/ KExiv2Iface::KExiv2::save: File Extension:  "dng"  is supported for writing mode             
<unknown program name>(11093)/ DNGIface::DNGWriter::convert: DNGWriter: DNG conversion complete...
Comment 5 caulier.gilles 2009-08-20 11:16:41 UTC
Alex,

ORF file come from E-420 model. What do you think about. 

picture converted is green. sound like an inverted color chanel somewhere.

"RGGBRGGBRGGBRGGB" is right for you as color filter ?

Gilles Caulier
Comment 6 Gustavo Claramunt 2009-08-20 11:22:17 UTC
I'm preparing my build environment with virtualbox, so i can test everything as i get info.
By the way, what annoys me is not having this issue with all my orf files, but only some of them, and i cannot isolate the problem.

Should i upload more *.orf files with and without this issue?

Thanks,
Goose.
Comment 7 Alex Tutubalin 2009-08-20 11:24:39 UTC
dcraw_emu converts this file without problems, so pixel order is right.
Comment 8 Gustavo Claramunt 2009-08-20 11:48:04 UTC
I've tested at work (windows xp workstation):
-opening DNG file with Camera Raw (adobe photoshop cs3) results in file working properly... no black/green...
-Opening DNG file with rawtherapee 2.4 fails exactly like in my ubuntu installation.
-Opening DNG with ufraw windows v0.15, wrong too...

Original orf works as expected with every software, but not the resulting dng converted with kipi.
Comment 9 Alex Tutubalin 2009-08-20 12:04:23 UTC
In general, DNG file contains two sets of metadata:
 - EXIF (MakerNotes) block copied from original RAW
 - DNG-specific metadata (color matrices, black level and so on)

I guess,  Adobe uses DNG-specific block, but other tools may use EXIF block on already processed data.
Comment 10 caulier.gilles 2009-08-20 12:10:14 UTC
Some progress :

If i comment this line :

http://lxr.kde.org/source/extragear/graphics/kipi-plugins/dngconverter/dngwriter/dngwriter.cpp#816

Now jpeg preview image embeded in DNG is fine. RAW data color still wrong.

If i uncomment this line, JPEG preview has the same side effect than RAW color. This is really strange because JPEG preview is computed with DNG sdk using common RAW image data extracted from ORF file, and stored in DNG in dedicated format.

So, there is problem with Exiv2 here, but not only... Andreas ?

Gilles Caulier
Comment 11 caulier.gilles 2009-08-20 12:12:33 UTC
Created attachment 36294 [details]
this is the jpeg preview image extracted from DNG and displayed as well when Exiv2 metadata backpup is commented.
Comment 12 caulier.gilles 2009-08-20 12:14:00 UTC
I remember the singularity of ORF makernotes : some pointers are link to Exif information, outside makernotes area. I will try to comment makernotes backup in DNG converter

Gilles Caulier
Comment 13 caulier.gilles 2009-08-20 12:24:04 UTC
Created attachment 36295 [details]
I found the problem : it's makernote backup !!!

Andreas, what's your tip here ? I'm using Exiv2 0.18.1 on this computer. Exiv2 from trunk can help me better with ORF makernotes ?

Gilles Caulier
Comment 15 caulier.gilles 2009-08-20 12:30:42 UTC
Andreas,

Important point : If i lets makernotes backup code commented and if i uncomments Exiv2 metadata backup code, DNG file generated is small and corrupted.

There is definitively something wrong with Exiv2 here i think. Your viewpoint ?

Gilles Caulier
Comment 16 Andreas Huggel 2009-08-20 13:01:58 UTC
Gilles,

Re comment #13: I don't think 0.18.2 or trunk is different in this area. But using trunk for the analysis is better anyway.

Re comment #15: Is that the same issue as discussed here: http://dev.exiv2.org/boards/3/topics/show/201 i.e., at least the "necessary" image tags need to be written anyway.

Check if this has something to do with the whitebalance: load the "corrupted"
image in ufraw and change the whitebalance setting from "automatic" (which means it reads it from the metadata) to something else. Does it make a difference?

Andreas
Comment 17 Gustavo Claramunt 2009-08-20 13:16:24 UTC
I've tested with various tools, playing with a lot of settings (including white balance, exposure, etc) and result is the same. Is not like the image got "green tint"... better take a look the image here:

http://www.flasherodepalo.com/digikam/kipi_050_dng.jpg

Looks like when you overexpose or underexpose a photo, and there's no way to recover image data with exposure controls... (sorry if i cannot write better explanation, my english is really limited).
Comment 18 Gustavo Claramunt 2009-08-21 12:34:15 UTC
Ok, finished testing in virtualbox with latest svn, resulting the same error.

Built within virtualized ubuntu karmic 9.10.
Pasted components info from showfoto:

showFoto version 1.0.0-beta4 (rev.: 1013570)
Exiv2 can write to Jp2: Yes
Exiv2 can write to Jpeg: Yes
Exiv2 can write to Pgf: No
Exiv2 can write to Png: Yes
Exiv2 can write to Tiff: Yes
Exiv2 supports XMP metadata: Yes
LibCImg: 130
LibExiv2: 0.18.2
LibJPEG: 62
LibJasper: 1.900.1
LibKDE: 4.3.00 (KDE 4.3.0)
LibKExiv2: 1.0.0
LibKdcraw: 1.0.0
LibLCMS: 118
LibPGF: 6.09.24
LibPNG: 1.2.37
LibQt: 4.5.2
LibRaw: 0.8.0-Beta5
LibTIFF: LIBTIFF, Version 3.8.2 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Parallelized demosaicing: Yes


If there is anything i can try to help, test, build....
Comment 19 caulier.gilles 2009-08-21 12:48:35 UTC
Gustavo,

There is nothing to test anymore with current code. bug is with markernote backup. just look my previous comments. You can comment lines in source code as described and test in your computer. It's simple to do...

Gilles Caulier
Comment 20 Gustavo Claramunt 2009-08-21 13:51:55 UTC
Sure it is, but the point of bug tracking is being able to find the guilty!
Just kidding ;)
BTW i don't think is dngconverter fault... i've make these 2 tests:

1-Using dcraw 8.86 with the converted dng, dcraw claims about not getting correct white balance BUT resulting TIFF is working OK (not white balanced but not Black/Green).

2.-With Adobe DNG Converter the resulting .dng is working right(as i said previously) BUT: if then I use exiftool to write the EXIF from the ORF to the DNG... the modified DNG is Black/Green again!!!

So... i guess is not dngconverter "feature"... just DNG feature when you inject it some type of exif metadata?

Where should it be informed/bugtracked/cryed?

Thanks again,
Goose.
Comment 21 caulier.gilles 2009-08-21 14:06:38 UTC
Yes, the problem is in Adobe DNG spec about how to handle markernotes. Here Andreas or Alex can explain better than mine.

This problem is also specific to ORF makernote format which link data outside makernotes area, as Exif for ex. 

I remember athread with Andreas about this subject about to make a wrapper in Exiv2 to fix the problem. I think nothing is done currently in this way.

Gilles Caulier
Comment 22 Gustavo Claramunt 2009-08-21 14:54:02 UTC
The way i see it: DNG Spec is not made to have all the exif data embeded "the same way" as original raw files... and maybe this is the reason they save some (don't know how much of it) exif infos into XMP space...

Should kipi-dngconverter acts same way Adobe converter does, and store all exif info found in original raw (same info displayed in digikam) into XMP space?
Comment 23 Gustavo Claramunt 2009-08-24 11:57:17 UTC
Hi, i found the guilty!!!
Try the following to see if anyone can reproduce it:
-Convert original ORF file with Kipi-dngconverter
-Assure result is bad (black/green picture in every raw developer)
-Delete "Exif.OlympusIp.BlackLevel" tag with exiv2
(exiv2 -M"del Exif.OlympusIp.BlackLevel" file_kipi.dng)
-Test resulting file with digikam, lightzone, rawtherapee: no black/green thing.
-Make an export from both orf and modified dng to png and compared both... looks identical.

So... maybe a bug within exiv2 interpretation of Exif.OlympusIp.BlackLevel?
Comment 24 Andreas Huggel 2009-08-24 17:03:50 UTC
Good work, Gustavo! But don't jump to a conclusion just yet.

OlympusIp is a Sub-IFD, that means there is nothing in there for Exiv2 to interpret. Check with another tool to see if the values differ.

As Alex already said in comment #9, is the Exif data in the makernote still valid for the converted image data? In particular the black level value?

Andreas
Comment 25 Andreas Huggel 2009-08-24 17:09:54 UTC
In other words: which Exif data values does libraw/dcraw use and do they need to be converted too when the image data is converted?

-ahu.
Comment 26 Alex Tutubalin 2009-08-24 17:42:25 UTC
There is serveral possible cases for DNG converter:

  - exact copy RAW data from RAW file to DNG file without any processing. In this case, camera metadata is better than DNG ones, so camera Exif should be preserved.
  - some RAW processing (black level subtraction, raw tone curves and so on). In this case black level should be set in DNG metadata, but corresponding data fields in camera metadata should be zeroed to avoid mis-interpretation for already processed data.
  - and, of course, demosaicing of RAW into full RGB bitmap.

I have not take look of Gilles' DNG converter sources (and will be unable to do it in nearest 2-3 days), so cannot say anything more definitive about this case yet.
Comment 27 Gustavo Claramunt 2009-08-25 00:51:09 UTC
@Andreas, i'm unable to catch you, sorry (maybe my english limitations :P ). I don't know if libraw/dcraw/other raw developers use that sub-ifd, so don't know why is causing so much trouble.
I'm programmer (well, web programmer though) and casual photographer, so i really don't know so much about exif "internals"... i only did some tests...
The only thing i know: removing Exif.OlympusIp.BlackLevel means that tag no longer appears (tested many exif/metadata "visualizers") but raw developers works as supposed to do...
 

@Alex, don't bother so much with Gilles DNGConverter... i did same tests using Adobe DNG Converter, injecting original orf exif data to it with exiv2 and the problem persisted. Then removed the infamous Exif.OlympusIp.BlackLevel and gone right as with dngconverter.

Same tests using exiftool instead exiv2, same results.

So maybe is missinterpretation from raw development softwares confused finding olympus specific exif into a file which should not have it?

I'd love using dng cause full xmp write support from Digikam (and so interoperability between other softwares) but maybe i'd better stop dreaming of that?
Comment 28 Andreas Huggel 2009-08-25 05:14:34 UTC
So, if the image data is processed for the DNG, it may be necessary to also adjust the (Exif) metadata, rather than just copy it.

The tests that Gustavo did with different image converters and different metadata tools indicate that libraw and exiv2 by themselves are ok, just not synchronized in this case.

-ahu.
Comment 29 Alex Tutubalin 2009-08-25 07:34:47 UTC
@Andreas:
In general, it is not possible to adjust metadata in all cases.

AFAIK, dcraw (and all derived programs/libraries) takes precedence of camera metadata over DNG metadata in some cases. I've seen this with canon files converted to DNG with adobe converter.
LibRaw do it too to preserve dcraw compatibility (over correctness).

There is two possible ways to deal with it
 1) It is possible to turn off all RAW preprocessing in LibRaw in Gilles' DNG converter, so raw data in DNG will be same as in source RAW
 2) It is possible to change LibRaw to prefer DNG metadata over camera ones, so all LibRaw-users (digiKam and so) will be happy.

I will take closer look into the problem on weekend. Sorry, no spare time on work-week.
Comment 30 caulier.gilles 2009-08-25 07:42:56 UTC
Look there : 

http://lxr.kde.org/source/extragear/graphics/kipi-plugins/dngconverter/dngwriter/dngwriter.cpp#398

This code use DNG sdk. If i do not set this value, DNG file is not suitable (black hole). Perhaps it's a special case with ORF and DNG tags is uncompatible with Olympus makernote where a duplicate tag is set ?

Gilles
Comment 31 Alex Tutubalin 2009-08-25 07:55:25 UTC
Gilles, 

the problem is not with DNG black level, it is correct (as we can see if we delete Olympus black level data from DNG). The problem is with Olympus black level data if this data are present in DNG file.
Comment 32 Andreas Huggel 2009-08-25 08:04:51 UTC
There is a third possibility, afaics:

3) Delete Exif.OlympusIp.BlackLevel before writing the metadata to the DNG image

(Or set the value to 0, if that makes more sense. And apply the same treatment to other tags as needed). These values are apparently related to the raw image data and since that has been processed, they are no longer valid. The Exif data which makes sense for the converted image remains intact.

Conceptually it is similar to what we do when we read metadata from a TIFF file and write it to a JPEG - some tags are filtered.

Out of curiosity, what's the rationale for processing the image data only to some extent, as it is apparently done now, if it still requires specialized RAW converters (dcraw) to display the image later? The resulting DNG seems to still contain very much proprietary image data in this case. Why not convert the image data to an open format? Or am I missing something? 

Andreas
Comment 33 Alex Tutubalin 2009-08-25 08:22:55 UTC
Andreas,

I guess, that Olympus is not the only problematic camera. There is some logical problems with other cameras too. So, deleting Oly black level data is only partial solution. It is better to find more generic one.
Comment 34 Gustavo Claramunt 2009-08-25 10:30:28 UTC
Maybe this make some sense...
After taking a  look to Adobe DNG Spec (http://www.adobe.com/products/dng/pdfs/dng_spec_1_3_0_0.pdf, page 22) I can see it uses BlackLevel 

(and other 3 related tags) in a different way than usual RAW files ... so the problem is a little bigger.

Maybe dcraw/libraw/etc use algorithms to demosaic dng files based in adobe specification, but in this case (don't know others) the value from 

BlackLevel is not suited to those calculations cause is a "different" blacklevel (in spec pdf note a "mapping raw values to linear reference 

values") cause is taken directly from camera's raw.


Possible "global" solutions as far as i'm an ignorant :)?

1.-DNG Should not have exif data from original raw files than interfere with the Adobe specs, so every tag in exif metadata from raw files 

should be converted u ommited if it make unconsistency with oficial spec (as noted by Alex in comment #31).

2.-Every raw developer (dcraw/libraw/etc) should adapt to not use those "duplicated" exif data when dealing with DNG files.

Drawbacks:
-In the first case, dngconverter and/or exiv2 should adapt/filter exif values according to official dng spec.
-In the second case... too many software develovers should be concerned about this issue... and maybe it isn't the correct methos cause we're breaking the official dng specs... but who knows.

Notes:
-I've tested zeroing instead deleting the problematic "Olympus BlackLevel" tag, resulting in "NO black/green" picture but not as good colour 

as when deleting the key (maybe related to Alex comment #33?)

-@Andreas, there are lot of discussion with DNG usage, take a look to Barry Pearson "Defend DNG to the max" page at http://barrypearson.co.uk/articles/dng/index.htm. In many cases, photographers wants to retain their digital negatives as "intact" as they can. In my case also want to be able to tag these negatives, fill xmp metadata, comments, ratings, etc, AND store those valuable informations within the file itself so i can see/use it in other softwares/workstations. With ORF i can only set ratings/comments/tags, no way to modify anything else AND the rating/comments/tags are stored into digikam database... so i can't see/use it outside Digikam. That's my personal point about using DNG, but there ar more out there :)

P.S.-One IMPORTANT thing i've noted (and posted at the begining of the bug file): I have "tons" of .orf files which converted to dng WITHOUT the problem mentioned here... but there are a bunch with the problem. Why sometimes right, sometimes wrong? Well, i've guessed why:
When my camera takes a picture and the resulting ORF has a BlackLevel value of (0,0,0,0), the converted DNG (with both DNG BlackLevel tag and Olympus BlackLevel tag having the same value) works allright.
But the problematic raw files has a BlackLevel of (68,68,68,68) (in both BlackLevel tags, dng and olympus)... Obviously, if i set both to (0,0,0,0) the image displays without the nasty black/green, but the color balance/white balance/black balance is not accurate.
Comment 35 Alex Tutubalin 2009-08-25 11:19:30 UTC
Gustavo, the LibRaw's black level (used in Gilles' dng converter) is really right value for DNG specs. LibRaw linearizes RAW (where appropriate, e.g. Sony RAWs or PhaseOne ones) on the reading stage, so black level subtraction is the logical next step in terms of DNG processing model.

The only problem is camera metadata precedence over DNG metadata. This is 'feature' (or logical bug?) of dcraw and all dcraw-derived tools, including LibRaw. I know about this feature of LibRaw (I've see some problems with DNGs converted from Canon RAWs) but have choose to keep dcraw compatibility instead of fixing it.

I'll take closer look into this problem on weekend. May be, there is easy way to fix it, at least for particular cameras (e.g. Oly)
Comment 36 Gustavo Claramunt 2009-08-25 11:42:38 UTC
Alex, i know libraw could be patched to fix it, i'me very confident about you all developers ;)... but that only fixes software using libraw... so dcraw based programs won't be aware of the problem. (i know bug could be filed against them).

And the other point mentioned in my previous comment... why is this "precedence" affecting only photographs with blacklevel different to (0,0,0,0)? Why that "precedence" affects the image with blacklevel of (68,68,68,68) if BOTH the conflictive tags has the SAME value(68,68,68,68)?

There is something i cannot get into...

Once again, thanks for all your effort.
Comment 37 caulier.gilles 2009-09-03 13:58:15 UTC
Andreas,

To simulate #23, I have just set this code after DNG file creation :

// See B.K.O #204437 : remove Olympus black level makernote tag.
meta.load(dngFilePath);
meta.setWriteRawFiles(true);
bool b = meta.removeExifTag("Exif.OlympusIp.BlackLevel", false);
kDebug() << "Exif.OlympusIp.BlackLevel found = " << b;
meta.applyChanges();

... tag is found but never removed. Tag in makernote still there.

Code to remove Exif tag is similar that actions.cpp code from Exiv2 :

http://lxr.kde.org/source/KDE/kdegraphics/libs/libkexiv2/libkexiv2/kexiv2exif.cpp#303

Note : line #318 is removed here.

Gilles Caulier
Comment 38 caulier.gilles 2009-09-04 13:47:25 UTC
Andreas,

I just created a test program to test makernote tag erase fonction. As expected, it doesn't work, and i don't know why.

Program is there :

http://websvn.kde.org/trunk/KDE/kdegraphics/libs/libkexiv2/test/erasetag.cpp?view=markup

Executing test program give results below :

./erasetag.shell olympus.dng
main: Exif.OlympusIp.BlackLevel found =  true
main: Exif.OlympusIp.BlackLevel removed =  true
KExiv2Iface::KExiv2::save: File Extension:  "dng"  is supported for writing mode
KExiv2Iface::KExiv2::save: Exif is supported for writing mode
KExiv2Iface::KExiv2::save: Exif is updated

You can see that Olympus makernote tag is removed in memory, but after when exiv2::writeMetadata() is called in libkexiv2::save() method

http://lxr.kde.org/source/KDE/kdegraphics/libs/libkexiv2/libkexiv2/kexiv2.cpp#283

...DNG file still untouched (same size, same timestamp)

Note: if i set debug statement in this method, i can see that Exif data is set to Exiv2::Image instance and exiv2::writeMetadata() is called properly.

Testing with Exiv2 0.18.2 command line tool using 

exiv2 -M"del Exif.OlympusIp.BlackLevel" olympus.dng

...work fine. Where is the problem ?

Gilles Caulier
Comment 39 Andreas Huggel 2009-09-04 20:04:50 UTC
Gilles,

Looking at the code, I can't see the problem right now (it's 2am here, I'm obviously overlooking something). Suggest to add debug output to Exiv2::Image::writeMetadata to see if it's called in the first place and if the tag is there or not. Or step through KExiv2::save.

Andreas
Comment 40 Andreas Huggel 2009-09-04 20:57:56 UTC
Gilles,

The problem is here:

http://lxr.kde.org/source/KDE/kdegraphics/libs/libkexiv2/libkexiv2/kexiv2.cpp#368

The loop copies all tags from d->exifMetadata except for those that are in the untouchedTags list. But this loop doesn't deal with tags that have been removed from d->exifMetadata, they are not deleted from exif.

Andreas
Comment 41 caulier.gilles 2009-09-05 09:55:14 UTC
Thanks Andreas, Code from libkexiv2 fixed...

Gustavo,

DNGConverter is fixed now in svn. Please test following these instructions :

- checkout libkexiv2 from svn trunk (1.0.0). Compile and install
- checkout kipi-plugins from svn trunk (0.7.0). Compile and install
- try again with you ORF file. Now it' loaded fine under showfoto.

Please confirm. I will backport libkexiv2 fix from trunk to KDE 4.3 and KDE 4.2 branches later. Thanks in advance
Comment 42 caulier.gilles 2009-09-05 09:58:09 UTC
SVN commit 1020041 by cgilles:

fix makernotes after DNG creation. This is require to ORF files
This fix require libkexiv2 for KDE 4.4 for this moment. I'm waiting 
confirmation from tester to backport fix to KDE 4.3. and KDE 4.2 branches
CCBUGS: 204437


 M  +7 -9      dngwriter.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1020041
Comment 43 Gustavo Claramunt 2009-09-07 12:43:51 UTC
I can confirm the resolution of the original problem :)

Well, i had another problem with dngconverter creating 2 temporal files and not removing the dummy one and not renaming the hidden ".kipi********" file. But this is maybe related to my mix of ubuntu packages and manual compiled ones in my test environment.

Tested with a bunch of raw files and everything works as expected.
Despite being or not "the best solution" (because of removing exif data), it should serve to close this bug.