Bug 431689

Summary: Nokia 9 PureView DNG image demosaicing preview corruption
Product: [Applications] digikam Reporter: doommaster29
Component: Preview-RAWAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: normal CC: caulier.gilles, lexa, mehmetgelisin, metzpinguin
Priority: NOR    
Version: 7.3.0   
Target Milestone: ---   
Platform: Microsoft Windows   
OS: Microsoft Windows   
Latest Commit: https://invent.kde.org/graphics/digikam/commit/b7f5446bdd59e50f1c44bea13182e1179fc64bdb Version Fixed In: 7.3.0
Sentry Crash Report:
Attachments: Shows corruption
RAW settings
Metadata
Metada 2

Description doommaster29 2021-01-16 13:47:14 UTC 
Created attachment 134930 [details]
Shows corruption

SUMMARY


STEPS TO REPRODUCE
1. Install digiKam on Windows 10.
2. When you select a RAW photo, it randomly gets horribly corrupted (taken with Nokia 9 PureView, at least)
3. Interestingly, issue propagates, and also happens in other programs as RawTherapee (5.8) BUT NOT TO LIGHTROOM. At least, it hasn't happened yet.
4. Also of note, the medium RAW image that serves for the miniature, doesn't seem to care either, even though is supposed to be the RAW itself, not a preview. 
4. Can't be traced what triggers this, so I don't know if its camera (Nokia 9 PureView) or software related. But as said Lightroom doesn't show this behaviour. 

OBSERVED RESULT
Image corrupted under both digiKam or RawTherappe, but not Lightroom, although processing seems different.

EXPECTED RESULT
Not corrupted Image.

SOFTWARE/OS VERSIONS
Windows: 10
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

ADDITIONAL INFORMATION
Comment 1 Maik Qualmann 2021-01-16 14:07:05 UTC 
I suspect based on the screenshot that they use embedded previews for display. You can activate in the digiKam settings that half a RAW preview is used for RAW files. Please test whether the image is displayed correctly. Do you write metadata in DNG files (is deactivated by default in digiKam)? If the images haven't been edited yet and RawTherappe shows the same symptom, I suspect the problem is related to Nokia 9 PureView. Or with the program that created the DNG files. Please upload a problematic DNG file, if not publicly, to my email address.

Maik
Comment 2 doommaster29 2021-01-16 14:24:33 UTC 
Created attachment 134931 [details]
RAW settings
Comment 3 doommaster29 2021-01-16 14:24:57 UTC 
Created attachment 134932 [details]
Metadata
Comment 4 doommaster29 2021-01-16 14:25:13 UTC 
Created attachment 134933 [details]
Metada 2
Comment 5 doommaster29 2021-01-16 14:28:49 UTC 
Thank you Maik.
Yes, I meant half RAW, I did a bad translation 'medium RAW'.
I checked no box for metadata. I see only the 'update mod. date when file is modified' and writing metadata when rotating image.
Comment 6 caulier.gilles 2021-01-16 15:03:59 UTC 
Which digiKam version do you use exactly ?

Gilles Caulier
Comment 7 doommaster29 2021-01-16 15:20:57 UTC 
7.1
Comment 8 caulier.gilles 2021-01-16 15:28:02 UTC 
Please try with 7.2.0 release candidate available here :

https://files.kde.org/digikam/

... where libraw have been updated to last stable version with plenty of fixes.

Gilles Caulier
Comment 9 doommaster29 2021-01-16 15:54:01 UTC 
Thank you Cauiler.
So I did, to no avail. But perhaps because images were already "corrupted". I couldn't replicate the effect. But I don't know how its triggered anyway.
Comment 10 caulier.gilles 2021-01-16 16:20:20 UTC 
I don't understand. Do you mean that 7.2.0rc fix your problem (aka preview is now displayed properly) ?

Gilles Caulier
Comment 11 doommaster29 2021-01-16 16:29:05 UTC 
No. The image previously corrupted stays the same.
Comment 12 caulier.gilles 2021-01-16 16:52:28 UTC 
How did you generate the DNg files ? directly from the camera ? Form the digiKam DNG converter ? From another RAW processing application ?

This DNG image can be shared through an internet file storage for investigations ?

Gilles Caulier
Comment 13 doommaster29 2021-01-16 17:22:14 UTC 
I do nothing to the DNGs, but I've detected some .pp3 files, so some modification could take place. But not in all cases. Delete them did nothing to solve the issue.
I include a link to some example images for your peruse.

https://1drv.ms/u/s!AuaRuWZfqhhGluIdUEpRQv7nLnoucw?e=PPDhTC

I hope you are now able to reproduce whatever is causing this.
I really thank you for your help.
Comment 14 Maik Qualmann 2021-01-16 20:11:35 UTC 
First I go from "critical" to "normal". DigiKam did not create or change the DNG. I can also reproduce the problem with the URAW program. Darktable displays it correctly.


Maik
Comment 15 doommaster29 2021-01-16 21:46:21 UTC 
Thank you Maik,
I've been able to reproduce it. At least a variant. Happens mostly to black and white images. And this could give a clue, I think, because this camera uses dedicated monochrome sensor for this: no color filter on-top. So no normal demosaicing should take place. So I guess, I'm not an expert.
I upload a new image I just took and opened with digiKam right away, with the same result. And jpeg as well.
Anyway, any idea of what could be done to use them with digiKam again?  
Perhaps this helps with this kind of hardware. 
Thank you!

https://1drv.ms/f/s!AuaRuWZfqhhGluIdUEpRQv7nLnoucw
Comment 16 caulier.gilles 2021-01-30 08:35:53 UTC 
Git commit efc5d2e6d84e377e3b5a994f97a10e0ce586e61a by Gilles Caulier.
Committed on 30/01/2021 at 08:29.
Pushed by cgilles into branch 'master'.

Update internal Libraw engine to last 0.21.0 Snapshot 202101 with many improvement about DNG support.
For more detail see the official announcement: https://www.libraw.org/news/libraw-202101-snapshot
Related: bug 418645

M  +19   -1    NEWS
M  +1    -1    core/libs/rawengine/libraw/COPYRIGHT
M  +154  -25   core/libs/rawengine/libraw/Changelog.txt
M  +1    -1    core/libs/rawengine/libraw/README.md
M  +1    -1    core/libs/rawengine/libraw/internal/dcraw_common.cpp
M  +9    -5    core/libs/rawengine/libraw/internal/dcraw_defs.h
M  +1    -1    core/libs/rawengine/libraw/internal/dcraw_fileio.cpp
M  +1    -1    core/libs/rawengine/libraw/internal/dcraw_fileio_defs.h
M  +2    -1    core/libs/rawengine/libraw/internal/defines.h
M  +1    -1    core/libs/rawengine/libraw/internal/dmp_include.h
M  +9    -3    core/libs/rawengine/libraw/internal/libraw_cameraids.h
M  +1    -1    core/libs/rawengine/libraw/internal/libraw_cxx_defs.h
M  +276  -269  core/libs/rawengine/libraw/internal/libraw_internal_funcs.h
M  +150  -149  core/libs/rawengine/libraw/internal/var_defines.h
M  +1    -1    core/libs/rawengine/libraw/internal/x3f_tools.h
M  +46   -8    core/libs/rawengine/libraw/libraw/libraw.h
M  +1    -1    core/libs/rawengine/libraw/libraw/libraw_alloc.h
M  +111  -51   core/libs/rawengine/libraw/libraw/libraw_const.h
M  +99   -2    core/libs/rawengine/libraw/libraw/libraw_datastream.h
M  +4    -4    core/libs/rawengine/libraw/libraw/libraw_internal.h
M  +185  -92   core/libs/rawengine/libraw/libraw/libraw_types.h
M  +5    -5    core/libs/rawengine/libraw/libraw/libraw_version.h
M  +1    -1    core/libs/rawengine/libraw/samples/4channels.cpp
M  +15   -18   core/libs/rawengine/libraw/samples/dcraw_emu.cpp
M  +1    -1    core/libs/rawengine/libraw/samples/dcraw_half.c
M  +1    -1    core/libs/rawengine/libraw/samples/half_mt.c
M  +1    -1    core/libs/rawengine/libraw/samples/half_mt_win32.c
M  +1    -1    core/libs/rawengine/libraw/samples/mem_image_sample.cpp
M  +1    -1    core/libs/rawengine/libraw/samples/multirender_test.cpp
M  +1    -1    core/libs/rawengine/libraw/samples/openbayer_sample.cpp
M  +2    -2    core/libs/rawengine/libraw/samples/postprocessing_benchmark.cpp
M  +528  -1481 core/libs/rawengine/libraw/samples/raw-identify.cpp
M  +1    -1    core/libs/rawengine/libraw/samples/rawtextdump.cpp
M  +2    -2    core/libs/rawengine/libraw/samples/simple_dcraw.cpp
M  +1    -1    core/libs/rawengine/libraw/samples/unprocessed_raw.cpp
M  +1    -1    core/libs/rawengine/libraw/src/decoders/canon_600.cpp
M  +674  -377  core/libs/rawengine/libraw/src/decoders/crx.cpp
M  +25   -41   core/libs/rawengine/libraw/src/decoders/decoders_dcraw.cpp
M  +8    -9    core/libs/rawengine/libraw/src/decoders/decoders_libraw.cpp
M  +8    -12   core/libs/rawengine/libraw/src/decoders/decoders_libraw_dcrdefs.cpp
M  +1    -1    core/libs/rawengine/libraw/src/decoders/dng.cpp
M  +234  -98   core/libs/rawengine/libraw/src/decoders/fp_dng.cpp
M  +375  -296  core/libs/rawengine/libraw/src/decoders/fuji_compressed.cpp
M  +6    -17   core/libs/rawengine/libraw/src/decoders/generic.cpp
M  +31   -58   core/libs/rawengine/libraw/src/decoders/kodak_decoders.cpp
M  +1    -1    core/libs/rawengine/libraw/src/decoders/load_mfbacks.cpp
M  +1    -1    core/libs/rawengine/libraw/src/decoders/smal.cpp
M  +18   -14   core/libs/rawengine/libraw/src/decoders/unpack.cpp
M  +21   -16   core/libs/rawengine/libraw/src/decoders/unpack_thumb.cpp
M  +0    -6    core/libs/rawengine/libraw/src/demosaic/aahd_demosaic.cpp
M  +1    -1    core/libs/rawengine/libraw/src/demosaic/ahd_demosaic.cpp
M  +1    -1    core/libs/rawengine/libraw/src/demosaic/misc_demosaic.cpp
M  +1    -1    core/libs/rawengine/libraw/src/demosaic/xtrans_demosaic.cpp
M  +21   -16   core/libs/rawengine/libraw/src/integration/dngsdk_glue.cpp
M  +1    -1    core/libs/rawengine/libraw/src/integration/rawspeed_glue.cpp
M  +7    -2    core/libs/rawengine/libraw/src/libraw_c_api.cpp
M  +1    -1    core/libs/rawengine/libraw/src/libraw_cxx.cpp
M  +342  -4    core/libs/rawengine/libraw/src/libraw_datastream.cpp
M  +1    -1    core/libs/rawengine/libraw/src/metadata/adobepano.cpp
M  +155  -125  core/libs/rawengine/libraw/src/metadata/canon.cpp
M  +5    -6    core/libs/rawengine/libraw/src/metadata/ciff.cpp
M  +45   -2    core/libs/rawengine/libraw/src/metadata/cr3_parser.cpp
M  +1    -1    core/libs/rawengine/libraw/src/metadata/epson.cpp
M  +1    -1    core/libs/rawengine/libraw/src/metadata/exif_gps.cpp
M  +434  -381  core/libs/rawengine/libraw/src/metadata/fuji.cpp
M  +82   -35   core/libs/rawengine/libraw/src/metadata/hasselblad_model.cpp
M  +250  -180  core/libs/rawengine/libraw/src/metadata/identify.cpp
M  +1    -1    core/libs/rawengine/libraw/src/metadata/identify_tools.cpp
M  +1    -1    core/libs/rawengine/libraw/src/metadata/kodak.cpp
M  +69   -52   core/libs/rawengine/libraw/src/metadata/leica.cpp
M  +71   -186  core/libs/rawengine/libraw/src/metadata/makernotes.cpp
M  +15   -7    core/libs/rawengine/libraw/src/metadata/mediumformat.cpp
M  +31   -33   core/libs/rawengine/libraw/src/metadata/minolta.cpp
M  +5    -3    core/libs/rawengine/libraw/src/metadata/misc_parsers.cpp
M  +55   -22   core/libs/rawengine/libraw/src/metadata/nikon.cpp
M  +67   -19   core/libs/rawengine/libraw/src/metadata/normalize_model.cpp
M  +180  -20   core/libs/rawengine/libraw/src/metadata/olympus.cpp
M  +32   -30   core/libs/rawengine/libraw/src/metadata/p1.cpp
M  +139  -14   core/libs/rawengine/libraw/src/metadata/pentax.cpp
M  +1    -1    core/libs/rawengine/libraw/src/metadata/samsung.cpp
M  +746  -538  core/libs/rawengine/libraw/src/metadata/sony.cpp
M  +113  -60   core/libs/rawengine/libraw/src/metadata/tiff.cpp
M  +1    -1    core/libs/rawengine/libraw/src/postprocessing/aspect_ratio.cpp
M  +7    -2    core/libs/rawengine/libraw/src/postprocessing/dcraw_process.cpp
M  +5    -2    core/libs/rawengine/libraw/src/postprocessing/mem_image.cpp
M  +1    -1    core/libs/rawengine/libraw/src/postprocessing/postprocessing_aux.cpp
M  +1    -1    core/libs/rawengine/libraw/src/postprocessing/postprocessing_ph.cpp
M  +1    -1    core/libs/rawengine/libraw/src/postprocessing/postprocessing_utils.cpp
M  +2    -2    core/libs/rawengine/libraw/src/postprocessing/postprocessing_utils_dcrdefs.cpp
M  +3    -6    core/libs/rawengine/libraw/src/preprocessing/ext_preprocess.cpp
M  +1    -1    core/libs/rawengine/libraw/src/preprocessing/preprocessing_ph.cpp
M  +7    -3    core/libs/rawengine/libraw/src/preprocessing/raw2image.cpp
M  +2    -2    core/libs/rawengine/libraw/src/preprocessing/subtract_black.cpp
M  +64   -17   core/libs/rawengine/libraw/src/tables/cameralist.cpp
M  +1    -1    core/libs/rawengine/libraw/src/tables/colorconst.cpp
M  +112  -71   core/libs/rawengine/libraw/src/tables/colordata.cpp
M  +11   -4    core/libs/rawengine/libraw/src/tables/wblists.cpp
M  +1    -1    core/libs/rawengine/libraw/src/utils/curves.cpp
M  +10   -5    core/libs/rawengine/libraw/src/utils/decoder_info.cpp
M  +97   -48   core/libs/rawengine/libraw/src/utils/init_close_utils.cpp
M  +114  -43   core/libs/rawengine/libraw/src/utils/open.cpp
M  +2    -2    core/libs/rawengine/libraw/src/utils/phaseone_processing.cpp
M  +41   -13   core/libs/rawengine/libraw/src/utils/read_utils.cpp
M  +10   -10   core/libs/rawengine/libraw/src/utils/thumb_utils.cpp
M  +3    -10   core/libs/rawengine/libraw/src/utils/utils_dcraw.cpp
M  +7    -1    core/libs/rawengine/libraw/src/utils/utils_libraw.cpp
M  +1    -1    core/libs/rawengine/libraw/src/write/apply_profile.cpp
M  +155  -108  core/libs/rawengine/libraw/src/write/file_write.cpp
M  +9    -2    core/libs/rawengine/libraw/src/write/tiff_writer.cpp
M  +1    -1    core/libs/rawengine/libraw/src/write/write_ph.cpp
M  +5    -23   core/libs/rawengine/libraw/src/x3f/x3f_parse_process.cpp
M  +1    -5    core/libs/rawengine/libraw/src/x3f/x3f_utils_patched.cpp

https://invent.kde.org/graphics/digikam/commit/efc5d2e6d84e377e3b5a994f97a10e0ce586e61a
Comment 17 doommaster29 2021-01-30 11:32:40 UTC 
Thank you Caulier. How do I update?
Comment 18 Maik Qualmann 2021-01-30 12:12:53 UTC 
The problem with this particular DNG image has not yet been resolved.

Maik
Comment 19 Maik Qualmann 2021-01-30 18:00:50 UTC 
I looked again at the DNG image that libraw decoded and I'm sure that the problem is not due to digiKam. Only the Nokia 7 is included in the list of supported cameras. Report the problem to the libraw team with a sample image so they can add support.

Maik
Comment 20 doommaster29 2021-02-01 16:45:03 UTC 
Where can I do that? Thank you
Comment 21 caulier.gilles 2021-02-01 17:16:58 UTC 
libraw project is hosted to github. open a new issue here :

https://github.com/LibRaw/LibRaw/issues

Gilles Caulier
Comment 22 caulier.gilles 2021-05-05 09:08:45 UTC 
Maik, 

If DNG SDK 1.5 is used with last libraw source code imported in git/master, this will fix certainly this issue...

Gilles
Comment 23 caulier.gilles 2021-05-11 03:59:39 UTC 
Git commit 2e4f528a1b5c0fbba8916f3a471b081f912cbf4e by Gilles Caulier.
Committed on 11/05/2021 at 03:54.
Pushed by cgilles into branch 'master'.

Internal DNG SDK 1.15 have been updated to digiKam 7.3.0. Link internal libraw with this DNG SDK to improve DNG decoding images.
Related: bug 435085, bug 330920, bug 418645

M  +2    -2    core/libs/rawengine/CMakeLists.txt
M  +3    -1    core/tests/rawengine/CMakeLists.txt

https://invent.kde.org/graphics/digikam/commit/2e4f528a1b5c0fbba8916f3a471b081f912cbf4e
Comment 24 caulier.gilles 2021-05-11 04:18:43 UTC 
doommaster29,

After upgrading DNG SDK to 1.5, and using this SDK with internal libraw, this do not fix the problem. Note that digiKam use internally Libraw source code taken from github/master repository at 2021-05-04. 

If i load your image in editor, the RAW import tool show a broken image due to wrong image size taken by libraw to process decoding.

Exiv2 and ExifTool show the right full size image properly(4032x3016). It sound like libraw use the reduced size version from Exif (377x304).

Note that embedded JPEG preview work as expected. Only demosaicing DNG do not work.

So this problem must be reported to libraw team as DOWNSTREAM bug. Project is hosted on gihub : 

https://github.com/LibRaw/LibRaw/issues

Thanks in advance

Gilles Caulier
Comment 25 doommaster29 2021-05-16 18:47:06 UTC 
Thank you. I'm unfamiliar with this code reporting, but I'll try doing so.
Comment 26 caulier.gilles 2021-05-16 21:18:30 UTC 
libraw UPSTREAM bug : https://github.com/LibRaw/LibRaw/issues/404
Comment 27 Alex Tutubalin 2021-05-17 05:08:28 UTC 
(In reply to caulier.gilles from comment #24)
> 
> After upgrading DNG SDK to 1.5, and using this SDK with internal libraw,
> this do not fix the problem. 

Have you set imgdata.options.use_dng_sdk to LIBRAW_DNG_ALL?

Alex Tutubalin, LibRaw LLC
Comment 28 caulier.gilles 2021-05-17 08:04:17 UTC 
Hi Alex,

Thanks for your guidance. No i forget to setup DNG sdk and dng_host with raw decoder.

Note: in README.DNGSDK, this line is wrong:

https://github.com/LibRaw/LibRaw/blob/master/README.DNGSDK.txt#L29

imgdata.params.use_dngsdk => imgdata.rawparams.use_dngsdk

Best

Gilles
Comment 29 caulier.gilles 2021-05-17 08:33:12 UTC 
Git commit b7f5446bdd59e50f1c44bea13182e1179fc64bdb by Gilles Caulier.
Committed on 17/05/2021 at 08:15.
Pushed by cgilles into branch 'master'.

Setup DNG SDK 1.5 with internal Libraw to process some rare flavours of DNG files (e.g. for 8-bit files).
These files never come from camera or DNG files created by Adobe DNG Converter, but created by some processing
software that uses DNG as intermediate format, aka generated with Nokia 9 PureView.

Screenshot of Nokia 9 PureView items properly decoded in half size (preview) or in full size (editor raw import tool)
with digiKam 7.3.0: https://i.imgur.com/Wi27qUC.png
FIXED-IN: 7.3.0

M  +105  -0    core/libs/rawengine/drawdecoder_p.cpp

https://invent.kde.org/graphics/digikam/commit/b7f5446bdd59e50f1c44bea13182e1179fc64bdb
Comment 30 Alex Tutubalin 2021-05-18 07:32:56 UTC 
Thanks, actualized with https://github.com/LibRaw/LibRaw/commit/fa042b324cb341fa63852d8d00b610f652b3b467