Created attachment 134930 [details] Shows corruption SUMMARY STEPS TO REPRODUCE 1. Install digiKam on Windows 10. 2. When you select a RAW photo, it randomly gets horribly corrupted (taken with Nokia 9 PureView, at least) 3. Interestingly, issue propagates, and also happens in other programs as RawTherapee (5.8) BUT NOT TO LIGHTROOM. At least, it hasn't happened yet. 4. Also of note, the medium RAW image that serves for the miniature, doesn't seem to care either, even though is supposed to be the RAW itself, not a preview. 4. Can't be traced what triggers this, so I don't know if its camera (Nokia 9 PureView) or software related. But as said Lightroom doesn't show this behaviour. OBSERVED RESULT Image corrupted under both digiKam or RawTherappe, but not Lightroom, although processing seems different. EXPECTED RESULT Not corrupted Image. SOFTWARE/OS VERSIONS Windows: 10 macOS: Linux/KDE Plasma: (available in About System) KDE Plasma Version: KDE Frameworks Version: Qt Version: ADDITIONAL INFORMATION
I suspect based on the screenshot that they use embedded previews for display. You can activate in the digiKam settings that half a RAW preview is used for RAW files. Please test whether the image is displayed correctly. Do you write metadata in DNG files (is deactivated by default in digiKam)? If the images haven't been edited yet and RawTherappe shows the same symptom, I suspect the problem is related to Nokia 9 PureView. Or with the program that created the DNG files. Please upload a problematic DNG file, if not publicly, to my email address. Maik
Created attachment 134931 [details] RAW settings
Created attachment 134932 [details] Metadata
Created attachment 134933 [details] Metada 2
Thank you Maik. Yes, I meant half RAW, I did a bad translation 'medium RAW'. I checked no box for metadata. I see only the 'update mod. date when file is modified' and writing metadata when rotating image.
Which digiKam version do you use exactly ? Gilles Caulier
7.1
Please try with 7.2.0 release candidate available here : https://files.kde.org/digikam/ ... where libraw have been updated to last stable version with plenty of fixes. Gilles Caulier
Thank you Cauiler. So I did, to no avail. But perhaps because images were already "corrupted". I couldn't replicate the effect. But I don't know how its triggered anyway.
I don't understand. Do you mean that 7.2.0rc fix your problem (aka preview is now displayed properly) ? Gilles Caulier
No. The image previously corrupted stays the same.
How did you generate the DNg files ? directly from the camera ? Form the digiKam DNG converter ? From another RAW processing application ? This DNG image can be shared through an internet file storage for investigations ? Gilles Caulier
I do nothing to the DNGs, but I've detected some .pp3 files, so some modification could take place. But not in all cases. Delete them did nothing to solve the issue. I include a link to some example images for your peruse. https://1drv.ms/u/s!AuaRuWZfqhhGluIdUEpRQv7nLnoucw?e=PPDhTC I hope you are now able to reproduce whatever is causing this. I really thank you for your help.
First I go from "critical" to "normal". DigiKam did not create or change the DNG. I can also reproduce the problem with the URAW program. Darktable displays it correctly. Maik
Thank you Maik, I've been able to reproduce it. At least a variant. Happens mostly to black and white images. And this could give a clue, I think, because this camera uses dedicated monochrome sensor for this: no color filter on-top. So no normal demosaicing should take place. So I guess, I'm not an expert. I upload a new image I just took and opened with digiKam right away, with the same result. And jpeg as well. Anyway, any idea of what could be done to use them with digiKam again? Perhaps this helps with this kind of hardware. Thank you! https://1drv.ms/f/s!AuaRuWZfqhhGluIdUEpRQv7nLnoucw
Git commit efc5d2e6d84e377e3b5a994f97a10e0ce586e61a by Gilles Caulier. Committed on 30/01/2021 at 08:29. Pushed by cgilles into branch 'master'. Update internal Libraw engine to last 0.21.0 Snapshot 202101 with many improvement about DNG support. For more detail see the official announcement: https://www.libraw.org/news/libraw-202101-snapshot Related: bug 418645 M +19 -1 NEWS M +1 -1 core/libs/rawengine/libraw/COPYRIGHT M +154 -25 core/libs/rawengine/libraw/Changelog.txt M +1 -1 core/libs/rawengine/libraw/README.md M +1 -1 core/libs/rawengine/libraw/internal/dcraw_common.cpp M +9 -5 core/libs/rawengine/libraw/internal/dcraw_defs.h M +1 -1 core/libs/rawengine/libraw/internal/dcraw_fileio.cpp M +1 -1 core/libs/rawengine/libraw/internal/dcraw_fileio_defs.h M +2 -1 core/libs/rawengine/libraw/internal/defines.h M +1 -1 core/libs/rawengine/libraw/internal/dmp_include.h M +9 -3 core/libs/rawengine/libraw/internal/libraw_cameraids.h M +1 -1 core/libs/rawengine/libraw/internal/libraw_cxx_defs.h M +276 -269 core/libs/rawengine/libraw/internal/libraw_internal_funcs.h M +150 -149 core/libs/rawengine/libraw/internal/var_defines.h M +1 -1 core/libs/rawengine/libraw/internal/x3f_tools.h M +46 -8 core/libs/rawengine/libraw/libraw/libraw.h M +1 -1 core/libs/rawengine/libraw/libraw/libraw_alloc.h M +111 -51 core/libs/rawengine/libraw/libraw/libraw_const.h M +99 -2 core/libs/rawengine/libraw/libraw/libraw_datastream.h M +4 -4 core/libs/rawengine/libraw/libraw/libraw_internal.h M +185 -92 core/libs/rawengine/libraw/libraw/libraw_types.h M +5 -5 core/libs/rawengine/libraw/libraw/libraw_version.h M +1 -1 core/libs/rawengine/libraw/samples/4channels.cpp M +15 -18 core/libs/rawengine/libraw/samples/dcraw_emu.cpp M +1 -1 core/libs/rawengine/libraw/samples/dcraw_half.c M +1 -1 core/libs/rawengine/libraw/samples/half_mt.c M +1 -1 core/libs/rawengine/libraw/samples/half_mt_win32.c M +1 -1 core/libs/rawengine/libraw/samples/mem_image_sample.cpp M +1 -1 core/libs/rawengine/libraw/samples/multirender_test.cpp M +1 -1 core/libs/rawengine/libraw/samples/openbayer_sample.cpp M +2 -2 core/libs/rawengine/libraw/samples/postprocessing_benchmark.cpp M +528 -1481 core/libs/rawengine/libraw/samples/raw-identify.cpp M +1 -1 core/libs/rawengine/libraw/samples/rawtextdump.cpp M +2 -2 core/libs/rawengine/libraw/samples/simple_dcraw.cpp M +1 -1 core/libs/rawengine/libraw/samples/unprocessed_raw.cpp M +1 -1 core/libs/rawengine/libraw/src/decoders/canon_600.cpp M +674 -377 core/libs/rawengine/libraw/src/decoders/crx.cpp M +25 -41 core/libs/rawengine/libraw/src/decoders/decoders_dcraw.cpp M +8 -9 core/libs/rawengine/libraw/src/decoders/decoders_libraw.cpp M +8 -12 core/libs/rawengine/libraw/src/decoders/decoders_libraw_dcrdefs.cpp M +1 -1 core/libs/rawengine/libraw/src/decoders/dng.cpp M +234 -98 core/libs/rawengine/libraw/src/decoders/fp_dng.cpp M +375 -296 core/libs/rawengine/libraw/src/decoders/fuji_compressed.cpp M +6 -17 core/libs/rawengine/libraw/src/decoders/generic.cpp M +31 -58 core/libs/rawengine/libraw/src/decoders/kodak_decoders.cpp M +1 -1 core/libs/rawengine/libraw/src/decoders/load_mfbacks.cpp M +1 -1 core/libs/rawengine/libraw/src/decoders/smal.cpp M +18 -14 core/libs/rawengine/libraw/src/decoders/unpack.cpp M +21 -16 core/libs/rawengine/libraw/src/decoders/unpack_thumb.cpp M +0 -6 core/libs/rawengine/libraw/src/demosaic/aahd_demosaic.cpp M +1 -1 core/libs/rawengine/libraw/src/demosaic/ahd_demosaic.cpp M +1 -1 core/libs/rawengine/libraw/src/demosaic/misc_demosaic.cpp M +1 -1 core/libs/rawengine/libraw/src/demosaic/xtrans_demosaic.cpp M +21 -16 core/libs/rawengine/libraw/src/integration/dngsdk_glue.cpp M +1 -1 core/libs/rawengine/libraw/src/integration/rawspeed_glue.cpp M +7 -2 core/libs/rawengine/libraw/src/libraw_c_api.cpp M +1 -1 core/libs/rawengine/libraw/src/libraw_cxx.cpp M +342 -4 core/libs/rawengine/libraw/src/libraw_datastream.cpp M +1 -1 core/libs/rawengine/libraw/src/metadata/adobepano.cpp M +155 -125 core/libs/rawengine/libraw/src/metadata/canon.cpp M +5 -6 core/libs/rawengine/libraw/src/metadata/ciff.cpp M +45 -2 core/libs/rawengine/libraw/src/metadata/cr3_parser.cpp M +1 -1 core/libs/rawengine/libraw/src/metadata/epson.cpp M +1 -1 core/libs/rawengine/libraw/src/metadata/exif_gps.cpp M +434 -381 core/libs/rawengine/libraw/src/metadata/fuji.cpp M +82 -35 core/libs/rawengine/libraw/src/metadata/hasselblad_model.cpp M +250 -180 core/libs/rawengine/libraw/src/metadata/identify.cpp M +1 -1 core/libs/rawengine/libraw/src/metadata/identify_tools.cpp M +1 -1 core/libs/rawengine/libraw/src/metadata/kodak.cpp M +69 -52 core/libs/rawengine/libraw/src/metadata/leica.cpp M +71 -186 core/libs/rawengine/libraw/src/metadata/makernotes.cpp M +15 -7 core/libs/rawengine/libraw/src/metadata/mediumformat.cpp M +31 -33 core/libs/rawengine/libraw/src/metadata/minolta.cpp M +5 -3 core/libs/rawengine/libraw/src/metadata/misc_parsers.cpp M +55 -22 core/libs/rawengine/libraw/src/metadata/nikon.cpp M +67 -19 core/libs/rawengine/libraw/src/metadata/normalize_model.cpp M +180 -20 core/libs/rawengine/libraw/src/metadata/olympus.cpp M +32 -30 core/libs/rawengine/libraw/src/metadata/p1.cpp M +139 -14 core/libs/rawengine/libraw/src/metadata/pentax.cpp M +1 -1 core/libs/rawengine/libraw/src/metadata/samsung.cpp M +746 -538 core/libs/rawengine/libraw/src/metadata/sony.cpp M +113 -60 core/libs/rawengine/libraw/src/metadata/tiff.cpp M +1 -1 core/libs/rawengine/libraw/src/postprocessing/aspect_ratio.cpp M +7 -2 core/libs/rawengine/libraw/src/postprocessing/dcraw_process.cpp M +5 -2 core/libs/rawengine/libraw/src/postprocessing/mem_image.cpp M +1 -1 core/libs/rawengine/libraw/src/postprocessing/postprocessing_aux.cpp M +1 -1 core/libs/rawengine/libraw/src/postprocessing/postprocessing_ph.cpp M +1 -1 core/libs/rawengine/libraw/src/postprocessing/postprocessing_utils.cpp M +2 -2 core/libs/rawengine/libraw/src/postprocessing/postprocessing_utils_dcrdefs.cpp M +3 -6 core/libs/rawengine/libraw/src/preprocessing/ext_preprocess.cpp M +1 -1 core/libs/rawengine/libraw/src/preprocessing/preprocessing_ph.cpp M +7 -3 core/libs/rawengine/libraw/src/preprocessing/raw2image.cpp M +2 -2 core/libs/rawengine/libraw/src/preprocessing/subtract_black.cpp M +64 -17 core/libs/rawengine/libraw/src/tables/cameralist.cpp M +1 -1 core/libs/rawengine/libraw/src/tables/colorconst.cpp M +112 -71 core/libs/rawengine/libraw/src/tables/colordata.cpp M +11 -4 core/libs/rawengine/libraw/src/tables/wblists.cpp M +1 -1 core/libs/rawengine/libraw/src/utils/curves.cpp M +10 -5 core/libs/rawengine/libraw/src/utils/decoder_info.cpp M +97 -48 core/libs/rawengine/libraw/src/utils/init_close_utils.cpp M +114 -43 core/libs/rawengine/libraw/src/utils/open.cpp M +2 -2 core/libs/rawengine/libraw/src/utils/phaseone_processing.cpp M +41 -13 core/libs/rawengine/libraw/src/utils/read_utils.cpp M +10 -10 core/libs/rawengine/libraw/src/utils/thumb_utils.cpp M +3 -10 core/libs/rawengine/libraw/src/utils/utils_dcraw.cpp M +7 -1 core/libs/rawengine/libraw/src/utils/utils_libraw.cpp M +1 -1 core/libs/rawengine/libraw/src/write/apply_profile.cpp M +155 -108 core/libs/rawengine/libraw/src/write/file_write.cpp M +9 -2 core/libs/rawengine/libraw/src/write/tiff_writer.cpp M +1 -1 core/libs/rawengine/libraw/src/write/write_ph.cpp M +5 -23 core/libs/rawengine/libraw/src/x3f/x3f_parse_process.cpp M +1 -5 core/libs/rawengine/libraw/src/x3f/x3f_utils_patched.cpp https://invent.kde.org/graphics/digikam/commit/efc5d2e6d84e377e3b5a994f97a10e0ce586e61a
Thank you Caulier. How do I update?
The problem with this particular DNG image has not yet been resolved. Maik
I looked again at the DNG image that libraw decoded and I'm sure that the problem is not due to digiKam. Only the Nokia 7 is included in the list of supported cameras. Report the problem to the libraw team with a sample image so they can add support. Maik
Where can I do that? Thank you
libraw project is hosted to github. open a new issue here : https://github.com/LibRaw/LibRaw/issues Gilles Caulier
Maik, If DNG SDK 1.5 is used with last libraw source code imported in git/master, this will fix certainly this issue... Gilles
Git commit 2e4f528a1b5c0fbba8916f3a471b081f912cbf4e by Gilles Caulier. Committed on 11/05/2021 at 03:54. Pushed by cgilles into branch 'master'. Internal DNG SDK 1.15 have been updated to digiKam 7.3.0. Link internal libraw with this DNG SDK to improve DNG decoding images. Related: bug 435085, bug 330920, bug 418645 M +2 -2 core/libs/rawengine/CMakeLists.txt M +3 -1 core/tests/rawengine/CMakeLists.txt https://invent.kde.org/graphics/digikam/commit/2e4f528a1b5c0fbba8916f3a471b081f912cbf4e
doommaster29, After upgrading DNG SDK to 1.5, and using this SDK with internal libraw, this do not fix the problem. Note that digiKam use internally Libraw source code taken from github/master repository at 2021-05-04. If i load your image in editor, the RAW import tool show a broken image due to wrong image size taken by libraw to process decoding. Exiv2 and ExifTool show the right full size image properly(4032x3016). It sound like libraw use the reduced size version from Exif (377x304). Note that embedded JPEG preview work as expected. Only demosaicing DNG do not work. So this problem must be reported to libraw team as DOWNSTREAM bug. Project is hosted on gihub : https://github.com/LibRaw/LibRaw/issues Thanks in advance Gilles Caulier
Thank you. I'm unfamiliar with this code reporting, but I'll try doing so.
libraw UPSTREAM bug : https://github.com/LibRaw/LibRaw/issues/404
(In reply to caulier.gilles from comment #24) > > After upgrading DNG SDK to 1.5, and using this SDK with internal libraw, > this do not fix the problem. Have you set imgdata.options.use_dng_sdk to LIBRAW_DNG_ALL? Alex Tutubalin, LibRaw LLC
Hi Alex, Thanks for your guidance. No i forget to setup DNG sdk and dng_host with raw decoder. Note: in README.DNGSDK, this line is wrong: https://github.com/LibRaw/LibRaw/blob/master/README.DNGSDK.txt#L29 imgdata.params.use_dngsdk => imgdata.rawparams.use_dngsdk Best Gilles
Git commit b7f5446bdd59e50f1c44bea13182e1179fc64bdb by Gilles Caulier. Committed on 17/05/2021 at 08:15. Pushed by cgilles into branch 'master'. Setup DNG SDK 1.5 with internal Libraw to process some rare flavours of DNG files (e.g. for 8-bit files). These files never come from camera or DNG files created by Adobe DNG Converter, but created by some processing software that uses DNG as intermediate format, aka generated with Nokia 9 PureView. Screenshot of Nokia 9 PureView items properly decoded in half size (preview) or in full size (editor raw import tool) with digiKam 7.3.0: https://i.imgur.com/Wi27qUC.png FIXED-IN: 7.3.0 M +105 -0 core/libs/rawengine/drawdecoder_p.cpp https://invent.kde.org/graphics/digikam/commit/b7f5446bdd59e50f1c44bea13182e1179fc64bdb
Thanks, actualized with https://github.com/LibRaw/LibRaw/commit/fa042b324cb341fa63852d8d00b610f652b3b467