When opening a NIKON Raw Picture (="NEF"), digiKam provides terrible visual artefacts Reproducible: Always Steps to Reproduce: 1. just open a recent NEF 2. 3. Expected Results: image editor should be able to edit NEFs directly please compare with pics in following mail
Created attachment 73272 [details] direct import of NEF leads to pattern/artefacts
Created attachment 73273 [details] GIMP 2.6 imports appearantly correct
Which libkdcraw and libraw you use (Help/components Info for details) Which Nikon camera you use ? Which RAW demosaicing settings you use exactly ? Can you share the NEF file to test here... Gilles Caulier
Hi Gilles, thank you for quick response: --- Components: digiKam version 2.6.0 Exiv2 kann in JP2 speichern: Ja Exiv2 kann in JPEG speichern: Ja Exiv2 kann in PGF speichern: Ja Exiv2 kann in PNG speichern: Ja Exiv2 kann in TIFF speichern: Ja Exiv2 unterstützt XMP-Metadaten: Ja LibCImg: 130 LibClapack: Interne Bibliothek LibExiv2: 0.22 LibJPEG: 80 LibJasper: 1.900.1 LibKDE: 4.9.00 LibKExiv2: 2.1.0 LibKGeoMap: 2.0.0 LibKdcraw: 2.0.1 LibLCMS: 119 LibLensFun: Externe gemeinsame Bibliothek LibLqr: Interne Bibliothek LibPGF: 6.11.42 - Interne Bibliothek LibPNG: 1.2.46 LibQt: 4.8.1 LibRaw: 0.14.4 LibTIFF: LIBTIFF, Version 3.9.5 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc. Marble-Funktion: 0.12.2 (stable release) Parallelisiertes Entfernen von Mosaikmustern: Ja Datenbanktreiber: QSQLITE LibGphoto2: 2.4.13 LibKface: 2.0.0 LibKipi: 1.3.0 LibOpenCV: 2.3.1 Libface: 0.2 --- NIKON D3100, irmware Version: A: 1.01 B: 1.01 L: 1.002 --- RAW demosaicing settings? settings > raw-decoding > behavior: - use pre-setting 16 bit presettings (using libraw 0.14.4): - please see attachment !! --- share BEF-pic? > YES, enclosed (hope, this pic is not topo large!?) Axel --- Am 18.08.2012 13:44, schrieb Gilles Caulier: > https://bugs.kde.org/show_bug.cgi?id=305382 > > Gilles Caulier <caulier.gilles@gmail.com> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |caulier.gilles@gmail.com > Component|Image Editor |libkdcraw > > --- Comment #3 from Gilles Caulier <caulier.gilles@gmail.com> --- > Which libkdcraw and libraw you use (Help/components Info for details) > > Which Nikon camera you use ? > > Which RAW demosaicing settings you use exactly ? > > Can you share the NEF file to test here... > > Gilles Caulier >
Apparently too large. You can send it via private mail to me.
Hi Marcel: I just returned from some holidays... please excuse me for delay... Thank you for your offer! - When opening DSC_8604_NEF (digiKam, Version 2.6.0, Unter KDE 4.9.00), I experience artefacts, looking like DSC_8604XX.jpg. Settings you'll find in screenshot DSC_8605_XX_GIMP27_b.jpg. Other details in my bug report. Hope you can work with it. Please let me know, if I can help/clarify further details. Axel encl. Am 22.08.2012 20:33, schrieb Marcel Wiesweg: > https://bugs.kde.org/show_bug.cgi?id=305382 > > --- Comment #5 from Marcel Wiesweg <marcel.wiesweg@gmx.de> --- > Apparently too large. You can send it via private mail to me. >
Marcel, I can reproduce the problem with all RAW files from my test collection using Bilinear demosaicing (and some other quality settings but not all). I talk with Alex about this problem. He ask me if i can reproduce it with official libraw tarball... I can confirm that it's not reproducible... I written some test program to check if problem is relevant of : 1/ multithreading from digiKam core ==> no. 2/ DImg image loader ==> no. 3/ Extra compilation flags set by KDE ==> no. 4/ Check OPenMP => yes. To reproduce the problem, you need a multicore processor, and libkdcraw/libraw must be compiled against OPenMP. If you disabled OPenMP support, it's work. There is no artefacts, but demosaicing is not parallelized. I suspected in first a bug in Libraw. But the problem is that i cannot reproduce the problem when it's compiled/linked against OpenMP. THE question is why ??? - I checked again and again all CMake rules from libraw. I well separated libraw compilation rules and libkdcraw into cmake scripts, just to be sure. I take a care about all definitions passed to compile libraw in official tarball, checked if i compiled libraw from libkdcraw with same dependencies than offcial libraw (lcms, rawspeed, gpl2 and gpl3 extra pack, libjpeg, libjpasper) : It's always reproducible... I cannot found the problem, and i don't have any idea... Help is welcome... Gilles Caulier
Just to precise the comparison : I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with libkdcraw => artefacts. I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with offcial tarball => no artefacts. Gilles Caulier
Created attachment 74809 [details] artefacts given with OpenMP support
So essentially, the problem is that when we compile the same sources using the same compiler and linker command lines with either CMake, or whatever libraw uses, we get a different result? Does the original libraw also compile into a static or a dynamic library? If shared: With libkdcraw, we compile into a static library. Assuming we use a shared library ADD_LIBRARY(libraw SHARED ${libraw_LIB_SRCS}) still the same problem? Do the files differ?
>So essentially, the problem is that when we compile the same sources using the same compiler and linker >command lines with either CMake, or whatever libraw uses, we get a different result? Yes, absolutly... Gilles Caulier
Created attachment 74824 [details] libraw as SHARED patch Marcel, See as attached my patch to use libraw as SHARED. libraw sample files do not link with it : [gilles@localhost libraw]$ make Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/internal/dcraw_common.cpp.o Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/internal/dcraw_fileio.cpp.o Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/internal/demosaic_packs.cpp.o Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/src/libraw_cxx.cpp.o Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/src/libraw_c_api.cpp.o Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/src/libraw_datastream.cpp.o Linking CXX shared library ../../../lib/libraw.so Built target raw Building CXX object extra/libkdcraw/libraw/CMakeFiles/4channels.dir/samples/4channels.cpp.o Linking CXX executable 4channels CMakeFiles/4channels.dir/samples/4channels.cpp.o: In function `main': /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:46: undefined reference to `LibRaw::LibRaw(unsigned int)' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:59: undefined reference to `LibRaw::cameraCount()' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:59: undefined reference to `LibRaw::version()' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:102: undefined reference to `LibRaw::open_file(char const*, long long)' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:104: undefined reference to `libraw_strerror' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:112: undefined reference to `LibRaw::unpack()' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:114: undefined reference to `libraw_strerror' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:117: undefined reference to `LibRaw::raw2image()' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:120: undefined reference to `LibRaw::subtract_black()' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:173: undefined reference to `LibRaw::dcraw_ppm_tiff_writer(char const*)' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:174: undefined reference to `libraw_strerror' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:178: undefined reference to `LibRaw::~LibRaw()' /mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:178: undefined reference to `LibRaw::~LibRaw()' collect2: ld a retourné 1 code d'état d'exécution make[2]: *** [extra/libkdcraw/libraw/4channels] Erreur 1 make[1]: *** [extra/libkdcraw/libraw/CMakeFiles/4channels.dir/all] Erreur 2 make: *** [all] Erreur 2 [gilles@localhost libraw]$ Why ? Gilles Caulier
Created attachment 74825 [details] original libraw 0.15.0-beta1 makefile See also makefile that i used to compile original libraw to compare with libkdcraw version... Gilles Caulier
Marcel, I tried another way : compile libraw source code directly with all sample files, instead to use a STATIC or a SHARED libraw : # -- Compile LibRaw library -------------------------------------------------------------------------------- # Flag to include demosaic pack GPL2 ADD_DEFINITIONS(-DLIBRAW_DEMOSAIC_PACK_GPL2) # Flag to include demosaic pack GPL3 ADD_DEFINITIONS(-DLIBRAW_DEMOSAIC_PACK_GPL3) # Flag to debug LibRaw ADD_DEFINITIONS(-DDCRAW_VERBOSE) # Flag used by default into LibRaw to not use dedicated external thread ADD_DEFINITIONS(-DLIBRAW_NOTHREADS) # Flag to export library symbols ADD_DEFINITIONS(-DLIBRAW_BUILDLIB) INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}/ ${CMAKE_CURRENT_SOURCE_DIR}/demosaic-pack-GPL2 ${CMAKE_CURRENT_SOURCE_DIR}/demosaic-pack-GPL3 ) # Do not compile LibRaw with KDE FINAL mode. KDE4_NO_ENABLE_FINAL(raw) SET(raw_LIB_SRCS ${CMAKE_CURRENT_SOURCE_DIR}/internal/dcraw_common.cpp ${CMAKE_CURRENT_SOURCE_DIR}/internal/dcraw_fileio.cpp ${CMAKE_CURRENT_SOURCE_DIR}/internal/demosaic_packs.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/libraw_cxx.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/libraw_c_api.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/libraw_datastream.cpp ) # Disable compilation warnings from LibRaw. Just to be clear on the console. # Adjust flag for static lib and 64 bits computers using -fPIC for GCC compiler (B.K.O: #269903) FOREACH(_curentfile ${raw_LIB_SRCS}) SET_SOURCE_FILES_PROPERTIES(${_curentfile} PROPERTIES COMPILE_FLAGS "-w") ENDFOREACH(_curentfile ${raw_LIB_SRCS}) # -- Compile LibRaw Samples -------------------------------------------------------------------------------- # add a small macro so that this is a bit cleaner MACRO(LIBRAW_BUILD_SAMPLES) SET(_filename ${ARGV0}) STRING(REPLACE "." ";" _temp ${_filename}) LIST(GET _temp 0 _target) SET(${_target}_SRCS ${CMAKE_CURRENT_SOURCE_DIR}/samples/${_filename}) SET_SOURCE_FILES_PROPERTIES(${${_target}_SRCS} PROPERTIES COMPILE_FLAGS -w) ADD_EXECUTABLE(${_target} ${raw_LIB_SRCS} ${${_target}_SRCS}) TARGET_LINK_LIBRARIES(${_target} ${OPENMP_LDFLAGS} ${MATH_LIBRARY} ${LCMS2_LIBRARIES}) IF(RAWSPEED_SUPPORT_CAN_BE_COMPILED) TARGET_LINK_LIBRARIES(${_target} librawspeed) ENDIF() IF(JPEG_FOUND) TARGET_LINK_LIBRARIES(${_target} ${JPEG_LIBRARY}) ENDIF() IF(JASPER_FOUND) TARGET_LINK_LIBRARIES(${_target} ${JASPER_LIBRARIES}) ENDIF() IF(WIN32) TARGET_LINK_LIBRARIES(${_target} ws2_32) ENDIF(WIN32) ENDMACRO(LIBRAW_BUILD_SAMPLES) LIBRAW_BUILD_SAMPLES(simple_dcraw.cpp) LIBRAW_BUILD_SAMPLES(mem_image.cpp) LIBRAW_BUILD_SAMPLES(dcraw_emu.cpp) LIBRAW_BUILD_SAMPLES(4channels.cpp) LIBRAW_BUILD_SAMPLES(unprocessed_raw.cpp) LIBRAW_BUILD_SAMPLES(raw-identify.cpp) LIBRAW_BUILD_SAMPLES(multirender_test.cpp) LIBRAW_BUILD_SAMPLES(postprocessing_benchmark.cpp) #IF(WIN32) # LIBRAW_BUILD_SAMPLES(half_mt_win32.c) #ELSE(WIN32) # LIBRAW_BUILD_SAMPLES(dcraw_half.c) # LIBRAW_BUILD_SAMPLES(half_mt.c) #ENDIF(WIN32) No chance. The result is exactly the same. output PPM image is always corrupted. This proof that linking with STATIC lib is not the problem here. Right ?
Created attachment 74829 [details] stand alone libraw tarball, based on cmake build system To be sure sure sure, especially to see if host KDE env is the problem, i created a stand alone tarball of libraw 0.15.0-beta1 with cmake compilation script, which do not create a SHARED or STATIC lib. The problem still the same.... So excepted a bug in libraw, i don't have any more idea to hack.... Alex ? Gilles Caulier
So the standalone tarball with its dcraw_emu also produces the problem, or not? If it does, it's a good isolated test case to start from.
Hello Gilles, Thank you for all these trouble shootings. I follow all these bug remarks with huge interest. But I am still wondering, whom do you talk to, as you address to Marcel (whose emailaddress I don't see) and to me, you don't talk to appearently. As I mentioned before, I can not contribute in software development, as much as I liked too. I do not even understand the last steps you were approaching the problem. The only thing which comes into my mind (as I am European Quality Manager and this discipline should offer some options even to software development): 1.) I would try rigous standardization, the less differences the better. American often say "keep it simple". 2.) I believe KDE is highly developped, but risky for complexity reasons. Every engineer could build a car, basicly. In reality, you need hundreds. I suspect KDEs technologies to still creep on a base level, somehow. 3.) One of the reasons of the tremendeous success of MACS is based on clear programming rules for developpers, if they want to be accpted by Masintoshes. - Why not use some of these rules? - Why not defining borders within software has to be if incorporated into the digiKam project? I could imagine, that undoubtful definition (as tif-format, e.g.) could help to stabilize digiKam overall. Use, what is defined, reject what is undefined. Sorry, some thoughts of a non-programmer... Coming back to your question mark. Please let me know, if I can help you and the digiKam project. Axel (P.S.: "Axel" _not_ "Alex") Am 27.10.2012 14:47, schrieb Gilles Caulier: > https://bugs.kde.org/show_bug.cgi?id=305382 > > --- Comment #15 from Gilles Caulier <caulier.gilles@gmail.com> --- > Created attachment 74829 [details] > --> https://bugs.kde.org/attachment.cgi?id=74829&action=edit > stand alone libraw tarball, based on cmake build system > > To be sure sure sure, especially to see if host KDE env is the problem, i > created a stand alone tarball of libraw 0.15.0-beta1 with cmake compilation > script, which do not create a SHARED or STATIC lib. > > The problem still the same.... > > So excepted a bug in libraw, i don't have any more idea to hack.... > > Alex ? > > Gilles Caulier >
Marcel, Yes, problem is reproducible with stand alone version. Axel, I talk to Alex Tutubalin, who is in CC in this file. He is libraw author... Gilles Caulier
Created attachment 74861 [details] Makefile.dist used to reproduce artifacts problem with LibRaw Alex, I found the problem... It's in libraw source code. I'm sure. I can reproduce the artifacts problem with official Libraw 0.15.0-beta1 tarball. The problem is about "-O4" gcc compilation optimization option. If you remove it from offcial MakeFile.dist, the artifacts problem is reproducible with "dcraw_emu -q0 myrawfile.NEF" (NEF is just and example, it's also reproducible with CR2 files...) Gilles Caulier
Alex, I can confirm also, that with my stand alone libraw compiled with CMake, if i use "-O4" optimization, as with official libraw tarball, artifacts disappears... It's definitively a libraw problem.... Gilles Caulier
Git commit ca11950fa5926ccfb4438d8b9cbd072370be3f0b by Gilles Caulier. Committed on 29/10/2012 at 11:55. Pushed by cgilles into branch 'master'. - Use -O4 compilation option to optimize speed with dosaicing process. - Fix artifacts produced by libraw when OpenMP is used. This problem still present in libraw and must be fixed as DOWNSTREAM Please confirm if probelm still reproducible on your computer with this fix. Related: bug 308489 M +2 -1 libraw/CMakeLists.txt http://commits.kde.org/libkdcraw/ca11950fa5926ccfb4438d8b9cbd072370be3f0b
Gilles + Marcel: This sounds great. Impressive work. Thank you guys! Please let me know, if I could help somehow. Axel (please, _NOT_ "Alex") --- Am 29.10.2012 10:13, schrieb Gilles Caulier: > https://bugs.kde.org/show_bug.cgi?id=305382 > > --- Comment #19 from Gilles Caulier <caulier.gilles@gmail.com> --- > Created attachment 74861 [details] > --> https://bugs.kde.org/attachment.cgi?id=74861&action=edit > Makefile.dist used to reproduce artifacts problem with LibRaw > > Alex, > > I found the problem... It's in libraw source code. I'm sure. I can reproduce > the artifacts problem with official Libraw 0.15.0-beta1 tarball. > > The problem is about "-O4" gcc compilation optimization option. If you remove > it from offcial MakeFile.dist, the artifacts problem is reproducible with > "dcraw_emu -q0 myrawfile.NEF" (NEF is just and example, it's also reproducible > with CR2 files...) > > Gilles Caulier >
Alex, Sorry, i make a mistake with bugzilla settings to CC you. Now it's fixed. Please review this entry and comment... Thanks in advance Gilles Caulier
There is no OpenMP speedup for linear interpolate in LibRaw 0.15-release I was unable to reproduce this problem (several months ago), anyway I've removed OpenMP speedup for safety. Please re-check with current LibRaw 0.15.1
Confirmed. Artefact are not reproducible with libraw 0.15.1 if compiled with OpenMP and no optimization -O4 is used. Gilles Caulier
(In reply to comment #8) > Just to precise the comparison : > > I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with libkdcraw => > artefacts. > > I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with offcial > tarball => no artefacts. > > Gilles Caulier Gilles: sorry for this late comment: the pic ***5199.CR2 are _not_ the ones which caused my bugreporting this issue!! The pic I reported was "DSC_8604_XX.jpg" Axel