Bug 305382

Created attachment 73272 [details]
direct import of NEF leads to pattern/artefacts

Created attachment 73273 [details]
GIMP 2.6 imports appearantly correct

Which libkdcraw and libraw you use (Help/components Info for details)

Which Nikon camera you use ?

Which RAW demosaicing settings you use exactly ?

Can you share the NEF file to test here...

Gilles Caulier

Hi Gilles,

thank you for quick response:

---
Components:
digiKam version 2.6.0
Exiv2 kann in JP2 speichern: Ja
Exiv2 kann in JPEG speichern: Ja
Exiv2 kann in PGF speichern: Ja
Exiv2 kann in PNG speichern: Ja
Exiv2 kann in TIFF speichern: Ja
Exiv2 unterstützt XMP-Metadaten: Ja
LibCImg: 130
LibClapack: Interne Bibliothek
LibExiv2: 0.22
LibJPEG: 80
LibJasper: 1.900.1
LibKDE: 4.9.00
LibKExiv2: 2.1.0
LibKGeoMap: 2.0.0
LibKdcraw: 2.0.1
LibLCMS: 119
LibLensFun: Externe gemeinsame Bibliothek
LibLqr: Interne Bibliothek
LibPGF: 6.11.42 - Interne Bibliothek
LibPNG: 1.2.46
LibQt: 4.8.1
LibRaw: 0.14.4
LibTIFF: LIBTIFF, Version 3.9.5 Copyright (c) 1988-1996 Sam Leffler
Copyright (c) 1991-1996 Silicon Graphics, Inc.
Marble-Funktion: 0.12.2 (stable release)
Parallelisiertes Entfernen von Mosaikmustern: Ja
Datenbanktreiber: QSQLITE
LibGphoto2: 2.4.13
LibKface: 2.0.0
LibKipi: 1.3.0
LibOpenCV: 2.3.1
Libface: 0.2
---

NIKON D3100, irmware Version:
A: 1.01
B: 1.01
L: 1.002
---
RAW demosaicing settings?
settings > raw-decoding >
behavior:
- use pre-setting 16 bit

presettings (using libraw 0.14.4):
- please see attachment

!!

---

share BEF-pic? > YES, enclosed (hope, this pic is not topo large!?)


Axel
---
Am 18.08.2012 13:44, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=305382
> 
> Gilles Caulier <caulier.gilles@gmail.com> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |caulier.gilles@gmail.com
>           Component|Image Editor                |libkdcraw
> 
> --- Comment #3 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Which libkdcraw and libraw you use (Help/components Info for details)
> 
> Which Nikon camera you use ?
> 
> Which RAW demosaicing settings you use exactly ?
> 
> Can you share the NEF file to test here...
> 
> Gilles Caulier
>

Apparently too large. You can send it via private mail to me.

Hi Marcel:

I just returned from some holidays... please excuse me for delay...


Thank you for your offer!

- When opening DSC_8604_NEF (digiKam, Version 2.6.0, Unter KDE 4.9.00),
I experience artefacts, looking like DSC_8604XX.jpg.

Settings you'll find in screenshot DSC_8605_XX_GIMP27_b.jpg.

Other details in my bug report.

Hope you can work with it.

Please let me know, if I can help/clarify further details.



Axel

encl.

Am 22.08.2012 20:33, schrieb Marcel Wiesweg:
> https://bugs.kde.org/show_bug.cgi?id=305382
> 
> --- Comment #5 from Marcel Wiesweg <marcel.wiesweg@gmx.de> ---
> Apparently too large. You can send it via private mail to me.
>

Marcel,

I can reproduce the problem with all RAW files from my test collection using Bilinear demosaicing (and some other quality settings but not all).

I talk with Alex about this problem. He ask me if i can reproduce it with official libraw tarball... I can confirm that it's not reproducible...

I written some test program to check if problem is relevant of :

1/ multithreading from digiKam core ==> no.
2/ DImg image loader ==> no.
3/ Extra compilation flags set by KDE ==> no.
4/ Check OPenMP => yes.

To reproduce the problem, you need a multicore processor, and libkdcraw/libraw must be compiled against OPenMP. If you disabled OPenMP support, it's work. There is no artefacts, but demosaicing is not parallelized.

I suspected in first a bug in Libraw. But the problem is that i cannot reproduce the problem when it's compiled/linked against OpenMP.

THE question is why ???

- I checked again and again all CMake rules from libraw. I well separated libraw compilation rules and libkdcraw into cmake scripts, just to be sure. I take a care about all definitions passed to compile libraw in official tarball, checked if i compiled libraw from libkdcraw with same dependencies than offcial libraw (lcms, rawspeed, gpl2 and gpl3 extra pack, libjpeg, libjpasper) : It's always reproducible... 

I cannot found the problem, and i don't have any idea... Help is welcome...

Gilles Caulier

Just to precise the comparison :

I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with libkdcraw => artefacts.

I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with offcial tarball => no artefacts.

Gilles Caulier

Created attachment 74809 [details]
artefacts given with OpenMP support

So essentially, the problem is that when we compile the same sources using the same compiler and linker command lines with either CMake, or whatever libraw uses, we get a different result?

Does the original libraw also compile into a static or a dynamic library? If shared: With libkdcraw, we compile into a static library. Assuming we use a shared library
ADD_LIBRARY(libraw SHARED ${libraw_LIB_SRCS})
still the same problem? Do the files differ?

>So essentially, the problem is that when we compile the same sources using the same compiler and linker >command lines with either CMake, or whatever libraw uses, we get a different result?

Yes, absolutly...

Gilles Caulier

Created attachment 74824 [details]
libraw as SHARED patch

Marcel,

See as attached my patch to use libraw as SHARED. libraw sample files do not link with it :

[gilles@localhost libraw]$ make
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/internal/dcraw_common.cpp.o
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/internal/dcraw_fileio.cpp.o
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/internal/demosaic_packs.cpp.o
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/src/libraw_cxx.cpp.o
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/src/libraw_c_api.cpp.o
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/src/libraw_datastream.cpp.o
Linking CXX shared library ../../../lib/libraw.so
Built target raw
Building CXX object extra/libkdcraw/libraw/CMakeFiles/4channels.dir/samples/4channels.cpp.o
Linking CXX executable 4channels
CMakeFiles/4channels.dir/samples/4channels.cpp.o: In function `main':
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:46: undefined reference to `LibRaw::LibRaw(unsigned int)'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:59: undefined reference to `LibRaw::cameraCount()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:59: undefined reference to `LibRaw::version()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:102: undefined reference to `LibRaw::open_file(char const*, long long)'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:104: undefined reference to `libraw_strerror'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:112: undefined reference to `LibRaw::unpack()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:114: undefined reference to `libraw_strerror'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:117: undefined reference to `LibRaw::raw2image()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:120: undefined reference to `LibRaw::subtract_black()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:173: undefined reference to `LibRaw::dcraw_ppm_tiff_writer(char const*)'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:174: undefined reference to `libraw_strerror'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:178: undefined reference to `LibRaw::~LibRaw()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:178: undefined reference to `LibRaw::~LibRaw()'
collect2: ld a retourné 1 code d'état d'exécution
make[2]: *** [extra/libkdcraw/libraw/4channels] Erreur 1
make[1]: *** [extra/libkdcraw/libraw/CMakeFiles/4channels.dir/all] Erreur 2
make: *** [all] Erreur 2

[gilles@localhost libraw]$

Why ?

Gilles Caulier

Created attachment 74825 [details]
original libraw 0.15.0-beta1 makefile

See also makefile that i used to compile original libraw to compare with libkdcraw version...

Gilles Caulier

Marcel,

I tried another way : compile libraw source code directly with all sample files, instead to use a STATIC or a SHARED libraw :

# -- Compile LibRaw library --------------------------------------------------------------------------------

# Flag to include demosaic pack GPL2
ADD_DEFINITIONS(-DLIBRAW_DEMOSAIC_PACK_GPL2)

# Flag to include demosaic pack GPL3
ADD_DEFINITIONS(-DLIBRAW_DEMOSAIC_PACK_GPL3)

# Flag to debug LibRaw
ADD_DEFINITIONS(-DDCRAW_VERBOSE)

# Flag used by default into LibRaw to not use dedicated external thread
ADD_DEFINITIONS(-DLIBRAW_NOTHREADS)

# Flag to export library symbols
ADD_DEFINITIONS(-DLIBRAW_BUILDLIB)


INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}/
                    ${CMAKE_CURRENT_SOURCE_DIR}/demosaic-pack-GPL2
                    ${CMAKE_CURRENT_SOURCE_DIR}/demosaic-pack-GPL3
                   )

# Do not compile LibRaw with KDE FINAL mode.
KDE4_NO_ENABLE_FINAL(raw)

SET(raw_LIB_SRCS ${CMAKE_CURRENT_SOURCE_DIR}/internal/dcraw_common.cpp
                 ${CMAKE_CURRENT_SOURCE_DIR}/internal/dcraw_fileio.cpp
                 ${CMAKE_CURRENT_SOURCE_DIR}/internal/demosaic_packs.cpp
                 ${CMAKE_CURRENT_SOURCE_DIR}/src/libraw_cxx.cpp
                 ${CMAKE_CURRENT_SOURCE_DIR}/src/libraw_c_api.cpp
                 ${CMAKE_CURRENT_SOURCE_DIR}/src/libraw_datastream.cpp
   )

# Disable compilation warnings from LibRaw. Just to be clear on the console.
# Adjust flag for static lib and 64 bits computers using -fPIC for GCC compiler (B.K.O: #269903)
FOREACH(_curentfile ${raw_LIB_SRCS})
    SET_SOURCE_FILES_PROPERTIES(${_curentfile} PROPERTIES COMPILE_FLAGS "-w")
ENDFOREACH(_curentfile ${raw_LIB_SRCS})

# -- Compile LibRaw Samples --------------------------------------------------------------------------------

# add a small macro so that this is a bit cleaner
MACRO(LIBRAW_BUILD_SAMPLES)

    SET(_filename ${ARGV0})
    STRING(REPLACE "." ";" _temp ${_filename})
    LIST(GET _temp 0 _target)

    SET(${_target}_SRCS ${CMAKE_CURRENT_SOURCE_DIR}/samples/${_filename})

    SET_SOURCE_FILES_PROPERTIES(${${_target}_SRCS} PROPERTIES COMPILE_FLAGS -w)

    ADD_EXECUTABLE(${_target} ${raw_LIB_SRCS} ${${_target}_SRCS})

    TARGET_LINK_LIBRARIES(${_target} ${OPENMP_LDFLAGS} ${MATH_LIBRARY} ${LCMS2_LIBRARIES})

    IF(RAWSPEED_SUPPORT_CAN_BE_COMPILED)
        TARGET_LINK_LIBRARIES(${_target} librawspeed)
    ENDIF()

    IF(JPEG_FOUND)
        TARGET_LINK_LIBRARIES(${_target} ${JPEG_LIBRARY})
    ENDIF()

    IF(JASPER_FOUND)
        TARGET_LINK_LIBRARIES(${_target} ${JASPER_LIBRARIES})
    ENDIF()

    IF(WIN32)
        TARGET_LINK_LIBRARIES(${_target} ws2_32)
    ENDIF(WIN32)

ENDMACRO(LIBRAW_BUILD_SAMPLES)

LIBRAW_BUILD_SAMPLES(simple_dcraw.cpp)
LIBRAW_BUILD_SAMPLES(mem_image.cpp)
LIBRAW_BUILD_SAMPLES(dcraw_emu.cpp)
LIBRAW_BUILD_SAMPLES(4channels.cpp)
LIBRAW_BUILD_SAMPLES(unprocessed_raw.cpp)
LIBRAW_BUILD_SAMPLES(raw-identify.cpp)
LIBRAW_BUILD_SAMPLES(multirender_test.cpp)
LIBRAW_BUILD_SAMPLES(postprocessing_benchmark.cpp)
#IF(WIN32)
#    LIBRAW_BUILD_SAMPLES(half_mt_win32.c)
#ELSE(WIN32)
#    LIBRAW_BUILD_SAMPLES(dcraw_half.c)
#    LIBRAW_BUILD_SAMPLES(half_mt.c)
#ENDIF(WIN32)

No chance. The result is exactly the same. output PPM image is always corrupted. This proof that linking with STATIC lib is not the problem here. Right ?

Created attachment 74829 [details]
stand alone libraw tarball, based on cmake build system

To be sure sure sure, especially to see if host KDE env is the problem, i created a stand alone tarball of libraw 0.15.0-beta1 with cmake compilation script, which do not create a SHARED or STATIC lib.

The problem still the same....

So excepted a bug in libraw, i don't have any more idea to hack....

Alex ?

Gilles Caulier

So the standalone tarball with its dcraw_emu also produces the problem, or not? If it does, it's a good isolated test case to start from.

Hello Gilles,

Thank you for all these trouble shootings. I follow all these bug
remarks with huge interest. But I am still wondering, whom do you talk
to, as you address to Marcel (whose emailaddress I don't see) and to me,
you don't talk to appearently.

As I mentioned before, I can not contribute in software development, as
much as I liked too. I do not even understand the last steps you were
approaching the problem.

The only thing which comes into my mind (as I am European Quality
Manager and this discipline should offer some options even to software
development):

1.) I would try rigous standardization, the less differences the better.
American often say "keep it simple".

2.) I believe KDE is highly developped, but risky for complexity
reasons. Every engineer could build a car, basicly. In reality, you need
hundreds. I suspect KDEs technologies to still creep on a base level,
somehow.

3.) One of the reasons of the tremendeous success of MACS is based on
clear programming rules for developpers, if they want to be accpted by
Masintoshes.
- Why not use some of these rules?
- Why not defining borders within software has to be if incorporated
into the digiKam project? I could imagine, that undoubtful definition
(as tif-format, e.g.) could help to stabilize digiKam overall. Use, what
is defined, reject what is undefined.

Sorry, some thoughts of a non-programmer...

Coming back to your question mark. Please let me know, if I can help you
and the digiKam project.


Axel

(P.S.: "Axel" _not_ "Alex")

Am 27.10.2012 14:47, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=305382
> 
> --- Comment #15 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Created attachment 74829 [details]
>   --> https://bugs.kde.org/attachment.cgi?id=74829&action=edit
> stand alone libraw tarball, based on cmake build system
> 
> To be sure sure sure, especially to see if host KDE env is the problem, i
> created a stand alone tarball of libraw 0.15.0-beta1 with cmake compilation
> script, which do not create a SHARED or STATIC lib.
> 
> The problem still the same....
> 
> So excepted a bug in libraw, i don't have any more idea to hack....
> 
> Alex ?
> 
> Gilles Caulier
>

Marcel,

Yes, problem is reproducible with stand alone version.

Axel,

I talk to Alex Tutubalin, who is in CC in this file. He is libraw author...

Gilles Caulier

Created attachment 74861 [details]
Makefile.dist used to reproduce artifacts problem with LibRaw

Alex,

I found the problem... It's in libraw source code. I'm sure. I can reproduce the artifacts problem with official Libraw 0.15.0-beta1 tarball.

The problem is about "-O4" gcc compilation optimization option. If you remove it from offcial MakeFile.dist, the artifacts problem is reproducible with "dcraw_emu -q0 myrawfile.NEF" (NEF is just and example, it's also reproducible with CR2 files...)

Gilles Caulier

Alex,

I can confirm also, that with my stand alone libraw compiled with CMake, if i use "-O4" optimization, as with official libraw tarball, artifacts disappears...

It's definitively a libraw problem....

Gilles Caulier

Git commit ca11950fa5926ccfb4438d8b9cbd072370be3f0b by Gilles Caulier.
Committed on 29/10/2012 at 11:55.
Pushed by cgilles into branch 'master'.

- Use -O4 compilation option to optimize speed with dosaicing process.

- Fix artifacts produced by libraw when OpenMP is used. This problem still present in libraw and must be fixed as DOWNSTREAM
  Please confirm if probelm still reproducible on your computer with this fix.
Related: bug 308489

M  +2    -1    libraw/CMakeLists.txt

http://commits.kde.org/libkdcraw/ca11950fa5926ccfb4438d8b9cbd072370be3f0b

Gilles + Marcel:

This sounds great. Impressive work. Thank you guys!

Please let me know, if I could help somehow.


Axel

(please, _NOT_ "Alex")
---
Am 29.10.2012 10:13, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=305382
> 
> --- Comment #19 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Created attachment 74861 [details]
>   --> https://bugs.kde.org/attachment.cgi?id=74861&action=edit
> Makefile.dist used to reproduce artifacts problem with LibRaw
> 
> Alex,
> 
> I found the problem... It's in libraw source code. I'm sure. I can reproduce
> the artifacts problem with official Libraw 0.15.0-beta1 tarball.
> 
> The problem is about "-O4" gcc compilation optimization option. If you remove
> it from offcial MakeFile.dist, the artifacts problem is reproducible with
> "dcraw_emu -q0 myrawfile.NEF" (NEF is just and example, it's also reproducible
> with CR2 files...)
> 
> Gilles Caulier
>

Alex,

Sorry, i make a mistake with bugzilla settings to CC you. Now it's fixed. Please review this entry and comment...

Thanks in advance

Gilles Caulier

There is no OpenMP speedup for linear interpolate in LibRaw 0.15-release

I was unable to reproduce this problem (several months ago), anyway I've removed OpenMP speedup for safety.

Please re-check with current LibRaw 0.15.1

Confirmed. Artefact are not reproducible with libraw 0.15.1 if compiled with OpenMP and no optimization -O4 is used.

Gilles Caulier

(In reply to comment #8)
> Just to precise the comparison :
> 
> I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with libkdcraw =>
> artefacts.
> 
> I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with offcial
> tarball => no artefacts.
> 
> Gilles Caulier

Gilles:

sorry for this late comment: the pic ***5199.CR2 are _not_ the ones which caused my bugreporting this issue!!  

The pic I reported was "DSC_8604_XX.jpg"


Axel