Bug 305382 - editing RAW gives artefacts with Bilinear quality
Summary: editing RAW gives artefacts with Bilinear quality
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Unclassified
Component: Plugin-DImg-RAW (show other bugs)
Version: 3.0.0
Platform: Ubuntu Packages Linux
: NOR normal (vote)
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-08-18 11:29 UTC by Axel Krebs
Modified: 2013-06-01 07:52 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 3.3.0


Attachments
direct import of NEF leads to pattern/artefacts (101.28 KB, image/jpeg)
2012-08-18 11:32 UTC, Axel Krebs
Details
GIMP 2.6 imports appearantly correct (161.91 KB, image/jpeg)
2012-08-18 11:33 UTC, Axel Krebs
Details
artefacts given with OpenMP support (612.02 KB, image/jpeg)
2012-10-26 13:07 UTC, caulier.gilles
Details
libraw as SHARED patch (3.15 KB, patch)
2012-10-27 10:16 UTC, caulier.gilles
Details
original libraw 0.15.0-beta1 makefile (5.89 KB, application/octet-stream)
2012-10-27 10:17 UTC, caulier.gilles
Details
stand alone libraw tarball, based on cmake build system (339.43 KB, application/x-bzip)
2012-10-27 12:47 UTC, caulier.gilles
Details
Makefile.dist used to reproduce artifacts problem with LibRaw (5.91 KB, text/plain)
2012-10-29 09:13 UTC, caulier.gilles
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Axel Krebs 2012-08-18 11:29:05 UTC
When opening a NIKON Raw Picture (="NEF"), digiKam provides terrible visual artefacts

Reproducible: Always

Steps to Reproduce:
1. just open a recent NEF
2.
3.


Expected Results:  
image editor should be able to edit NEFs directly

please compare with pics in following mail
Comment 1 Axel Krebs 2012-08-18 11:32:09 UTC
Created attachment 73272 [details]
direct import of NEF leads to pattern/artefacts
Comment 2 Axel Krebs 2012-08-18 11:33:12 UTC
Created attachment 73273 [details]
GIMP 2.6 imports appearantly correct
Comment 3 caulier.gilles 2012-08-18 11:44:53 UTC
Which libkdcraw and libraw you use (Help/components Info for details)

Which Nikon camera you use ?

Which RAW demosaicing settings you use exactly ?

Can you share the NEF file to test here...

Gilles Caulier
Comment 4 Axel Krebs 2012-08-18 12:13:08 UTC
Hi Gilles,

thank you for quick response:

---
Components:
digiKam version 2.6.0
Exiv2 kann in JP2 speichern: Ja
Exiv2 kann in JPEG speichern: Ja
Exiv2 kann in PGF speichern: Ja
Exiv2 kann in PNG speichern: Ja
Exiv2 kann in TIFF speichern: Ja
Exiv2 unterstützt XMP-Metadaten: Ja
LibCImg: 130
LibClapack: Interne Bibliothek
LibExiv2: 0.22
LibJPEG: 80
LibJasper: 1.900.1
LibKDE: 4.9.00
LibKExiv2: 2.1.0
LibKGeoMap: 2.0.0
LibKdcraw: 2.0.1
LibLCMS: 119
LibLensFun: Externe gemeinsame Bibliothek
LibLqr: Interne Bibliothek
LibPGF: 6.11.42 - Interne Bibliothek
LibPNG: 1.2.46
LibQt: 4.8.1
LibRaw: 0.14.4
LibTIFF: LIBTIFF, Version 3.9.5 Copyright (c) 1988-1996 Sam Leffler
Copyright (c) 1991-1996 Silicon Graphics, Inc.
Marble-Funktion: 0.12.2 (stable release)
Parallelisiertes Entfernen von Mosaikmustern: Ja
Datenbanktreiber: QSQLITE
LibGphoto2: 2.4.13
LibKface: 2.0.0
LibKipi: 1.3.0
LibOpenCV: 2.3.1
Libface: 0.2
---

NIKON D3100, irmware Version:
A: 1.01
B: 1.01
L: 1.002
---
RAW demosaicing settings?
settings > raw-decoding >
behavior:
- use pre-setting 16 bit

presettings (using libraw 0.14.4):
- please see attachment

!!

---

share BEF-pic? > YES, enclosed (hope, this pic is not topo large!?)


Axel
---
Am 18.08.2012 13:44, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=305382
> 
> Gilles Caulier <caulier.gilles@gmail.com> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |caulier.gilles@gmail.com
>           Component|Image Editor                |libkdcraw
> 
> --- Comment #3 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Which libkdcraw and libraw you use (Help/components Info for details)
> 
> Which Nikon camera you use ?
> 
> Which RAW demosaicing settings you use exactly ?
> 
> Can you share the NEF file to test here...
> 
> Gilles Caulier
>
Comment 5 Marcel Wiesweg 2012-08-22 18:33:01 UTC
Apparently too large. You can send it via private mail to me.
Comment 6 Axel Krebs 2012-08-25 15:52:08 UTC
Hi Marcel:

I just returned from some holidays... please excuse me for delay...


Thank you for your offer!

- When opening DSC_8604_NEF (digiKam, Version 2.6.0, Unter KDE 4.9.00),
I experience artefacts, looking like DSC_8604XX.jpg.

Settings you'll find in screenshot DSC_8605_XX_GIMP27_b.jpg.

Other details in my bug report.

Hope you can work with it.

Please let me know, if I can help/clarify further details.



Axel

encl.

Am 22.08.2012 20:33, schrieb Marcel Wiesweg:
> https://bugs.kde.org/show_bug.cgi?id=305382
> 
> --- Comment #5 from Marcel Wiesweg <marcel.wiesweg@gmx.de> ---
> Apparently too large. You can send it via private mail to me.
>
Comment 7 caulier.gilles 2012-10-26 13:01:07 UTC
Marcel,

I can reproduce the problem with all RAW files from my test collection using Bilinear demosaicing (and some other quality settings but not all).

I talk with Alex about this problem. He ask me if i can reproduce it with official libraw tarball... I can confirm that it's not reproducible...

I written some test program to check if problem is relevant of :

1/ multithreading from digiKam core ==> no.
2/ DImg image loader ==> no.
3/ Extra compilation flags set by KDE ==> no.
4/ Check OPenMP => yes.

To reproduce the problem, you need a multicore processor, and libkdcraw/libraw must be compiled against OPenMP. If you disabled OPenMP support, it's work. There is no artefacts, but demosaicing is not parallelized.

I suspected in first a bug in Libraw. But the problem is that i cannot reproduce the problem when it's compiled/linked against OpenMP.

THE question is why ???

- I checked again and again all CMake rules from libraw. I well separated libraw compilation rules and libkdcraw into cmake scripts, just to be sure. I take a care about all definitions passed to compile libraw in official tarball, checked if i compiled libraw from libkdcraw with same dependencies than offcial libraw (lcms, rawspeed, gpl2 and gpl3 extra pack, libjpeg, libjpasper) : It's always reproducible... 

I cannot found the problem, and i don't have any idea... Help is welcome...

Gilles Caulier
Comment 8 caulier.gilles 2012-10-26 13:04:41 UTC
Just to precise the comparison :

I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with libkdcraw => artefacts.

I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with offcial tarball => no artefacts.

Gilles Caulier
Comment 9 caulier.gilles 2012-10-26 13:07:38 UTC
Created attachment 74809 [details]
artefacts given with OpenMP support
Comment 10 Marcel Wiesweg 2012-10-26 21:33:34 UTC
So essentially, the problem is that when we compile the same sources using the same compiler and linker command lines with either CMake, or whatever libraw uses, we get a different result?

Does the original libraw also compile into a static or a dynamic library? If shared: With libkdcraw, we compile into a static library. Assuming we use a shared library
ADD_LIBRARY(libraw SHARED ${libraw_LIB_SRCS})
still the same problem? Do the files differ?
Comment 11 caulier.gilles 2012-10-27 09:24:44 UTC
>So essentially, the problem is that when we compile the same sources using the same compiler and linker >command lines with either CMake, or whatever libraw uses, we get a different result?

Yes, absolutly...

Gilles Caulier
Comment 12 caulier.gilles 2012-10-27 10:16:06 UTC
Created attachment 74824 [details]
libraw as SHARED patch

Marcel,

See as attached my patch to use libraw as SHARED. libraw sample files do not link with it :

[gilles@localhost libraw]$ make
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/internal/dcraw_common.cpp.o
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/internal/dcraw_fileio.cpp.o
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/internal/demosaic_packs.cpp.o
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/src/libraw_cxx.cpp.o
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/src/libraw_c_api.cpp.o
Building CXX object extra/libkdcraw/libraw/CMakeFiles/raw.dir/src/libraw_datastream.cpp.o
Linking CXX shared library ../../../lib/libraw.so
Built target raw
Building CXX object extra/libkdcraw/libraw/CMakeFiles/4channels.dir/samples/4channels.cpp.o
Linking CXX executable 4channels
CMakeFiles/4channels.dir/samples/4channels.cpp.o: In function `main':
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:46: undefined reference to `LibRaw::LibRaw(unsigned int)'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:59: undefined reference to `LibRaw::cameraCount()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:59: undefined reference to `LibRaw::version()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:102: undefined reference to `LibRaw::open_file(char const*, long long)'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:104: undefined reference to `libraw_strerror'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:112: undefined reference to `LibRaw::unpack()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:114: undefined reference to `libraw_strerror'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:117: undefined reference to `LibRaw::raw2image()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:120: undefined reference to `LibRaw::subtract_black()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:173: undefined reference to `LibRaw::dcraw_ppm_tiff_writer(char const*)'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:174: undefined reference to `libraw_strerror'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:178: undefined reference to `LibRaw::~LibRaw()'
/mnt/devel/GIT/3.x/extra/libkdcraw/libraw/samples/4channels.cpp:178: undefined reference to `LibRaw::~LibRaw()'
collect2: ld a retourné 1 code d'état d'exécution
make[2]: *** [extra/libkdcraw/libraw/4channels] Erreur 1
make[1]: *** [extra/libkdcraw/libraw/CMakeFiles/4channels.dir/all] Erreur 2
make: *** [all] Erreur 2

[gilles@localhost libraw]$

Why ?

Gilles Caulier
Comment 13 caulier.gilles 2012-10-27 10:17:59 UTC
Created attachment 74825 [details]
original libraw 0.15.0-beta1 makefile

See also makefile that i used to compile original libraw to compare with libkdcraw version...

Gilles Caulier
Comment 14 caulier.gilles 2012-10-27 12:28:15 UTC
Marcel,

I tried another way : compile libraw source code directly with all sample files, instead to use a STATIC or a SHARED libraw :

# -- Compile LibRaw library --------------------------------------------------------------------------------

# Flag to include demosaic pack GPL2
ADD_DEFINITIONS(-DLIBRAW_DEMOSAIC_PACK_GPL2)

# Flag to include demosaic pack GPL3
ADD_DEFINITIONS(-DLIBRAW_DEMOSAIC_PACK_GPL3)

# Flag to debug LibRaw
ADD_DEFINITIONS(-DDCRAW_VERBOSE)

# Flag used by default into LibRaw to not use dedicated external thread
ADD_DEFINITIONS(-DLIBRAW_NOTHREADS)

# Flag to export library symbols
ADD_DEFINITIONS(-DLIBRAW_BUILDLIB)


INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}/
                    ${CMAKE_CURRENT_SOURCE_DIR}/demosaic-pack-GPL2
                    ${CMAKE_CURRENT_SOURCE_DIR}/demosaic-pack-GPL3
                   )

# Do not compile LibRaw with KDE FINAL mode.
KDE4_NO_ENABLE_FINAL(raw)

SET(raw_LIB_SRCS ${CMAKE_CURRENT_SOURCE_DIR}/internal/dcraw_common.cpp
                 ${CMAKE_CURRENT_SOURCE_DIR}/internal/dcraw_fileio.cpp
                 ${CMAKE_CURRENT_SOURCE_DIR}/internal/demosaic_packs.cpp
                 ${CMAKE_CURRENT_SOURCE_DIR}/src/libraw_cxx.cpp
                 ${CMAKE_CURRENT_SOURCE_DIR}/src/libraw_c_api.cpp
                 ${CMAKE_CURRENT_SOURCE_DIR}/src/libraw_datastream.cpp
   )

# Disable compilation warnings from LibRaw. Just to be clear on the console.
# Adjust flag for static lib and 64 bits computers using -fPIC for GCC compiler (B.K.O: #269903)
FOREACH(_curentfile ${raw_LIB_SRCS})
    SET_SOURCE_FILES_PROPERTIES(${_curentfile} PROPERTIES COMPILE_FLAGS "-w")
ENDFOREACH(_curentfile ${raw_LIB_SRCS})

# -- Compile LibRaw Samples --------------------------------------------------------------------------------

# add a small macro so that this is a bit cleaner
MACRO(LIBRAW_BUILD_SAMPLES)

    SET(_filename ${ARGV0})
    STRING(REPLACE "." ";" _temp ${_filename})
    LIST(GET _temp 0 _target)

    SET(${_target}_SRCS ${CMAKE_CURRENT_SOURCE_DIR}/samples/${_filename})

    SET_SOURCE_FILES_PROPERTIES(${${_target}_SRCS} PROPERTIES COMPILE_FLAGS -w)

    ADD_EXECUTABLE(${_target} ${raw_LIB_SRCS} ${${_target}_SRCS})

    TARGET_LINK_LIBRARIES(${_target} ${OPENMP_LDFLAGS} ${MATH_LIBRARY} ${LCMS2_LIBRARIES})

    IF(RAWSPEED_SUPPORT_CAN_BE_COMPILED)
        TARGET_LINK_LIBRARIES(${_target} librawspeed)
    ENDIF()

    IF(JPEG_FOUND)
        TARGET_LINK_LIBRARIES(${_target} ${JPEG_LIBRARY})
    ENDIF()

    IF(JASPER_FOUND)
        TARGET_LINK_LIBRARIES(${_target} ${JASPER_LIBRARIES})
    ENDIF()

    IF(WIN32)
        TARGET_LINK_LIBRARIES(${_target} ws2_32)
    ENDIF(WIN32)

ENDMACRO(LIBRAW_BUILD_SAMPLES)

LIBRAW_BUILD_SAMPLES(simple_dcraw.cpp)
LIBRAW_BUILD_SAMPLES(mem_image.cpp)
LIBRAW_BUILD_SAMPLES(dcraw_emu.cpp)
LIBRAW_BUILD_SAMPLES(4channels.cpp)
LIBRAW_BUILD_SAMPLES(unprocessed_raw.cpp)
LIBRAW_BUILD_SAMPLES(raw-identify.cpp)
LIBRAW_BUILD_SAMPLES(multirender_test.cpp)
LIBRAW_BUILD_SAMPLES(postprocessing_benchmark.cpp)
#IF(WIN32)
#    LIBRAW_BUILD_SAMPLES(half_mt_win32.c)
#ELSE(WIN32)
#    LIBRAW_BUILD_SAMPLES(dcraw_half.c)
#    LIBRAW_BUILD_SAMPLES(half_mt.c)
#ENDIF(WIN32)

No chance. The result is exactly the same. output PPM image is always corrupted. This proof that linking with STATIC lib is not the problem here. Right ?
Comment 15 caulier.gilles 2012-10-27 12:47:55 UTC
Created attachment 74829 [details]
stand alone libraw tarball, based on cmake build system

To be sure sure sure, especially to see if host KDE env is the problem, i created a stand alone tarball of libraw 0.15.0-beta1 with cmake compilation script, which do not create a SHARED or STATIC lib.

The problem still the same....

So excepted a bug in libraw, i don't have any more idea to hack....

Alex ?

Gilles Caulier
Comment 16 Marcel Wiesweg 2012-10-27 14:04:57 UTC
So the standalone tarball with its dcraw_emu also produces the problem, or not? If it does, it's a good isolated test case to start from.
Comment 17 Axel Krebs 2012-10-27 14:15:22 UTC
Hello Gilles,

Thank you for all these trouble shootings. I follow all these bug
remarks with huge interest. But I am still wondering, whom do you talk
to, as you address to Marcel (whose emailaddress I don't see) and to me,
you don't talk to appearently.

As I mentioned before, I can not contribute in software development, as
much as I liked too. I do not even understand the last steps you were
approaching the problem.

The only thing which comes into my mind (as I am European Quality
Manager and this discipline should offer some options even to software
development):

1.) I would try rigous standardization, the less differences the better.
American often say "keep it simple".

2.) I believe KDE is highly developped, but risky for complexity
reasons. Every engineer could build a car, basicly. In reality, you need
hundreds. I suspect KDEs technologies to still creep on a base level,
somehow.

3.) One of the reasons of the tremendeous success of MACS is based on
clear programming rules for developpers, if they want to be accpted by
Masintoshes.
- Why not use some of these rules?
- Why not defining borders within software has to be if incorporated
into the digiKam project? I could imagine, that undoubtful definition
(as tif-format, e.g.) could help to stabilize digiKam overall. Use, what
is defined, reject what is undefined.

Sorry, some thoughts of a non-programmer...

Coming back to your question mark. Please let me know, if I can help you
and the digiKam project.


Axel

(P.S.: "Axel" _not_ "Alex")

Am 27.10.2012 14:47, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=305382
> 
> --- Comment #15 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Created attachment 74829 [details]
>   --> https://bugs.kde.org/attachment.cgi?id=74829&action=edit
> stand alone libraw tarball, based on cmake build system
> 
> To be sure sure sure, especially to see if host KDE env is the problem, i
> created a stand alone tarball of libraw 0.15.0-beta1 with cmake compilation
> script, which do not create a SHARED or STATIC lib.
> 
> The problem still the same....
> 
> So excepted a bug in libraw, i don't have any more idea to hack....
> 
> Alex ?
> 
> Gilles Caulier
>
Comment 18 caulier.gilles 2012-10-27 15:50:13 UTC
Marcel,

Yes, problem is reproducible with stand alone version.

Axel,

I talk to Alex Tutubalin, who is in CC in this file. He is libraw author...

Gilles Caulier
Comment 19 caulier.gilles 2012-10-29 09:13:17 UTC
Created attachment 74861 [details]
Makefile.dist used to reproduce artifacts problem with LibRaw

Alex,

I found the problem... It's in libraw source code. I'm sure. I can reproduce the artifacts problem with official Libraw 0.15.0-beta1 tarball.

The problem is about "-O4" gcc compilation optimization option. If you remove it from offcial MakeFile.dist, the artifacts problem is reproducible with "dcraw_emu -q0 myrawfile.NEF" (NEF is just and example, it's also reproducible with CR2 files...)

Gilles Caulier
Comment 20 caulier.gilles 2012-10-29 09:15:40 UTC
Alex,

I can confirm also, that with my stand alone libraw compiled with CMake, if i use "-O4" optimization, as with official libraw tarball, artifacts disappears...

It's definitively a libraw problem....

Gilles Caulier
Comment 21 caulier.gilles 2012-10-29 11:03:23 UTC
Git commit ca11950fa5926ccfb4438d8b9cbd072370be3f0b by Gilles Caulier.
Committed on 29/10/2012 at 11:55.
Pushed by cgilles into branch 'master'.

- Use -O4 compilation option to optimize speed with dosaicing process.

- Fix artifacts produced by libraw when OpenMP is used. This problem still present in libraw and must be fixed as DOWNSTREAM
  Please confirm if probelm still reproducible on your computer with this fix.
Related: bug 308489

M  +2    -1    libraw/CMakeLists.txt

http://commits.kde.org/libkdcraw/ca11950fa5926ccfb4438d8b9cbd072370be3f0b
Comment 22 Axel Krebs 2012-10-29 18:39:28 UTC
Gilles + Marcel:

This sounds great. Impressive work. Thank you guys!

Please let me know, if I could help somehow.


Axel

(please, _NOT_ "Alex")
---
Am 29.10.2012 10:13, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=305382
> 
> --- Comment #19 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Created attachment 74861 [details]
>   --> https://bugs.kde.org/attachment.cgi?id=74861&action=edit
> Makefile.dist used to reproduce artifacts problem with LibRaw
> 
> Alex,
> 
> I found the problem... It's in libraw source code. I'm sure. I can reproduce
> the artifacts problem with official Libraw 0.15.0-beta1 tarball.
> 
> The problem is about "-O4" gcc compilation optimization option. If you remove
> it from offcial MakeFile.dist, the artifacts problem is reproducible with
> "dcraw_emu -q0 myrawfile.NEF" (NEF is just and example, it's also reproducible
> with CR2 files...)
> 
> Gilles Caulier
>
Comment 23 caulier.gilles 2013-05-25 20:58:44 UTC
Alex,

Sorry, i make a mistake with bugzilla settings to CC you. Now it's fixed. Please review this entry and comment...

Thanks in advance

Gilles Caulier
Comment 24 Alex Tutubalin 2013-05-26 05:21:36 UTC
There is no OpenMP speedup for linear interpolate in LibRaw 0.15-release

I was unable to reproduce this problem (several months ago), anyway I've removed OpenMP speedup for safety.

Please re-check with current LibRaw 0.15.1
Comment 25 caulier.gilles 2013-05-26 07:13:54 UTC
Confirmed. Artefact are not reproducible with libraw 0.15.1 if compiled with OpenMP and no optimization -O4 is used.

Gilles Caulier
Comment 26 Axel Krebs 2013-06-01 07:52:34 UTC
(In reply to comment #8)
> Just to precise the comparison :
> 
> I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with libkdcraw =>
> artefacts.
> 
> I run "./dcraw_emu -q 0 5D3_5199.CR2" from libraw compiled with offcial
> tarball => no artefacts.
> 
> Gilles Caulier

Gilles:

sorry for this late comment: the pic ***5199.CR2 are _not_ the ones which caused my bugreporting this issue!!  

The pic I reported was "DSC_8604_XX.jpg"


Axel