Created attachment 101101 [details]
Screenshot of digikam album view with blurred orf thumbnail

Can you share an ORF sample file to test here with digiKam 5.x ? Please use cloud service to share file.

Gilles Caulier

Hi Gilles,

I have put one sample orf, one rw2 sample and the screenshot put on GMX cloud:
https://cloud.gmx.net/ngcloud/external?guestToken=IIaKtrn9RvGsY3IGwMMumQ&loginName=guenther.erhard@gmx.de

BR
Günther

No problem here with digiKam 5.2.0 :

https://www.flickr.com/photos/digikam/29078699804/in/dateposted-public/

Did you enable Color Managed View with a wrong screen color profile ?

Gilles Caulier

Created attachment 101110 [details]
thumbnail.png

The extracted thumbnail from ORF image with libraw has only 160x120 pixels. The thumbnail from RW2 image has 1920x1440 pixels. Good to see at maximum thumbnail size view.

Maik

Created attachment 101112 [details]
Thumbnail extracted from ORF by dcraw

Hi Gilles and Maik,

No, CMS is correctly installed and setup.

But I'm confused that libraw extracts such a small preview. If I use dcraw -x GME15390.ORF then I got an embedded jpg with 2400x3000 which is perfectly sharp. Maybe there is something wrong with libraw.

Just a thought: maybe the ORF raw file has several previews in different sizes embedded and libraw pcks the first or smallest one?

Guenther

One more thought: I use Digikam since version 0.8 and had never experienced this in all versions (did regularly updates from svn). Also after it was possible to have the thumbnails greater than 256px the pictures were always sharp. And digikam always used the embedded previews (which were already at 1600x1200 for my old Olympus E-510 raw) and not the halfsize raw because otherwise pictures shot in b/w would have a color thumbnail in album view.

So I guess it must be some change in libraw or a related lib.

Guenther

> But I'm confused that libraw extracts such a small preview. If I use dcraw
> -x GME15390.ORF then I got an embedded jpg with 2400x3000 which is perfectly
Just saw a typo: the command is dcraw -e FILENAME

Yes, there are 2 preview images in different sizes available:

maik@linux-tpgn:~> exiv2 -pp GME15390.ORF
Preview 1: image/jpeg, 160x120 pixels, 6946 bytes
Preview 2: image/jpeg, 3200x2400 pixels, 1004415 bytes

I have searched another RAW image from this camera on the Web. At this file libraw used the large preview for the thumbnail. However, the firmware version from your camera is greater. Strange.

Maik

Mi Maik,

That is maybe a glue. I use the experimental feature write meta data to raw files. 
To compare it I have made a new picture and imported it into digikam and did nothing else. But the result was the same.

To give you more files to test I have uploaded to the GMX cloud some more raw files which all show only the small embedded preview in digikam:
E-M5_no-Meta-write.ORF -> the virgin raw file mentioned above
E-M5_old_firmware_Meta-written.ORF -> one of the first pictures I've made with this camera. Firmware is the original one. Meta was written.
E-510_Meta-written.orf -> a file from my old E-510 camera with meta data written
E-420_Meta-written.ORF -> a file from my wifes old E-420 camera with meta data written

HTH
Guenther

And all these files showed the correct preview with the same digikam version (e.g. 4.13) under Ubuntu 14.04.

Warning.

In digiKam core, the preview and thumbnails extraction can be processed by 2 ways for RAW files.

1/ Exiv2 API
2/ Libraw API

Depending of Exiv2 version used by digiKam the step 1/ can extract the thumbnail instead the preview and vis versa (if i remember).

Note : raw are big data container, mostly based on TIFF/EP. one tiff tag can store thumb, one others can store more than one preview images in different sizes. In fact RAW are big mess, and sometime this can give strange behavior.

Gilles Caulier

In my opinion the problem is the libraw version. Exiv2 is the same version (0.25) on both Ubuntu 14.04 and 16.04.

I have now installed libraw 0.16.2 instead of 0.17.1 in my Ubuntu 16.04 and recompiled digikam 4.13 for a test and it is working! After thumbnails recreating they are sharp. So something has changed in libraw 0.17.

Guenther

Are you sure that problem is not fixed with digiKam 5.1.0 which include libraw 0.18 ?

If no, please report this problem to Libraw team, through github project page.

https://github.com/LibRaw/LibRaw/issues

Gilles Caulier

Just compiled 5.3 from git with libraw 0.18 and the error is the same. So it is libraw.
I'll report to the libraw team.

Thanks,
Guenther

The libraw team has tested it and thinks that libraw is o.k.:
https://github.com/LibRaw/LibRaw/issues/77

So what is broken?

Guenther

With digiKam 5.2.0, there is no problem here to preview your ORF with good quality :

https://www.flickr.com/photos/digikam/29717186702/in/dateposted-public/

Check your Preview settings, as mine.

Gilles Caulier

Gilles,

previewe is ok, only the thumbnail. Use maximum thumbnail size.

Maik

There is feedback from libraw project:
"In Libraw 0.17 and newer we added additional data type check.
Original (from camera) images passes it, while digikam-imported are not.
After digikam procesing (on import), type field for metadata tag 0x2020 is altered and set to 4 instead of original value (13)."

It seems that this has an influence on the cooperation between dk and libraw.

I work currently with dk 4.13 compiled to use libraw 0.16 (my systemwide lib). 
Is there a way to compile dk5.x from git or tarball with the systems libraw? 
I have tried to replace the libraw 0.18 src in core/libs/rawengine with my 0.16 src, but it did not work well.

If i understand the thread in libraw forum, the Thumbnail Thread interface in digiKam do not take the largest preview to render the thumb in icon view.

Typically, ORF is based on TIFF/EP, and the first thumbnail returned in a small one from Exif thumbnail tags.

We need to check this image size to see if it enough largest accordingly with thumbnail max size set in config dialog (typically 256 or 512).

If the condition is false, we must take the preview JPEG embeddedin RAW files.

The code relevant is in this file :

https://quickgit.kde.org/?p=digikam.git&a=blob&f=libs%2Fthreadimageio%2Fthumbnailcreator.cpp

The main wrapper is in ThumbnailCreator::createThumbnail(), which will call ThumbnailCreator::loadImagePreview(). This method will use Exiv2 API to get preview. I think the problem is here with ORF because it take Exif thumb, not preview. We must check if image size taken with Exiv2 is enough large, if not QImage returned must be null.

Later in ThumbnailCreator::createThumbnail(), we will try with libraw through DRawDecoder::loadEmbeddedPreview(). This must do the stuff properly.

Gilles

I can reproduce it. If the option "If possible write metadata to RAW files (experimental)" is enabled Makernotes be changed and the preview image is unsharp after reload (F5). A simple change (rating) is sufficient.

Maik

No problem with CR2, RW2 or NEF files. Guenther, the option "If possible write Metadata to RAW files (experimental)" is not good for ORF files! I see the problem now in libexiv2.

Maik

Git commit 7be20c293714a0f325f956e49ec58c54f3161dfc by Maik Qualmann.
Committed on 28/09/2016 at 18:07.
Pushed by mqualmann into branch 'master'.

only perform automatic rotation of JPEG images in the import tool

M  +1    -1    utilities/importui/main/importui.cpp

http://commits.kde.org/digikam/7be20c293714a0f325f956e49ec58c54f3161dfc

Hello Maik,

Is there any progress here? I've compiled a current version from git, but the problem is still there.
You mention libexiv2 is probably the root cause when writing to raw files. Is there anything I can check?

Thanks
Guenther

Have you the option "If possible write Metadata to RAW files (experimental)" enabled? When yes, please disable. Import new images from the camera. Thumbnail now ok?

Maik

Yes, the thumbnail after import from camera is now o.k. - regardless of the option setting.
The problem starts when meta data is written to the raw, e.g. adding a tag or correcting time.

I've done some tests: 
- change of time in dk53 with the batch tool on a virgin raw file (with thumbnail o.k.)
- after change of time thumbnail get blurry
- change of time with exiftool on this file (exiftool -createdate+=00:00:01 FILENAME.ORF)
- exiftool throws out a warning:
  "Invalid time string (2016:10:06T15:14:12) when shifting CreateDate - GME15826_1.ORF"
  and rewrites the file with the new time 
- after that thumbnail is o.k. again
- doing the time change with exiftool on a virgin raw file gives no warning and the thumbnail is still o.k.

Now I tried it with exiv2: exiv2 -v -Q d -a 01:01:00 ad FILENAME.ORF:
- changing time on a virgin raw file leaves the thumbnail o.k.
- changing time on the dk53 time modified file throws no warning and the thumbnail is still blurry

So I guess you are right there is something with libexiv2 as exiftool works correctly.

My conclusions: 
- Exiv2 seems not so picky about errors in metadata and copies/writes them again on file changes
- If the file metadata is correct exiv2 does not break the file structure when writing
- DK works correct when metadata structure is not broken
- Writing metadata with DK (using libexiv2 I assume) into orf raw breaks the structure

So I think there is either something wrong in DKs interface to libexiv2 or libexiv2 works different to exiv2.

HTH
Guenther

Some more warnings exiftool gives on my test files:
"[minor] Ignored empty rdf:Bag list for mwg-rs:RegionList - E-M5_Meta-write.ORF"
"[minor] Fixed incorrect URI for xmlns:MicrosoftPhoto - GME15825.ORF"

Definitively, the problem is not in digiKam. It's in Exiv2 shared lib.

As it's annotated in Setup/metadata dialog, writting into RAW file still experimental. It's recommended to use XMP sidecar instead

Gilles Caulier

Git commit 2c67e6ffd23d8e7fda2704e875378b7f2da20905 by Gilles Caulier.
Committed on 01/11/2016 at 17:00.
Pushed by cgilles into branch 'master'.

Internal Libraw updated to 0.18-beta1 with 78 cameras added,
floating point DNG support, decode exotic DNG formats e.g. 8-bit DNG,
and more metadata parsed while decoding as white balance presets,
DNG colordata, vendor specific metadata.

See Libraw announcement for details : http://www.libraw.org/news/libraw-0-18-beta1
Related: bug 367640, bug 328321, bug 257737, bug 347010

FIXED-IN: 5.3.0

M  +5    -0    NEWS
M  +1    -5    libs/rawengine/libraw/COPYRIGHT
M  +230  -235  libs/rawengine/libraw/Changelog.txt
M  +0    -0    libs/rawengine/libraw/LICENSE.CDDL
M  +0    -0    libs/rawengine/libraw/LICENSE.LGPL
M  +4    -5    libs/rawengine/libraw/README
A  +34   -0    libs/rawengine/libraw/README.DNGSDK.txt
M  +1    -4    libs/rawengine/libraw/internal/aahd_demosaic.cpp
M  +1682 -1038 libs/rawengine/libraw/internal/dcraw_common.cpp
M  +1    -4    libs/rawengine/libraw/internal/dcraw_fileio.cpp
M  +3    -6    libs/rawengine/libraw/internal/defines.h
M  +1    -10   libs/rawengine/libraw/internal/demosaic_packs.cpp
M  +1    -4    libs/rawengine/libraw/internal/dht_demosaic.cpp
M  +16   -6    libs/rawengine/libraw/internal/libraw_internal_funcs.h
M  +1080 -551  libs/rawengine/libraw/internal/libraw_x3f.cpp
M  +1    -3    libs/rawengine/libraw/internal/var_defines.h
M  +1    -10   libs/rawengine/libraw/internal/wf_filtering.cpp
M  +29   -21   libs/rawengine/libraw/libraw/libraw.h
M  +1    -4    libs/rawengine/libraw/libraw/libraw_alloc.h
M  +10   -32   libs/rawengine/libraw/libraw/libraw_const.h
M  +5    -8    libs/rawengine/libraw/libraw/libraw_datastream.h
M  +2    -6    libs/rawengine/libraw/libraw/libraw_internal.h
M  +67   -28   libs/rawengine/libraw/libraw/libraw_types.h
M  +3    -6    libs/rawengine/libraw/libraw/libraw_version.h
M  +1    -5    libs/rawengine/libraw/samples/4channels.cpp
M  +51   -10   libs/rawengine/libraw/samples/dcraw_emu.cpp
M  +1    -5    libs/rawengine/libraw/samples/dcraw_half.c
M  +1    -4    libs/rawengine/libraw/samples/half_mt.c
M  +1    -4    libs/rawengine/libraw/samples/half_mt_win32.c
M  +1    -4    libs/rawengine/libraw/samples/mem_image.cpp
M  +1    -5    libs/rawengine/libraw/samples/multirender_test.cpp
M  +1    -6    libs/rawengine/libraw/samples/postprocessing_benchmark.cpp
M  +124  -20   libs/rawengine/libraw/samples/raw-identify.cpp
M  +16   -5    libs/rawengine/libraw/samples/simple_dcraw.cpp
M  +2    -5    libs/rawengine/libraw/samples/unprocessed_raw.cpp
M  +38   -3    libs/rawengine/libraw/src/libraw_c_api.cpp
M  +458  -100  libs/rawengine/libraw/src/libraw_cxx.cpp
M  +1    -4    libs/rawengine/libraw/src/libraw_datastream.cpp
A  +811  -0    libs/rawengine/libraw/src/libraw_xtrans_compressed.cpp     [License: UNKNOWN]  *

The files marked with a * at the end have a non valid license. Please read: http://techbase.kde.org/Policies/Licensing_Policy and use the headers which are listed at that page.


http://commits.kde.org/digikam/2c67e6ffd23d8e7fda2704e875378b7f2da20905