Bug 491888 - DNG files misread as TIFF
Summary: DNG files misread as TIFF
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Engine (show other bugs)
Version: 8.4.0
Platform: Microsoft Windows Microsoft Windows
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-08-19 07:44 UTC by Bertil Rolf
Modified: 2024-08-30 09:44 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 8.5.0
Sentry Crash Report:


Attachments
DNG file identified as such by ExifTool but falsely marked as TIFF by DigiKam (3.61 MB, image/DNG)
2024-08-19 08:06 UTC, Bertil Rolf
Details
attachment-761745-0.html (7.64 KB, text/html)
2024-08-30 08:00 UTC, Bertil Rolf
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Bertil Rolf 2024-08-19 07:44:20 UTC
***
If you're not sure this is actually a bug, instead post about it at https://discuss.kde.org

If you're reporting a crash, attach a backtrace with debug symbols; see https://community.kde.org/Guidelines_and_HOWTOs/Debugging/How_to_create_useful_crash_reports
***

SUMMARY


STEPS TO REPRODUCE
1. 
2. 
3. 

OBSERVED RESULT


EXPECTED RESULT


SOFTWARE/OS VERSIONS
Windows: 
macOS: 
(available in the Info Center app, or by running `kinfo` in a terminal window)
Linux/KDE Plasma: 
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

ADDITIONAL INFORMATION
I am running DigiKam on W11. Using Adobe Lightroom, I have lots of DNG-files, written there. I want to import them into DigiKam. They are imported, all right, but almost (!) all of them cannot be shown. "Failed to load image", DK reports. In the ExifTool window, they are correctly shown as DNG files under the heading File. But over those DNG photos, DK has pasted TIFF (which is false).// 
Now, a few DNG files are imported and correctly marked DNG. They can be opened in DK.
--
Hope this helps. I think DK handles JPG files and RW2 files neatly and would like to try it for handling the large collection of DNG photos I have.
Comment 1 Maik Qualmann 2024-08-19 07:55:31 UTC
Please provide a DNG sample image that cannot be loaded via private mail or file upload service.

Maik
Comment 2 Bertil Rolf 2024-08-19 08:06:18 UTC
Created attachment 172745 [details]
DNG file identified as such by ExifTool but falsely marked as TIFF by DigiKam

OK, here's a sample.
Comment 3 Maik Qualmann 2024-08-19 08:41:09 UTC
LibRaw cannot load the file, the reason is currently unknown. A TIFF file is then recognized by the magic file bytes detection.

digikam.rawengine: LibRaw: failed to run open_file:  Unsupported file format or not RAW file

Maik
Comment 4 Maik Qualmann 2024-08-20 05:58:04 UTC
From what I have debugged so far, LibRaw does not find a DNG image in the file. It is logical given the file size, which is much too small for a DNG. The file contains a thumbnail and 2 preview images. The largest preview image is currently not loading because QImage rejects it due to unsupported "PhotometricInterpretation=34892".
I would describe the DNG file as "broken".

digikam should at least load the larger preview image, like the Gwenview image viewer does under Linux. By the way, XnView only loads the small thumbnail.

Maik
Comment 5 Bertil Rolf 2024-08-20 06:42:35 UTC
Hello.

The KDE site has a size restriction of 4MB for attachments. So I managed 
to select a DNG file that was small. But lots of my DNG files are marked 
as TIFF, even though they have a size of about 10 MB.

I did find some DNG files that were correctly labelled as DNG files by 
DigiKam. Typically, they are very large, about 20 MB.

--

Small DNG files can arise in Adobe Lightroom 5 in two ways:

1) If you import a JPG and then convert it to a DNG, the result will 
typically be small, little bigger than the original JPG. It has 
advantages to store all files as DNG, even if they originate as JPG.

2) In Lightroom, you can convert all files to DNG using "Lossy 
compression". The size is thereby reduced from typically 20-25 MB to 
6-10 MB. On a screen, you lose almost no quality, for this would be 
visible (at most) in a large print.

--

Would it help if I bundled some DNG photo files, some falsely marked as 
TIFF, some correctly marked as DNG? But the package would be very large, 
so pse tell me how.

Best regards

Bertil Rolf

Baden bei Wien




On 2024-08-20 07:58, Maik Qualmann wrote:
> https://bugs.kde.org/show_bug.cgi?id=491888
>
> --- Comment #4 from Maik Qualmann <metzpinguin@gmail.com> ---
>  From what I have debugged so far, LibRaw does not find a DNG image in the file.
> It is logical given the file size, which is much too small for a DNG. The file
> contains a thumbnail and 2 preview images. The largest preview image is
> currently not loading because QImage rejects it due to unsupported
> "PhotometricInterpretation=34892".
> I would describe the DNG file as "broken".
>
> digikam should at least load the larger preview image, like the Gwenview image
> viewer does under Linux. By the way, XnView only loads the small thumbnail.
>
> Maik
>
Comment 6 caulier.gilles 2024-08-20 06:56:45 UTC
Please use a cloud storage as google drive for example to share a tarball with your DNG files.
Comment 7 Bertil Rolf 2024-08-20 08:18:19 UTC
Hello,

Here's the tarball.

https://drive.google.com/file/d/1m7rqXguiC6QL5NwiNxlFNLxNV1D9LcTN/view?usp=drive_link

The comment file explains the content.

Best regards

Bertil Rolf

Baden bei Wien



On 2024-08-20 08:56, bugzilla_noreply@kde.org wrote:
> https://bugs.kde.org/show_bug.cgi?id=491888
>
> caulier.gilles@gmail.com changed:
>
>             What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                   CC|                            |caulier.gilles@gmail.com
>
> --- Comment #6 from caulier.gilles@gmail.com ---
> Please use a cloud storage as google drive for example to share a tarball with
> your DNG files.
>
Comment 8 Maik Qualmann 2024-08-21 10:42:42 UTC
@Gilles, the cause is the LibJpeg we use. In order for LibRaw to support the Adobe Lossy DNG format, a Libjpeg > 6+ is required. I had also compiled my digiKam with libJpeg 62, now after switching to LibJpeg 80, the DNG files work without problems. Can we switch to this LibJpeg version in the bundle?

Maik
Comment 9 caulier.gilles 2024-08-21 10:46:24 UTC
This affect the AppImage and the Windows version ?

For the AppImage no problem. 

For the Windows, VCPKG provide libjpeg-turbo version 3:

https://vcpkg.io/en/package/libjpeg-turbo

Gilles
Comment 10 Maik Qualmann 2024-08-21 10:59:56 UTC
If the AppImage is also compiled with libJpeg 62, it is also affected. I can only test it this evening. LibJpeg 80 is already part of libjpeg-turbo, I think.

Maik
Comment 11 Maik Qualmann 2024-08-22 18:43:30 UTC
The AppImage works with Adobe Lossy DNG because it also uses libJpeg8(0). I don't really understand the libJpeg versions, but when I look at the openSUSE description, it should be based on the almost current libjpeg-turbo-3.0.2.

Maik
Comment 12 caulier.gilles 2024-08-22 21:31:23 UTC
the legacy libjpeg has may API release. 6, 7, 8, and 9 if i remember.

http://ijg.org/

libjpeg-turbo is not libjpeg, but a modern implementation using acceleration features and a mode to emulate the libjpeg API at configuration time.
this lib must be configured at least with libjpeg API version 8 support, else...

https://github.com/libjpeg-turbo/libjpeg-turbo?tab=readme-ov-file#using-libjpeg-turbo

Gilles
Comment 13 caulier.gilles 2024-08-22 21:35:01 UTC
Maik,

Look like the script to install libjpeg turbo in the bundles (if necessary) already compile with libjoeg API version 8 :

https://invent.kde.org/graphics/digikam/-/blob/master/project/bundles/3rdparty/ext_jpeg/CMakeLists.txt?ref_type=heads#L18

This script is not used since a while as ubuntu, macports, and vcpkg already provide this libjpeg-turbo with the right libjpeg emulation.

Gilles
Comment 14 Maik Qualmann 2024-08-23 06:28:03 UTC
Our current vcpkg Windows uses libJpeg 6 at the moment.

Maik
Comment 15 caulier.gilles 2024-08-23 07:03:49 UTC
Ah, i will need to change it well...

Gilles
Comment 16 Maik Qualmann 2024-08-23 18:47:09 UTC
Git commit 90787314a7db74bdb8fde340411b4f2104860fb1 by Maik Qualmann.
Committed on 23/08/2024 at 18:46.
Pushed by mqualmann into branch 'master'.

activate libjpeg 8 for vcpkg Windows bundle

M  +1    -1    project/bundles/vcpkg/01-build-vcpkg.sh

https://invent.kde.org/graphics/digikam/-/commit/90787314a7db74bdb8fde340411b4f2104860fb1
Comment 17 caulier.gilles 2024-08-26 08:48:43 UTC
Maik,

I will try to rebuild the Windows VM asap. I hope that libjpeg-turb option only touch private implementations and not the public ones, else whole VCPKG must be recompiled.

Gilles
Comment 18 Maik Qualmann 2024-08-27 06:35:57 UTC
I could imagine that some things will be recompiled...

Maik
Comment 19 caulier.gilles 2024-08-28 03:19:56 UTC
For Macports, libjpeg 9f is installed. So the problem do not exist with DNG.

https://ports.macports.org/port/jpeg/

Gilles
Comment 20 caulier.gilles 2024-08-28 08:16:08 UTC
Git commit 8e8b39bcef2d1959f0e8c3ff4f629529f243ee2c by Gilles Caulier.
Committed on 28/08/2024 at 08:13.
Pushed by cgilles into branch 'master'.

set libjpeg version dependency to 8.0 or later for DNG support

M  +1    -1    Mainpage.dox
M  +1    -1    core/cmake/rules/RulesLibJpeg.cmake

https://invent.kde.org/graphics/digikam/-/commit/8e8b39bcef2d1959f0e8c3ff4f629529f243ee2c
Comment 21 caulier.gilles 2024-08-28 09:45:00 UTC
Maik, 

VCPKG is well done and warm about libjpeg changes :

Computing installation plan...
The following packages will be rebuilt:
  * ffmpeg[all,all-gpl,all-nonfree,amf,aom,ass,avcodec,avdevice,avfilter,avformat,avisynthplus,bzip2,core,dav1d,fdk-aac,ffmpeg,fontconfig,freetype,fribidi,gpl,iconv,ilbc,lzma,modplug,mp3lame,nonfree,nvcodec,opencl,opengl,openh264,openjpeg,openmpt,openssl,opus,postproc,qsv,sdl2,snappy,soxr,speex,srt,ssh,swresample,swscale,tesseract,theora,version3,vorbis,vpx,webp,x264,x265,xml2,zlib]:x64-windows@6.1.1#9
  * jasper[core,default-features,opengl]:x64-windows@4.2.1
  * leptonica:x64-windows@1.84.1#1
  * libavif[aom,core]:x64-windows@1.0.4#1
    libjpeg-turbo[core,jpeg8]:x64-windows@3.0.2#1
  * libyuv:x64-windows@1857
  * opencv[core,default-features]:x64-windows@4.8.0#1
  * opencv4[core,default-features,dnn,jpeg,png,quirc,tiff,webp]:x64-windows@4.8.0#20
  * qt5compat[big-codecs,codecs,core,default-features,qml,textcodec]:x64-windows@6.7.2
  * qtbase[brotli,concurrent,core,dbus,default-features,doubleconversion,freetype,gui,harfbuzz,icu,jpeg,network,opengl,openssl,pcre2,png,sql,sql-mysql,sql-psql,sql-sqlite,testlib,thread,widgets,zstd]:x64-windows@6.7.2
  * qtdeclarative:x64-windows@6.7.2
  * qtimageformats[core,default-features,jasper,tiff,webp]:x64-windows@6.7.2
  * qtlanguageserver:x64-windows@6.7.2
  * qtmultimedia[core,ffmpeg,widgets]:x64-windows@6.7.2
  * qtnetworkauth:x64-windows@6.7.2
  * qtpositioning:x64-windows@6.7.2
  * qtscxml:x64-windows@6.7.2
  * qtserialport:x64-windows@6.7.2
  * qtshadertools:x64-windows@6.7.2
  * qtspeech:x64-windows@6.7.2
  * qtsvg:x64-windows@6.7.2
  * qttools[core,designer,linguist]:x64-windows@6.7.2
  * qttranslations[core,default-features]:x64-windows@6.7.2
  * qtwebchannel:x64-windows@6.7.2
  * qtwebengine[core,default-features]:x64-windows@6.7.2
  * tesseract:x64-windows@5.3.4
  * tiff[core,jpeg,lzma,zip]:x64-windows@4.6.0#4
Additional packages (*) will be modified to complete this operation.
warning: If you are sure you want to rebuild the above packages, run the command with the --recurse option.
FAILED COMMAND: $VCPKG_DIR/vcpkg ${VCPKG_COMMON_OPTIONS[@]} install libjpeg-turbo[jpeg8]

So, qtwebengine will explode the compilation times as usual. So be patient (:=)))

Gilles
Comment 22 caulier.gilles 2024-08-28 15:11:15 UTC
Maik, 

Both Qt5 and Qt6 vcpkg VM are up to date and new 8.5.0 Windows installer are ready to test.

I'm not sure if the Qt recompilation was fine with libjpeg. If not as crash occurs, well i will need to recompile whole from scratch.

Gilles
Comment 23 Maik Qualmann 2024-08-28 16:09:47 UTC
Hmm, there are no updated bundles on the server yet.

Maik
Comment 24 caulier.gilles 2024-08-29 06:54:57 UTC
Exact... compilation is broken. Something become wrong on Qt install now:

-- Could NOT find Qt6WebEngineWidgets (missing: Qt6WebEngineWidgets_DIR)
CMake Error at C:/vcpkg/scripts/buildsystems/vcpkg.cmake:857 (_find_package):
  Found package configuration file:

    E:/dk/x64-windows/share/Qt6/Qt6Config.cmake

  but it set Qt6_FOUND to FALSE so package "Qt6" is considered to be NOT
  FOUND.  Reason given by package:

  Failed to find required Qt component "WebEngineWidgets".

  Expected Config file at
  "E:/dk/x64-windows/share/Qt6WebEngineWidgets/Qt6WebEngineWidgetsConfig.cmake"
  does NOT exist



  Configuring with --debug-find-pkg=Qt6WebEngineWidgets might reveal details
  why the package was not found.

  Configuring with -DQT_DEBUG_FIND_PACKAGE=ON will print the values of some
  of the path variables that find_package uses to try and find the package.

Call Stack (most recent call first):
  core/cmake/rules/RulesQtFramework.cmake:21 (find_package)
  core/CMakeLists.txt:37 (include)


I need to recompile whole VCPKG from scratch.

Gilles
Comment 25 caulier.gilles 2024-08-29 16:11:44 UTC
Maik,

Whole Qt6 installation is done in the VM. digiKam is compiling. It wil be online in one hour...

Gilles
Comment 26 Maik Qualmann 2024-08-29 18:31:03 UTC
Great, Adobe lossy DNG images are now displayed correctly in the Windows version.

@Bertil Rolf: You have to read the metadata of the images in question again, once to change the format to DNG and to store the correct image size in the database. Either via the Item menu for selected images or via the Album menu for the entire folder -> Read metadata from the file to database.

digiKam-8.5.0 pre-release with Adobe lossy DNG support:

https://files.kde.org/digikam/

Maik
Comment 27 Bertil Rolf 2024-08-30 08:00:13 UTC
Created attachment 173119 [details]
attachment-761745-0.html

Hello!

The new DigiKam digiKam-8.5.0-20240829T162145-Qt6-Win64-debug.exe does 
what it should to my DNG photos, fully sized or lossy compressed. Only, 
it still marks the lossy compressed DNGs as “TIFF” in the overview of 
the whole album.

I removed the old version of digiKam and installed said file. The new 
installation preserves a memory of the old catalogue.

Thanks for your work. Looking forward to experiment with the new version 
of digiKam!

Best regards

Bertil Rolf

Baden bei Wien

On 2024-08-29 20:31, Maik Qualmann wrote:
> https://bugs.kde.org/show_bug.cgi?id=491888
>
> Maik Qualmann<metzpinguin@gmail.com>  changed:
>
>             What    |Removed                     |Added
> ----------------------------------------------------------------------------
>     Version Fixed In|                            |8.5.0
>               Status|REPORTED                    |RESOLVED
>           Resolution|---                         |FIXED
>
> --- Comment #26 from Maik Qualmann<metzpinguin@gmail.com>  ---
> Great, Adobe lossy DNG images are now displayed correctly in the Windows
> version.
>
> @Bertil Rolf: You have to read the metadata of the images in question again,
> once to change the format to DNG and to store the correct image size in the
> database. Either via the Item menu for selected images or via the Album menu
> for the entire folder -> Read metadata from the file to database.
>
> digiKam-8.5.0 pre-release with Adobe lossy DNG support:
>
> https://files.kde.org/digikam/
>
> Maik
>
Comment 28 caulier.gilles 2024-08-30 08:09:56 UTC
Maik,

The Qt5 Windows version of digiKam 8.5.0 will be also only today. I fixed the broken compilation in qtwebengine by upgrading MSVC.

Gilles
Comment 29 Maik Qualmann 2024-08-30 09:35:54 UTC
@Bertil Rolf:

You didn't read my comment, you have to re-read the metadata from the relevant images that are marked as TIFF. Also, the wrong image size was saved. Read Comment 26 again.

Maik
Comment 30 Bertil Rolf 2024-08-30 09:44:38 UTC
You're right. I hoped the reinstallation would take care of that. After 
re-reading metadata, digiKam now works beautifully.

Bertil


On 2024-08-30 11:35, Maik Qualmann wrote:
> https://bugs.kde.org/show_bug.cgi?id=491888
>
> --- Comment #29 from Maik Qualmann <metzpinguin@gmail.com> ---
> @Bertil Rolf:
>
> You didn't read my comment, you have to re-read the metadata from the relevant
> images that are marked as TIFF. Also, the wrong image size was saved. Read
> Comment 26 again.
>
> Maik
>