Bug 441698

Can confirm that only the thumbnail is displayed, not the full image.

There is already a bug at Astron Software (the `file` maintainers): https://bugs.astron.com/view.php?id=271

This seems to also affect programs other than Gwenview.

Isn't that the same as https://bugs.kde.org/show_bug.cgi?id=347574 ?

Displaying the Jpeg preview instead of the RAW file is a feature and not a bug, isn't it ?
Then the "issue" is that the embedded jpeg preview is smaller than the RAW file, at least for ARW files, I guess it's the same for NEF files ?

(In reply to Lapineige from comment #3)
> Isn't that the same as https://bugs.kde.org/show_bug.cgi?id=347574 ?
> 
> Displaying the Jpeg preview instead of the RAW file is a feature and not a
> bug, isn't it ?
> Then the "issue" is that the embedded jpeg preview is smaller than the RAW
> file, at least for ARW files, I guess it's the same for NEF files ?


A month ago I was able to view NEF images in (nearly?) full resolution, but now the resolution is 120x160, which isn't much for a 24MP image.

But since `file` detects it as a TIFF image, Gwenview might not be able to generate the JPEG preview.

I think if `file` would detect the image as `image/x-nikon-nef` the problem would be solved, is there any way to simulate or fake the MIME type for Gwenview to test it?

(In reply to Maximilian Schmeling from comment #4)
> but now the resolution is 120x160, which isn't much for a 24MP image.


Ok then that's clearly not the same issue.
ARW (Sony) files only contain a 1600*900px preview, hence that's what is displayed. They also contain a much smaller version (maybe something similar to 120*160 actually). Maybe in your case Gwenview switched from one to another ?

Could you run ```identify -verbose file.NEF  |grep review``` to see if several "Preview" are included in the Exifs ?

(In reply to Lapineige from comment #5)

> Could you run ```identify -verbose file.NEF  |grep review``` to see if
> several "Preview" are included in the Exifs ?

Nope, nothing is returned from ```identify -verbose file.NEF  |grep review``` (file.nef changed).

Maybe it's called thumbnail or something like that in NEF file exifs ?

NO thumbnail or anything like that found in the exif.
Output of identify -verbose (if it is useful to you):
Image:
  Filename: blackbird.nef
  Format: NEF (Nikon Digital SLR Camera Raw Image File)
  Class: DirectClass
  Geometry: 4284x2844+0+0
  Units: Undefined
  Colorspace: sRGB
  Type: TrueColor
  Base type: Undefined
  Endianness: Undefined
  Depth: 16-bit
  Channel depth:
    Red: 16-bit
    Green: 16-bit
    Blue: 16-bit
  Channel statistics:
    Pixels: 12183696
    Red:
      min: 0  (0)
      max: 65535 (1)
      mean: 26766.2 (0.408426)
      median: 26996 (0.411933)
      standard deviation: 8750.92 (0.13353)
      kurtosis: 6.56578
      skewness: 0.991362
      entropy: 0.890515
    Green:
      min: 0  (0)
      max: 65535 (1)
      mean: 19808.2 (0.302254)
      median: 20141 (0.307332)
      standard deviation: 7053.62 (0.107631)
      kurtosis: 10.7367
      skewness: 1.84245
      entropy: 0.845379
    Blue:
      min: 0  (0)
      max: 65535 (1)
      mean: 12425.3 (0.189598)
      median: 11672 (0.178103)
      standard deviation: 5535.46 (0.0844656)
      kurtosis: 13.9584
      skewness: 2.83654
      entropy: 0.807484
  Image statistics:
    Overall:
      min: 0  (0)
      max: 65535 (1)
      mean: 19666.6 (0.300093)
      median: 19603 (0.299123)
      standard deviation: 7113.33 (0.108542)
      kurtosis: 3.63934
      skewness: 1.14002
      entropy: 0.847793
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Matte color: grey74
  Background color: white
  Border color: srgb(223,223,223)
  Transparent color: none
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 4288x2844+2+0
  Origin geometry: +2+0
  Dispose: Undefined
  Iterations: 0
  Compression: Undefined
  Orientation: Undefined
  Profiles:
    Profile-icc: 940 bytes
    Profile-xmp: 6080 bytes
  Properties:
    date:create: 2021-08-30T13:08:03+00:00
    date:modify: 2008-05-31T18:10:12+00:00
    dng:camera.model.name: D3
    dng:create.date: 2008-05-09T15:56:53+00:00
    dng:exposure.time: 1/1000.0
    dng:f.number: 7.1
    dng:focal.length.in.35mm.format: 1000 mm
    dng:iso.setting: 1400.0
    dng:lens: 600.0-600.0mm f/4.0-4.0
    dng:lens.f.stops: 5.00
    dng:lens.type: G VR
    dng:make: Nikon
    dng:max.aperture.at.max.focal: 4.0
    dng:max.aperture.at.min.focal: 4.0
    dng:max.aperture.value: 6.7
    dng:max.focal.length: 600.0 mm
    dng:min.focal.length: 600.0 mm
    dng:ocal.length: 1000.0
    dng:serial.number: 2004426
    dng:software: Capture NX 2.0.0 M
    dng:wb.rb.levels: 465.000000 347.000000 256.000000 256.000000
    icc:copyright: Copyright (c) Eastman Kodak Company, 1999, all rights reserved.
    icc:description: ProPhoto RGB
    icc:manufacturer: KODAK
    icc:model: Reference Output Medium Metric(ROMM)  
    MicrosoftPhoto:Rating: 0
    photomechanic:Prefs: 0:0:0:006567
    photomechanic:TimeCreated: 175653-0700
    photoshop:DateCreated: 2008-05-09
    signature: 5fbb8d79a4142398f35fb31018408ce9f545f3603f2a4b1a90fe1cba89db2bad
    xap:Rating: 0
    xapRights:Marked: True
    xapRights:WebStatement: www.luminescentphoto.com
  Artifacts:
    verbose: true
  Tainted: False
  Filesize: 18.2634MiB
  Number pixels: 12.1837M
  Pixels per second: 6.37156MP
  User time: 5.530u
  Elapsed time: 0:02.912
  Version: ImageMagick 7.1.0-5 Q16 x86_64 2021-08-22 https://imagemagick.org

I believe we have the same issue in all 3 bugs:
- this one
- https://bugs.kde.org/show_bug.cgi?id=441189#c5
- https://bugs.kde.org/show_bug.cgi?id=441018

*** Bug 441189 has been marked as a duplicate of this bug. ***

*** Bug 441018 has been marked as a duplicate of this bug. ***

(sorry, I mistakenly submitted my last message too quickly :/)

And I can confirm the TIFF mime type issue, based on this error https://bugs.kde.org/show_bug.cgi?id=441018#c1 and on your screenshot : I have the same one (TIFF format recognized) with ARW files.

I took the liberty to mark those bug report as duplicates, and to change the title of this one to better represent the issue.


For the record, Gwenview is not the only software to be affected (https://bugs.kde.org/show_bug.cgi?id=441018#c2), but some others software are working correctly (such as Geeqie/showfoto/photoqt).

Created attachment 141312 [details]
Properties dialog of ARW image in Dolphin (or Gwenview)

So if it is only the MIME being detected wrongly, we will have to wait until these bug reports get resolved:
 - https://bugs.astron.com/view.php?id=271
 - https://bugs.astron.com/view.php?id=286

And yes, in the ArchLinux Forums I got the confirmation that other programs (lximage, Okular, kuickshow, and others) are also affected by the same problem. (https://bbs.archlinux.org/viewtopic.php?id=269188 comment #7)

(In reply to Maximilian Schmeling from comment #14)
> So if it is only the MIME being detected wrongly...
For what it's worth, I get:
    
    $ kmimetypefinder5 sample1.nef 
    image/x-nikon-nef

    $ xdg-mime query filetype sample1.nef
    image/x-nikon-nef

But:

    $ file sample1.nef
    sample1.nef: TIFF image data, little-endian, direntries=27, height=320, bps=338, compression=none, Photometric

This is with:

    Arch Linux (from pkgbuild 20210815)

    Plasma: 5.22.5
    Frameworks: 5.85.0
    Qt: 5.15.2

and

    $ pacman -Q shared-mime-info
    shared-mime-info 2.0+57+gc1d1c70-1

Does Gwenview even use `file` internally?

Seems as if it uses `kmimetypefinder5` or `xdg-mime` for the file type and `file` for the content type.

Gwenview on Manjaro stable got update to  21.08.1-1, but the issue is not fixed. 

I have downgraded file to 5.40-2, but also no help. 

Gwenview broke after update. Probably file and gwenview both got update. But thats not logical that they wont work after ei downgraded. So there must be something else that causing the issue? Cant downgrade libtiff because dependencies.

I have tried file 5.40-2 and gwenview 21.04.1 with no luck

Anyone able to test it with a fresh install (VM) / live usb ? I wonder if that would work…

I tried Gwenview 21.08 flatpak (from flathub) in Elementary Os 6 (using Boxes). Same issue (nef-s only with previews, no actual raw)

(In reply to Maximilian Schmeling from comment #14)
> So if it is only the MIME being detected wrongly, we will have to wait until
> these bug reports get resolved...
The issue with "file" and "xdg-mime query filetype" reporting different results seems common (checked on recent and older Neon and Fedora)

I don't see a correlation with the gwenview behaviour.

The behaviour of gwenview differs with distribution, each up-to-date:

    On Arch, it says it cannot display documents of type image/tiff.
    On Neon, it displays the small (thumbnail) image with the 212x320
        size that "file" reports
    On Fedora, it displays the full sized 2832x4256 image

The behaviour on Neon changed between March and May. Digging down into a series of snapshots, I see that:

    Neon Testing, 2021-03-??
    Plasma: 5.21.0
    Frameworks: 5.82.0
    Gwenview: 21.04.0

(and earlier) displays the proper 2832x4256 image, whereas:

    Neon Unstable, 2021-05-11
    Plasma: 5.21.80
    Frameworks: 5.83.0
    Gwenview: 21.07.70

(and later) displays the thumbnail 212x320 size.

(In reply to tagwerk19 from comment #22)
>     On Arch, it says it cannot display documents of type image/tiff.


You have to install kimageformats and/or qt5-imageformats for it to display the thumbnail

(In reply to tagwerk19 from comment #22)

>     On Fedora, it displays the full sized 2832x4256 image

What does the properties dialog show on Fedora?

And what Gwenview version Fedora have?

>>
>     On Arch, it says it cannot display documents of type image/tiff.


You have to install kimageformats and/or qt5-imageformats for it to display the thumbnail

kimageformat-plugins already installed on Kubuntu 21.04, still no preview.

Created attachment 141405 [details]
Sample1.nef properties - Dolphin, Fedora 34

Created attachment 141406 [details]
Sample1.nef properties - Gwenview, Fedora 34

Looks like Gwenview is seeing duplicate attributes

(In reply to AL from comment #25)
> And what Gwenview version Fedora have?
Fedora first then...
    Fedora 34
    Plasma: 5.22.4
    Frameworks: 5.85.0
    Qt: 5.15.2
    Gwenview: 21.04.1

(In reply to Maximilian Schmeling from comment #24)
> What does the properties dialog show on Fedora?
I've attached a couple of screenshots of the Dolphin and Gwenview properties popups..

Looks like what Gwenview is picking up is "a mess".

Created attachment 141407 [details]
Sample1.nef properties - Dolphin, Arch Linux

I'd say the properties as per Arch and Fedora match...

Created attachment 141408 [details]
Sample1.nef properties - Gwenview, Arch Linux

Similarly, it looks like the properties as per Gwenview (Arch and Fedora) also match

(In reply to Maximilian Schmeling from comment #23)
> You have to install kimageformats and/or qt5-imageformats for it to display
> the thumbnail
Not easy to keep things in the right order when replying...

I needed to install qt5-imageformats.

What's interesting is that the properties "as seen by" Gwenview look the same on (current versions) of Arch and Fedora. Both look "messed up" with attributes duplicated.

I wonder if Gwenview is just trying to make the of conflicting data. It would also be interesting to confirm that it's not just this image causing problems.

(In reply to tagwerk19 from comment #32)
> I wonder if Gwenview is just trying to make the of conflicting data...
Ouch...
    I wonder if Gwenview is just trying to make the best of conflicting data...

I see the duplicated attributes in the Gwenview image properties even when I go back to a snapshot from 2020/12

    Neon Unstable
    Plasma: 5.20.80
    Frameworks: 5.78.0
    Qt: 5.15.2
    Gwenview: 21.03.70

There also a reddit thread here:
    https://old.reddit.com/r/kde/comments/pa8wqc/gwenview_not_supporting_raw_files_anymore/

According to this site: http://lclevy.free.fr/nef/
The Nikon Electronic File format (NEF) also stores an (approximately) 160x120 preview thumbnail (uncompressed TIFF).
Gwenview might choose the thumbnail instead of the raw image.
This would also explain the duplicate attributes, one for the real raw image and the other for the thumbnail.

Yes. Old "file" versions also showed nikon raw as tiff, so this is probably gwenview (or some kde package) issue and not mimetype

May be someone should make new bug report (about duplicate entries) for this to be sorted out

Created attachment 141546 [details]
gwenview_fix_raw_detection_with_embedded_tiff_previews.patch

This patch should fix RAW files mime-type detection with embedded TIFF previews.

Also I've applied this patch to fix opening of Sony RAW files:
```
diff --git a/lib/mimetypeutils.cpp b/lib/mimetypeutils.cpp
index 3eb2c157..9780cc8a 100644
--- a/lib/mimetypeutils.cpp
+++ b/lib/mimetypeutils.cpp
@@ -74,6 +74,7 @@ static inline QStringList rawMimeTypes()
QStringLiteral("image/x-pentax-pef"),
QStringLiteral("image/x-adobe-dng"),
QStringLiteral("image/x-sony-arw"),
+        QStringLiteral("image/x-sony-sr2"),
QStringLiteral("image/x-minolta-mrw"),
QStringLiteral("image/x-panasonic-raw"),
QStringLiteral("image/x-panasonic-raw2"),
```

You can download RAW-file examples here: https://rawsamples.ch/index.php/en/sony.

Thank you for the patches! Please submit them at https://invent.kde.org/graphics/gwenview/-/merge_requests/ :)

https://invent.kde.org/graphics/gwenview/-/merge_requests/110

(In reply to Yaroslav Sidlovsky from comment #36)
> This patch should fix RAW files mime-type detection with embedded TIFF
> previews.
I see in Comment 14 that it's not just gwenview affected. Is it a question that an underlying service is not cleanly parsing the data and the user-facing applications are having to untangle the mess?

Of course, better untangling is good but it would be nice to know we've got to the root cause...

Normal disclaimers apply here. I won't pretend to follow what's happening in the code.

Thank you very much for that patch !
If any test version is available, I'm ready to test it if you want :)

For the second patch, if I understand well, it only applies to SR2 RAW files, not ARW files, right ?

Will this be fixed in Plasma 5.23?

I have the same issue on Gentoo. I can confirm that patch works. Hope it will be fixed in next release.

I updated Gwenview to 21.08.2-1 but they didnt merge the patch, so Gwenview still cant open raw files. Or at least for me

But i changed my workflow and use now Xnview. Not opensource, but faster and works well :) So dont care about gwenview anymore

Still not working in Gwenview 21.12.1

(In reply to Marcelo Sales from comment #45)
> Still not working in Gwenview 21.12.1

I can confirm that.
Tested on Kubuntu 21.10 using Kubuntu backports PPA.

Yes, the bug is still in Gwenview 21.12.
Thats sad

(In reply to Maximilian Schmeling from comment #34)
> According to this site: http://lclevy.free.fr/nef/
> The Nikon Electronic File format (NEF) also stores an (approximately)
> 160x120 preview thumbnail (uncompressed TIFF).
> Gwenview might choose the thumbnail instead of the raw image.
> This would also explain the duplicate attributes, one for the real raw image
> and the other for the thumbnail.

Well, there are acutally two jpeg previews in each and every RAW file: 160x120 and full-res or reduced-but-high-enough. What's the former one purpose, I don't know. The latter one is there so you can review the image on the back of the camera whle not draining the battery for demosaicing. dcraw -e abdc.raw gives you the preview. Gwenview uses libraw to extract the high-res jpeg preview and feed it into the jpeg pipeline. I know that, because I implemented the feature back in 2013. Unfortunatelly, I kind of fell out of the process and haven't written a line in C++ since then so I am not sure I can fix the breakage which I discovered right now. What a mess...

I believe this patch shall be reverted:

https://invent.kde.org/graphics/gwenview/-/commit/6a79391a9ab68bec839897369c665ee2e3afe7e5

When fixing a bug, there always shall be a) regression testing and b) consideration if the benefit of the fix outweights the problems it causes. In this case, it fails in both.

So, Gwenview developers are very aware of broken raw support, they knowingly did it because they thought more import is if somebody accidently renames png to gif and gwenview wont be able to open that? So Gwenview will open this gif file  as png but wont open raw files. Wow, fixing 1 bug but making very big regression

I really hope they will revert this patch

I'll look into this and handle it tomorrow.

Git commit 1281004fff1fbafcee221400aee8dae644260092 by Nate Graham.
Committed on 29/01/2022 at 18:10.
Pushed by ngraham into branch 'release/21.12'.

Revert "Prefer mime type from content over file name when loading"

This reverts commit 6a79391a9ab68bec839897369c665ee2e3afe7e5.

That commit caused Gwenview to lose the ability to open RAW files. This
is a more serious regression than the bug being fixed by the commit
(329140), so it needs to be reverted for now until that bug can be
fixed in a way that doesn't regress RAW loading.
FIXED-IN: 21.12.2

CCMAIL: ahiemstra@heimr.nl

M  +4    -6    lib/document/loadingdocumentimpl.cpp

https://invent.kde.org/graphics/gwenview/commit/1281004fff1fbafcee221400aee8dae644260092

(In reply to Nate Graham from comment #53)
> Revert "Prefer mime type from content over file name when loading"
Looks good

Neon Testing now works for me:
    Plasma : 5.23.90
    Frameworks : 5.91.0
    Qt : 5.15.3
    Gwenview : 21.12.2

Thank you!

I *think* the patch takes the behaviour back to whatever internally coded; don't see mime types (as per freedesktop mime types) having an influence

(In reply to tagwerk19 from comment #54)
> I *think* the patch takes the behaviour back to whatever internally coded;
> don't see mime types (as per freedesktop mime types) having an influence
I've dug into the format of the sample1.nef file. I'll give a warning that it's not my day-job so it's fair to treat my observations with suspicion but maybe it will help future travellers...

The mime database:
    /usr/share/mime/packages/freedesktop.org.xml
has
    <mime-type type="image/x-nikon-nef">
      <comment>Nikon NEF raw image</comment>
       ... translations ...
      <acronym>NEF</acronym>
      <expanded-acronym>Nikon Electronic Format</expanded-acronym>
      <sub-class-of type="image/x-dcraw"/>
      <sub-class-of type="image/tiff"/>
      <glob pattern="*.nef"/>
    </mime-type>

    <mime-type type="image/tiff">
      <comment>TIFF image</comment>
       ... translations ...
      <acronym>TIFF</acronym>
      <expanded-acronym>Tagged Image File Format</expanded-acronym>
      <magic priority="50">
        <match type="string" value="MM\x00\x2a" offset="0"/>
        <match type="string" value="II\x2a\x00" offset="0"/>
      </magic>
      <glob pattern="*.tif"/>
      <glob pattern="*.tiff"/>
    </mime-type>

So, it seems not to be able to distinguish (beyond looking at the file extension) between an image/x-nikon-nef and an image/tiff file.

If I try imagemagick identify, it _can_ pull out correct details if I give an extension:

    $ identify -verbose sample1.nef

    Image:
      Filename: sample1.nef
      Format: NEF (Nikon Digital SLR Camera Raw Image File)
      Class: DirectClass
      Geometry: 2844x4284+0+0
       ...

whereas renaming sample1.nef to sample1.tiff

    $ identify -verbose sample1.tiff

    Image:
      Filename: sample1.tiff
      Format: TIFF (Tagged Image File Format)
      Mime type: image/tiff
      Class: DirectClass
      Geometry: 212x320+0+0
         ...
        tiff:subfiletype: REDUCEDIMAGE
         ...

Gwenview is now following the same logic (after the rollback); give it an extension and it will find that format image and display it: the sample1 file is effectively a container and holds the image in three formats.

Before the rollback, Gwenview displayed the REDUCEDIMAGE.

Gut feeling is that the logic needed to "pick out" the primary image is too complicated to implement in the mimetype database. You can see though that there are 'ad hoc' patterns based on the Manufacturer tag set up for some other "raw" types.

It might be worth working on some image/x-nikon-nef "magic" but it won't change the Gwenview behaviour (something of a supposition...)

The behaviour, of Gwenview showing multiple copies of tags, seen in
    https://bugsfiles.kde.org/attachment.cgi?id=141406
is better Moved To A New Bug

I can confirm that this bug has been resolved for me too!
Thanks a lot to all the contributors who helped fixing this

Sorry it took so long!

Fixed here too, thank you very much !