Bug 488662

Summary: Incorrect Thumbnails showing on scanned Tiff files
Product: [Applications] digikam Reporter: AlanB <gingerlefty>
Component: Thumbs-EngineAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: normal CC: caulier.gilles, metzpinguin
Priority: NOR    
Version: 8.3.0   
Target Milestone: ---   
Platform: Microsoft Windows   
OS: Microsoft Windows   
Latest Commit: Version Fixed In: 8.5.0
Sentry Crash Report:

Description AlanB 2024-06-18 11:48:04 UTC
Hi,
I've recently started digitising my archive of film negatives, mainly 35mm. I am using an Epson V600 flat bed scanner and Epson Scan 2 software. The scanner is set to High quality, 24 Bit Colour, 3200dpi and the output set to save as Tiff and Adobe RGB colour space.
When I look at the files in Windows File Explorer everything is fine and as expected.
When I import the files into digiKam as Albums most of the scans  are fine, however with a significant number (between 10 and 20 percent) of the images the thumbnails shown in the Album view of digiKam do not relate to the photo. If the file is opened in editing software (I've tried a few) the file always opens as the correct image.
I've tried several ways within digiKam to try and correct the issues in the Album view, including a complete uninstall of digiKam and deleting of the database and all the xmp files attached to the scanned photos on a secondary installation I have on a Windows 11 laptop. Upon re-installation the problem was still present.
If I refresh an Album sometimes the files affected with incorrect thumbnails are changed.
The only way I have found, so far, of solving the issue is to batch convert in Affinity Photo 2 all the files in every Album of scanned images to 16bit Tiff. This seems to correct all the problems as the thumbnails for both original scans and the processed tiffs are correct.
Just for information I still use film occasionally and I get the processed images returned as high quality Tiffs. I've not had a problem with any of these images.



Windows:10 & 11 
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

Alan
Comment 1 caulier.gilles 2024-06-18 13:49:21 UTC
Hi,

Please test with the current 8.4.0 pre-release installer for Windows available here :

https://files.kde.org/digikam/

Gilles Caulier
Comment 2 Maik Qualmann 2024-06-18 16:28:43 UTC
My guess is that the TIFF file has an incorrect thumbnail or preview image embedded in it. Can you provide a TIFF file whose thumbnail is incorrect in digiKam?

Maik
Comment 3 Maik Qualmann 2024-06-18 17:39:11 UTC
Hmm, I can reproduce the problem with 2 large TIFF files without metadata, where only the middle image content changes. The problem is our unique hash V2. This is the same for the two sample files I created (first and last 100 KB identical). I think it's time to think about a V3 of our hash.

Maik
Comment 4 Maik Qualmann 2024-06-18 19:46:14 UTC
first idea for a unique hash V3 function
2 zones in the middle of the file are also used.
Possibly the block size could be reduced from 100 kB.

https://invent.kde.org/graphics/digikam/-/commit/9c1298243e5c241d4c399cc1ca296f632fdfa0a7

Maik
Comment 5 Maik Qualmann 2024-06-19 20:39:50 UTC
Git commit cdfa0763d343b06a2df74fe48d1813f0ed5ab475 by Maik Qualmann.
Committed on 19/06/2024 at 20:38.
Pushed by mqualmann into branch 'master'.

unique hash V3++

M  +9    -6    core/libs/dimg/dimg_metadata.cpp

https://invent.kde.org/graphics/digikam/-/commit/cdfa0763d343b06a2df74fe48d1813f0ed5ab475
Comment 6 Maik Qualmann 2024-06-19 20:43:58 UTC
We definitely need sample TIFF images as a few whose thumbnails influence each other to confirm our theory.

Use an upload service, you can send the link to my private mail.

Maik
Comment 7 AlanB 2024-06-20 10:14:24 UTC
Hi,
I've uploaded 7 of the affected files to Dropbox and emailed the link as requested,

Alan
Comment 8 Maik Qualmann 2024-06-20 10:42:16 UTC
Thank you for your sample TIFF files. The theory of identical unique hash values ​​has been confirmed. Although the images are all very different, identical hash pairs were formed. This is partly because the TIFF images are uncompressed and have a narrow white frame.

A test with my new hash creation shows that it can solve this problem. I have already done some tests, e.g. that it has the same performance on network drives, etc. It is not yet clear whether we will add it in digiKam-8.4.0, although the update would have to be carried out manually via the setup and you could decide for yourself...

Maik
Comment 9 AlanB 2024-06-20 14:41:07 UTC
Thank you Maik.
One other follow on if possible.
The work round I have been using i.e. Batch processing the images to 16bit Tiffs seems to correct the problem in the original scanned files . Do you think it would be OK to delete the batched 16bit tiffs and use the originals. The reason I ask is that the colour fidelity seems better in the unprocessed files.
The work round doesn't take a long time and I'd rather not start messing about when digikam works just fine, other then this bug. This is assuming your hash creation will be incorporated in the not too distant future.

Thanks again,

Alan
Comment 10 Maik Qualmann 2024-06-29 05:42:13 UTC
Git commit ee34065972def1f32990a54d9921e62d11fe45ad by Maik Qualmann.
Committed on 29/06/2024 at 05:41.
Pushed by mqualmann into branch 'master'.

prepare for the switch to a new unique hash
API optimized to be more universal for unique hash functions.

M  +2    -6    core/libs/database/coredb/coredb.cpp
M  +0    -2    core/libs/database/coredb/coredb.h
M  +1    -1    core/libs/database/item/containers/iteminfo_history.cpp
M  +9    -7    core/libs/database/item/scanner/itemscanner_history.cpp
M  +11   -5    core/libs/dimg/dimg.h
M  +27   -11   core/libs/dimg/dimg_metadata.cpp
M  +2    -1    core/libs/dimg/dimg_p.h

https://invent.kde.org/graphics/digikam/-/commit/ee34065972def1f32990a54d9921e62d11fe45ad
Comment 11 Maik Qualmann 2024-07-07 18:39:56 UTC
Git commit 189e3af8fc2805e2ffd8ebd6783b9f7771212469 by Maik Qualmann.
Committed on 07/07/2024 at 18:38.
Pushed by mqualmann into branch 'master'.

unique file hash V3++

M  +2    -2    core/libs/dimg/dimg.h
M  +23   -12   core/libs/dimg/dimg_metadata.cpp

https://invent.kde.org/graphics/digikam/-/commit/189e3af8fc2805e2ffd8ebd6783b9f7771212469
Comment 12 Maik Qualmann 2024-08-04 17:32:28 UTC
Git commit 049566774378a7b5bd3c65ad11e860f8460bec6f by Maik Qualmann.
Committed on 04/08/2024 at 17:31.
Pushed by mqualmann into branch 'master'.

enable update to unique hash V3
FIXED-IN: 8.5.0

M  +1    -1    NEWS
M  +1    -1    core/libs/database/coredb/coredbschemaupdater.cpp
M  +3    -3    core/utilities/setup/setupdatabase.cpp

https://invent.kde.org/graphics/digikam/-/commit/049566774378a7b5bd3c65ad11e860f8460bec6f