Bug 354819 - Specific pictures not showing up in digikam
Summary: Specific pictures not showing up in digikam
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Preview-Image (show other bugs)
Version: 4.11.0
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-11-04 07:54 UTC by Andreas Heinlein
Modified: 2018-05-03 19:33 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In: 6.0.0
Sentry Crash Report:


Attachments
Sample picture which does not work (2.52 MB, image/jpeg)
2015-11-04 07:57 UTC, Andreas Heinlein
Details
testcase for picture missing from preview list (918.45 KB, image/jpeg)
2017-12-19 20:34 UTC, Gernot Hillier
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andreas Heinlein 2015-11-04 07:54:11 UTC
I have the problem that some particular pictures refuse to show up in
digikam. They all come from one specific person resp. his camera. They
look like ordinary JPEGs, they open just fine in the GNOME picture
viewer as well as in GIMP. I have re-scanned several times as well as
doing a database maintenance, nothing helped.

If I re-export them from GIMP as JPEG, they also show up in digikam.
This is, however, no general solution for me, as this means loss of
quality and I expect to receive such pictures regularly in the future.

I can provide one of the files in question, if someone can tell me where
to put it.

I am using digikam 4.11 on Ubuntu 12.04. As it was quite a hassle to get
this to build, I'd rather not try the latest 4.x unless someone tells me
this will likely help. Sorry about that, I'd really to wait with
rebuildung until 5.x is ready instead.

Reproducible: Always

Steps to Reproduce:
1. Copy the picture file to an existing album
2. Re-scan the album
3. Picture is not shown

Actual Results:  
Rest of the album is shown, but not the file in question.

Expected Results:  
Picture is added to the album and a preview is shown
Comment 1 Andreas Heinlein 2015-11-04 07:57:55 UTC
Created attachment 95303 [details]
Sample picture which does not work

One of the files in question. Permission is granted by the photographer to use this file for bug testing. Please do not redistribute.
Comment 2 caulier.gilles 2015-11-04 08:01:03 UTC
Turn on debug trace and run digiKam in a console as explained here :

https://www.digikam.org/contrib

Try to reproduce the problem and report all messages printed on the console.
Comment 3 caulier.gilles 2016-07-06 16:30:13 UTC
This file still valid using last digiKam 5.0.0 ?

Gilles Caulier
Comment 4 caulier.gilles 2016-11-25 20:26:35 UTC
What's about this file using digiKam AppImage bundle 5.4.0 pre release given at
this url :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Gilles Caulier
Comment 5 Gernot Hillier 2017-12-19 20:32:00 UTC
I have a similar issue since several digikam releases including 5.7. I'm running it on OpenSUSE 42.3. 

It seems only pictures taken with my smartphone SM-G900FD are affected.

Sometimes, it helps to rename the problematic photos, but with the image attached here as "testcase.jpg", the problem persists, independent of filename. 

When I copy it in a new directory, this just looks like an empty album in Digikam. When I copy it in an existing album, digikam will show all other photos, but not "testcase.jpg".
Comment 6 Gernot Hillier 2017-12-19 20:34:37 UTC
Created attachment 109454 [details]
testcase for picture missing from preview list

Test picture, can be displayed with gthumb, dolphin, GIMP, etc. without problems, but digikam 5.7 on OpenSUSE 42.3 refuses to show it in the preview list.
Comment 7 Gernot Hillier 2017-12-19 21:26:30 UTC
Ok, after checking the digikam debug output on stdout, I found that digikam says this when the problematic image is added: 

----
digikam.database: Recognized "/home/gernot/sync/digibilder/to_sort/aa/testcase.jpg" as identical to item 94617. 
----

So, I'm likely hit by #375483. My smartphone producing those images is a Galaxy S5 and all images have the same ImageUniqueID.

As the initial sample picture from Andreas in this report has no ImageUniqueID, my problem is likely distince and I probably shouldn't have added my issue here, sorry.
Comment 8 caulier.gilles 2017-12-19 21:29:28 UTC
I recommend to test with 5.8.0 pre-release Linux AppImage bundle where some code have been fixed around the non single image ID provided by Samsung camera.

File is here :

https://files.kde.org/digikam/

Gilles Caulier
Comment 9 Gernot Hillier 2017-12-20 20:02:08 UTC
Unfortunately, 5.8.0 pre-release as of today does NOT fix the issue for me. I still have photos which are not shown by digikam. Also, I don't think this is directly connected to bug 375483 any more. Even removing the uuid (exiftool -ImageUniqId= <file>) doesn't change anything.

So far I found no rule when those files are visible in digikam and when not. With 5.7, I even had the same file (identical md5sum) twice under two different names in test directory "aa", one was visible, the other not. When moving them to another folder they both disappeared.

So far, it seems this does only affect photos taken by my Samsung smartphone. So there's probably another weirdness in the way Samsung creates JPGs.

When touching a visible and an invisible photo while digikam is running, stdout with 5.8 looks nearly identical:

invisible file:
digikam.general: Detected change, triggering rescan of "/home/gernot/sync/digibilder/to_sort/2017-08-xx Familie Landshut/"
digikam.database: Starting scan!
digikam.dimg: "/home/gernot/sync/digibilder/to_sort/2017-08-xx Familie Landshut/20170824_094516.jpg"  : JPEG file identified
digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>  QDateTime(2017-08-24 09:45:16.000 CEST Qt::TimeSpec(LocalTime))
digikam.metaengine: DateTime (Exif digitalized):  Do. Aug. 24 09:45:16 2017
digikam.metaengine: Orientation => Exif.Image.Orientation =>  6
digikam.metaengine: "/home/gernot/sync/digibilder/to_sort/2017-08-xx Familie Landshut/20170824_094516.jpg"  ==> Read Iptc Keywords:  ()
digikam.metaengine: Loading image history  ""
digikam.database: Scanning took 3 ms
digikam.database: Finishing took 36 ms
digikam.dimg: "/home/gernot/sync/digibilder/to_sort/2017-08-xx Familie Landshut/20170824_094516.jpg"  : JPEG file identified
digikam.metaengine: Orientation => Exif.Image.Orientation =>  6

visible file:
digikam.general: Detected change, triggering rescan of "/home/gernot/sync/digibilder/to_sort/2017-08-xx Familie Landshut/"
digikam.database: Starting scan!
digikam.dimg: "/home/gernot/sync/digibilder/to_sort/2017-08-xx Familie Landshut/20170824_091250.JPG"  : JPEG file identified
digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>  QDateTime(2017-08-24 09:12:50.000 CEST Qt::TimeSpec(LocalTime))
digikam.metaengine: DateTime (Exif digitalized):  Do. Aug. 24 09:12:50 2017
digikam.metaengine: Orientation => Exif.Image.Orientation =>  1
digikam.metaengine: "/home/gernot/sync/digibilder/to_sort/2017-08-xx Familie Landshut/20170824_091250.JPG"  ==> Read Iptc Keywords:  ()
digikam.metaengine: Loading image history  ""
digikam.database: Scanning took 6 ms
digikam.database: Finishing took 38 ms
digikam.dimg: "/home/gernot/sync/digibilder/to_sort/2017-08-xx Familie Landshut/20170824_091250.JPG"  : JPEG file identified
digikam.dimg: "/home/gernot/sync/digibilder/to_sort/2017-08-xx Familie Landshut/20170824_091250.JPG"  : JPEG file identified
digikam.metaengine: Orientation => Exif.Image.Orientation =>  1
digikam.metaengine: Orientation => Exif.Image.Orientation =>  1

The only difference is that "JEPG file identified" and the "Orientation" are printed *three* times for a working file while they are only printed *two* times for a file which is not shown in the UI. Does that somehow ring a bell for you?

Any further idea how I could narrow down this issue?
Comment 10 Gernot Hillier 2017-12-20 20:05:27 UTC
Just to make this clear: touching a problematic file while digikam is running leads to a rescan and the file is found according to debug output (see my last comment), but it still does NOT appear in the UI.

I enabled "Rescan file when files are modified" in the metadata settings as suggested in bug 375483.
Comment 11 Maik Qualmann 2017-12-20 20:30:50 UTC
Can you provide a test image from this smartphone?

Maik
Comment 12 Maik Qualmann 2017-12-21 07:21:08 UTC
Git commit 9a59b9d72b4d7606cf2bdb52be12a91fd22dbf21 by Maik Qualmann.
Committed on 21/12/2017 at 07:19.
Pushed by mqualmann into branch 'master'.

fix search or the Uuid if no XMP available

M  +29   -27   libs/dmetadata/dmetadata.cpp

https://commits.kde.org/digikam/9a59b9d72b4d7606cf2bdb52be12a91fd22dbf21
Comment 13 Gernot Hillier 2017-12-21 19:22:19 UTC
(In reply to Maik Qualmann from comment #11)
> Can you provide a test image from this smartphone?
> 
> Maik

I already attached it, see attachment 109454 [details]. Do you need further images? Unfortunately, this Heisen-bug isn't 100% reproducible, but if it helps I can add more examples to this bug or put them on my website.

Thanks for CC'ing https://commits.kde.org/digikam/9a59b9d72b4d7606cf2bdb52be12a91fd22dbf21, but you don't expect this to fix my bug, do you? I always add XMP:Creator to my pictures, so this commit shouldn't change anything for me, if I'm not mistaken.
Comment 14 Gernot Hillier 2017-12-21 21:23:36 UTC
Ok, I'm one large step ahead: all problematic images are internally tagged as "Original Version" despite I surely never edited those files:

sqlite> select * from ImageTags where imageid in (93042);
93042|112
sqlite> select * from Tags where id=112;
112|110|Original Version|0|

I can either delete the wrong internal tag:
delete from ImageTags where imageid in (select id from Images where name="20170907_102007.jpg");
or enable the "always show original version" setting in digikam.

My theory: During initial import from the Samsung smartphone, the files had the non-unique ImageUniqueID in EXIF data which Samsung deliberately uses to confuse people. Due to importing several files with the same unique ID, digikam somehow internally thought they are related and tagged some of them internally as "original version". 

Now, we could do what we want: manually change/remove EXIF ImageUniqueID field, internally create own uniqueIDs for it, moving the file away and back to digikam folders - this would never remove the internal originalVersion tag again.

Is it possible that those pictures received this flag because they
Comment 15 Maik Qualmann 2017-12-21 21:36:53 UTC
Right, the first import was the problem. We read though no XMP data exists an XMP key. Exiv2 returns the Exif key for this, but we will not check it there. My last commit fixes this issue. We always create a new Uuid now when the metadata is completely read. The problem should not occur now. I myself have a Samsung smartphone with this problem.

Maik
Comment 16 Gernot Hillier 2017-12-21 21:43:56 UTC
We should probably continue discussion in bug 375483, so that others can find this information, too, right?
Comment 17 Maik Qualmann 2018-05-03 19:33:49 UTC
The cause of this bug is now known and the bug with the Samsung UniqueID is fixed (Bug 375483). I close this bug now.

Maik