Bug 320466 - SCAN : after to run BQM a lot of files are missing in icon-view
Summary: SCAN : after to run BQM a lot of files are missing in icon-view
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Unclassified
Component: Database-Scan (show other bugs)
Version: 3.5.0
Platform: Ubuntu Packages Linux
: NOR grave with 5 votes (vote)
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-05-30 07:45 UTC by Leo
Modified: 2017-07-19 13:48 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.11.0


Attachments
The origin (703.14 KB, application/gzip)
2013-06-03 08:30 UTC, Leo
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Leo 2013-05-30 07:45:34 UTC
I processed (converted to jpg and resized) 96 files (some jpg some crw) crating it in a new folder
Everything went fine but if I look in the new folder with dolphin I see all 96 files, but if I look in the main digikam gui, I find only 84!!!!

Reproducible: Always

Steps to Reproduce:
1. produce some file with a batch process into a new dir
2. diff the 2 dir
3. the number of files is different
I've tryed to do the same actions on the same file with different target dir... Everytime I find ALL the files in the dirs with dolphin but the digikam gui shows me differents files (some times 84, sometimes 86) but NEVER all the files!!!! :(
Actual Results:  
not all the file produced are visible in digikam

Expected Results:  
all files should be visible
Comment 1 caulier.gilles 2013-05-30 07:55:01 UTC
How do you process RAW to JPEG ?

Can you run "ls -al" in album with JPEG files.
Also take a shot of icon view from digiKam with this album.

Gilles Caulier
Comment 2 Leo 2013-06-03 08:30:42 UTC
Created attachment 80273 [details]
The origin

Here are the file requested.
A list of all the files in both dir, a snapshot of gui in the 2 dirs, and as you can see in dolphin snapshot, the count of files is correct in both dirs, but in DK some files in jpg are missing ;(
Comment 3 Nathan 2013-07-23 13:12:04 UTC
I have experienced this situation twice.  The first time is via a process similar to the above steps, but I used dark table to do the conversion from Olympus RAW to tiff (265 files, 265 preview images) and then after a bit more processing and renaming from tiff to jpg (265 files, 186 preview images) in digiKam.  This was first seen on digiKam 3.1 on Kubuntu 13.04.  I upgraded to 3.2 and still see the same issue.

In the second case, the setup is a bit different.  I took 459 files from someone else and copied them to my computer.  I then opened the gallery in digiKam and it shows only 47 files with 47 preview images.  These were taken in camera as jpeg images and have not been converted at all.  I have only noticed this on digiKam 3.2 on Kubuntu 13.04

The second one is not identical, but is close enough in effect that I thought I should mention them both.  All files are owned by me with at least r&w permissions for me.
Comment 4 Nathan 2013-07-27 16:24:36 UTC
I have a work around in case anyone needs it.  In every case I have tried it, importing the "missing" pictures causes them to show up.  So the steps for this work around are as follows:

1) In digiKam, create a new gallery.  This is just temporary storage (call it DKTemp).
2) Select all thumbnails and move them to the new temporary folder (DKTemp).
3) In some other program (not digiKam - shell, dolphin, etc.) move all of the files that remain in the original folder (call it DKGallery) to some other location - I used /tmp
4) In digiKam, move everything from DKTemp to DKGallery.
5) In digiKam, import everything from /tmp to DKGallery.
6) Delete everything in /tmp
Comment 5 caulier.gilles 2013-10-31 07:56:55 UTC
digiKam 3.5.0 is out.

Can you give a fresh feedback about your report ? Problem still reproducible ?

Thanks in advance

Gilles Caulier
Comment 6 Nathan 2013-11-11 16:57:21 UTC
I was able to reproduce it in 3.4.0, but I'll see if I can do the same in 3.5.0.

One negative on 3.4.0 was that my work around (the import process) didn't work anymore.
Comment 7 Nathan 2013-11-28 17:08:59 UTC
I can confirm that the issue still exists in 3.5.0
Comment 8 caulier.gilles 2013-11-28 17:27:17 UTC
Are you sure that problem is not relevant of digiKam versioning settings ? Check on setup dialog for details, especially the section Editing Images where "Always show original image" and "Always show intermediate snapshot" options can be a solution...

Gilles Caulier
Comment 9 Nathan 2013-11-28 18:16:29 UTC
I've thought about that and it isn't the case here.  When I do a batch process, I am going to a new folder with a rename - so it shouldn't be a versioning thing.  In addition, I have it set to destructive save, so I don't get versions.  This is truly a new file/photo that just doesn't show in the main thumbnail window no matter what I do.

One change on this version - this is the first time on this bug that I have compiled digiKam from source.  I have historically waited until KUbuntu has it in the repositories prior to upgrading.  I wanted to troubleshoot this more though, so I compiled instead of waiting.  I'm happy to run tests or check other things if needed!
Comment 10 Faldrian 2014-03-03 10:07:16 UTC
I use digikam 3.5.0 on Gentoo and also have this bug.
When I use batch to resize pictures into another folder, there are sometimes 1 or 2 pictures missing in the target folder - at least digikam does not have them in list. On Disk all files are there, so it's just an issue with the GUI not showing them or not indexing them.
Comment 11 Nathan 2014-05-09 13:20:40 UTC
ShowFoto works:

I just realized something interesting that might be a decent work around for now.  If I open a folder in showFoto, it will display all of the pictures in the folder window even for folders/galleries that do not display all of the images in digiKam.  There must be a difference between the way these 2 applications generate their folder view thumbnails despite how closely related they are.
Comment 12 caulier.gilles 2014-05-09 13:49:01 UTC
digiKam use DB to register file system items.
Showfoto do not use DB.

Gilles Caulier
Comment 13 Nathan 2014-05-14 13:54:52 UTC
It appears that the photos are being registered in the database correctly - as in the insert is at least partially successful.  Details:

Look in the gallery and I see 10 thumbnails, but the Album pane shows there should be 23.

Database queries:
select * from Images where album = 1154;  (23 rows - correct)

select *
from Images i
	inner join ImageMetadata im on i.id = im.imageid
where i.album = 1154; (8 rows - Every image here has a thumbnail, but so do 2 others)

select *
from Images i
	inner join ImageInformation ii on i.id = ii.imageid
where i.album = 1154; (23 rows - correct)

select *
from Images i
	inner join ImageHistory ih on i.id = ih.imageid
where i.album = 1154; (23 rows - correct)

I'll try and get the MySQL general log output of a batch editing run shortly.
Comment 14 Nathan 2014-06-21 16:13:34 UTC
I experienced this bug on import of photos and turned on the general log at that point.  Deleted and re-imported the files.  The same files were missing and it appears that the reason it is always the same files is that the insert into ImageInformation uses the previous row as the source for the new import.  Would it be possible for import to do it from scratch each time instead of trying to be smart about it and failing consistently?

Line from the log:
INSERT INTO ImageInformation  (imageid, rating, creationDate, digitizationDate, orientation,   width, height, format, colorDepth, colorModel) SELECT 224073, rating, creationDate, digitizationDate, orientation,   width, height, format, colorDepth, colorModel FROM ImageInformation WHERE imageid=224056
Comment 15 Nathan 2014-06-21 16:18:44 UTC
(In reply to comment #14)
> I experienced this bug on import of photos and turned on the general log at
> that point.  Deleted and re-imported the files.  The same files were missing
> and it appears that the reason it is always the same files is that the
> insert into ImageInformation uses the previous row as the source for the new
> import.  Would it be possible for import to do it from scratch each time
> instead of trying to be smart about it and failing consistently?
> 
> Line from the log:
> INSERT INTO ImageInformation  (imageid, rating, creationDate,
> digitizationDate, orientation,   width, height, format, colorDepth,
> colorModel) SELECT 224073, rating, creationDate, digitizationDate,
> orientation,   width, height, format, colorDepth, colorModel FROM
> ImageInformation WHERE imageid=224056

(In reply to comment #14)
> I experienced this bug on import of photos and turned on the general log at
> that point.  Deleted and re-imported the files.  The same files were missing
> and it appears that the reason it is always the same files is that the
> insert into ImageInformation uses the previous row as the source for the new
> import.  Would it be possible for import to do it from scratch each time
> instead of trying to be smart about it and failing consistently?
> 
> Line from the log:
> INSERT INTO ImageInformation  (imageid, rating, creationDate,
> digitizationDate, orientation,   width, height, format, colorDepth,
> colorModel) SELECT 224073, rating, creationDate, digitizationDate,
> orientation,   width, height, format, colorDepth, colorModel FROM
> ImageInformation WHERE imageid=224056

I believe the missing row in "ImageInformation" is controlling as the query run when viewing a gallery joins to that table.  Such as:
SELECT DISTINCT Images.id, Images.name, Images.album,        ImageInformation.rating, Images.category,        ImageInformation.format, ImageInformation.creationDate,        Images.modificationDate, Images.fileSize,        ImageInformation.width, ImageInformation.height  FROM Images        INNER JOIN ImageInformation ON Images.id=ImageInformation.imageid  WHERE Images.status=1 AND Images.album = ?


I have yet to capture the initial insertion for the parent file into ImageInformation for why it failed.
Comment 16 caulier.gilles 2015-05-10 08:16:41 UTC
Maik,

Your last changes in BQM about image scanner handling must fix this issue with 4.10.0.

Gilles
Comment 17 Maik Qualmann 2015-05-10 10:20:59 UTC
Gilles,

Possible that it is related with the blocked DB.

Maik
Comment 18 caulier.gilles 2015-05-10 10:24:53 UTC
I think it's the problem.

I close file as resolved with 4.10.0 release. We will see the user feedback with this release.

Gilles Caulier