Bug 320466

How do you process RAW to JPEG ?

Can you run "ls -al" in album with JPEG files.
Also take a shot of icon view from digiKam with this album.

Gilles Caulier

Created attachment 80273 [details]
The origin

Here are the file requested.
A list of all the files in both dir, a snapshot of gui in the 2 dirs, and as you can see in dolphin snapshot, the count of files is correct in both dirs, but in DK some files in jpg are missing ;(

I have experienced this situation twice.  The first time is via a process similar to the above steps, but I used dark table to do the conversion from Olympus RAW to tiff (265 files, 265 preview images) and then after a bit more processing and renaming from tiff to jpg (265 files, 186 preview images) in digiKam.  This was first seen on digiKam 3.1 on Kubuntu 13.04.  I upgraded to 3.2 and still see the same issue.

In the second case, the setup is a bit different.  I took 459 files from someone else and copied them to my computer.  I then opened the gallery in digiKam and it shows only 47 files with 47 preview images.  These were taken in camera as jpeg images and have not been converted at all.  I have only noticed this on digiKam 3.2 on Kubuntu 13.04

The second one is not identical, but is close enough in effect that I thought I should mention them both.  All files are owned by me with at least r&w permissions for me.

I have a work around in case anyone needs it.  In every case I have tried it, importing the "missing" pictures causes them to show up.  So the steps for this work around are as follows:

1) In digiKam, create a new gallery.  This is just temporary storage (call it DKTemp).
2) Select all thumbnails and move them to the new temporary folder (DKTemp).
3) In some other program (not digiKam - shell, dolphin, etc.) move all of the files that remain in the original folder (call it DKGallery) to some other location - I used /tmp
4) In digiKam, move everything from DKTemp to DKGallery.
5) In digiKam, import everything from /tmp to DKGallery.
6) Delete everything in /tmp

digiKam 3.5.0 is out.

Can you give a fresh feedback about your report ? Problem still reproducible ?

Thanks in advance

Gilles Caulier

I was able to reproduce it in 3.4.0, but I'll see if I can do the same in 3.5.0.

One negative on 3.4.0 was that my work around (the import process) didn't work anymore.

I can confirm that the issue still exists in 3.5.0

Are you sure that problem is not relevant of digiKam versioning settings ? Check on setup dialog for details, especially the section Editing Images where "Always show original image" and "Always show intermediate snapshot" options can be a solution...

Gilles Caulier

I've thought about that and it isn't the case here.  When I do a batch process, I am going to a new folder with a rename - so it shouldn't be a versioning thing.  In addition, I have it set to destructive save, so I don't get versions.  This is truly a new file/photo that just doesn't show in the main thumbnail window no matter what I do.

One change on this version - this is the first time on this bug that I have compiled digiKam from source.  I have historically waited until KUbuntu has it in the repositories prior to upgrading.  I wanted to troubleshoot this more though, so I compiled instead of waiting.  I'm happy to run tests or check other things if needed!

I use digikam 3.5.0 on Gentoo and also have this bug.
When I use batch to resize pictures into another folder, there are sometimes 1 or 2 pictures missing in the target folder - at least digikam does not have them in list. On Disk all files are there, so it's just an issue with the GUI not showing them or not indexing them.

ShowFoto works:

I just realized something interesting that might be a decent work around for now.  If I open a folder in showFoto, it will display all of the pictures in the folder window even for folders/galleries that do not display all of the images in digiKam.  There must be a difference between the way these 2 applications generate their folder view thumbnails despite how closely related they are.

digiKam use DB to register file system items.
Showfoto do not use DB.

Gilles Caulier

It appears that the photos are being registered in the database correctly - as in the insert is at least partially successful.  Details:

Look in the gallery and I see 10 thumbnails, but the Album pane shows there should be 23.

Database queries:
select * from Images where album = 1154;  (23 rows - correct)

select *
from Images i
	inner join ImageMetadata im on i.id = im.imageid
where i.album = 1154; (8 rows - Every image here has a thumbnail, but so do 2 others)

select *
from Images i
	inner join ImageInformation ii on i.id = ii.imageid
where i.album = 1154; (23 rows - correct)

select *
from Images i
	inner join ImageHistory ih on i.id = ih.imageid
where i.album = 1154; (23 rows - correct)

I'll try and get the MySQL general log output of a batch editing run shortly.

I experienced this bug on import of photos and turned on the general log at that point.  Deleted and re-imported the files.  The same files were missing and it appears that the reason it is always the same files is that the insert into ImageInformation uses the previous row as the source for the new import.  Would it be possible for import to do it from scratch each time instead of trying to be smart about it and failing consistently?

Line from the log:
INSERT INTO ImageInformation  (imageid, rating, creationDate, digitizationDate, orientation,   width, height, format, colorDepth, colorModel) SELECT 224073, rating, creationDate, digitizationDate, orientation,   width, height, format, colorDepth, colorModel FROM ImageInformation WHERE imageid=224056

(In reply to comment #14)
> I experienced this bug on import of photos and turned on the general log at
> that point.  Deleted and re-imported the files.  The same files were missing
> and it appears that the reason it is always the same files is that the
> insert into ImageInformation uses the previous row as the source for the new
> import.  Would it be possible for import to do it from scratch each time
> instead of trying to be smart about it and failing consistently?
> 
> Line from the log:
> INSERT INTO ImageInformation  (imageid, rating, creationDate,
> digitizationDate, orientation,   width, height, format, colorDepth,
> colorModel) SELECT 224073, rating, creationDate, digitizationDate,
> orientation,   width, height, format, colorDepth, colorModel FROM
> ImageInformation WHERE imageid=224056

(In reply to comment #14)
> I experienced this bug on import of photos and turned on the general log at
> that point.  Deleted and re-imported the files.  The same files were missing
> and it appears that the reason it is always the same files is that the
> insert into ImageInformation uses the previous row as the source for the new
> import.  Would it be possible for import to do it from scratch each time
> instead of trying to be smart about it and failing consistently?
> 
> Line from the log:
> INSERT INTO ImageInformation  (imageid, rating, creationDate,
> digitizationDate, orientation,   width, height, format, colorDepth,
> colorModel) SELECT 224073, rating, creationDate, digitizationDate,
> orientation,   width, height, format, colorDepth, colorModel FROM
> ImageInformation WHERE imageid=224056

I believe the missing row in "ImageInformation" is controlling as the query run when viewing a gallery joins to that table.  Such as:
SELECT DISTINCT Images.id, Images.name, Images.album,        ImageInformation.rating, Images.category,        ImageInformation.format, ImageInformation.creationDate,        Images.modificationDate, Images.fileSize,        ImageInformation.width, ImageInformation.height  FROM Images        INNER JOIN ImageInformation ON Images.id=ImageInformation.imageid  WHERE Images.status=1 AND Images.album = ?


I have yet to capture the initial insertion for the parent file into ImageInformation for why it failed.

Maik,

Your last changes in BQM about image scanner handling must fix this issue with 4.10.0.

Gilles

Gilles,

Possible that it is related with the blocked DB.

Maik

I think it's the problem.

I close file as resolved with 4.10.0 release. We will see the user feedback with this release.

Gilles Caulier