Bug 278927 - Not all files read (group items relevant)
Summary: Not all files read (group items relevant)
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Preview-Image (show other bugs)
Version: 2.0.0
Platform: Microsoft Windows Microsoft Windows
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-07-31 11:04 UTC by Rainer Lay
Modified: 2012-06-27 07:39 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 2.6.0


Attachments
picture, which is not displayed (660.97 KB, image/jpeg)
2011-07-31 11:05 UTC, Rainer Lay
Details
Screenshot (146.21 KB, image/png)
2011-07-31 11:27 UTC, Rainer Lay
Details
Screencopy of the content folder (134.25 KB, image/jpeg)
2011-12-24 07:37 UTC, mnaugendre
Details
Screencopy of the album view in Digikam (355.09 KB, image/jpeg)
2011-12-24 07:40 UTC, mnaugendre
Details
View of the album where pictures are missing (495.04 KB, image/jpeg)
2011-12-24 20:19 UTC, mnaugendre
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Rainer Lay 2011-07-31 11:04:08 UTC
Version:           2.0.0 (using KDE 4.5.4) 
OS:                MS Windows

In a directory, some files are not read, even after pressing f5.
The File could be jpg files or nefs, no difference. 
Even after renaming the files, they are not read (i.e. displayed).
The files can be displayed by other programs without problems or by showfoto by open the files manualy.
Some files cannt be read, while most of the files are read. An example file/picture is attached.

Reproducible: Sometimes

Steps to Reproduce:
put some picutures in a directory

Actual Results:  
most are displayed, some not

Expected Results:  
alle pictures are displayed

OS: WindowsNT (i686) release Windows XP
Compiler: cl.exe
Comment 1 Rainer Lay 2011-07-31 11:05:00 UTC
Created attachment 62377 [details]
picture, which is not displayed
Comment 2 Rainer Lay 2011-07-31 11:27:35 UTC
Created attachment 62378 [details]
Screenshot

the picture should be displayed in digikam, but is not.
In the windows explorer, one can see the picture.
Showfoto can display the picture.
Comment 3 Rainer Lay 2011-07-31 11:28:30 UTC
Comment on attachment 62378 [details]
Screenshot

In the explorer, the picture ist displayed, in digikam not.
Showfoto can display the picture
Comment 4 caulier.gilles 2011-07-31 11:45:17 UTC
Run kdebugdialog and turn on digiKam, kexiv2, and kdcraw debug space.

install and run debugview.exe and start digiKam. What did you see in debugview when you try to show the images.

Gilles Caulier
Comment 5 Rainer Lay 2011-07-31 13:25:24 UTC
After Pressing F5, I get the following lines

[6084] digikam(6084) void __thiscall KMap::ItemMarkerTiler::slotSourceModelReset(void): ----
[6100] Debug:kio_digikamalbums(6100)/digikam (kio-slave) void __thiscall kio_digikamalbums::special(const class QByteArray &): kio_digikamalbums::special  KUrl("digikamalbums:/00 Arbeitsordner/Import/?albumRoot=B%3A%2FBilder&albumRootId=2&databaseType=QSQLITE&databaseName=C%3A%2FDokumente%20und%20Einstellungen%2Frla%2FEigene%20Dateien%2FEigene%20Bilder%2FDigikamDB%2Fdigikam4.db&connectOptions=&hostName=&userName=&password=")
[6084] digikam(6084) void __thiscall KMap::ItemMarkerTiler::slotSourceModelReset(void): ----


Does this help?

kind regards, Rainer
Comment 6 Rainer Lay 2011-07-31 13:27:56 UTC
These lines after rebuilding the thumbnails in the album:

[6084] digikam(6084)/KDCRAW bool __cdecl KDcrawIface::KDcraw::loadEmbeddedPreview(class QImage &,const class QString &): Preview data size: 702441
[6084] digikam(6084)/KDCRAW bool __cdecl KDcrawIface::KDcraw::loadEmbeddedPreview(class QImage &,const class QString &): Using embedded RAW preview extraction
[6084] digikam(6084)/KEXIV2 enum KExiv2Iface::KExiv2::ImageOrientation __thiscall KExiv2Iface::KExiv2::getImageOrientation(void) const: Orientation => Exif.Image.Orientation =>  1
[6084] digikam(6084)/digikam (core) bool __cdecl Digikam::isJpegImage(const class QString &): mimetype =  "JPEG"
[6084] digikam(6084)/KEXIV2 enum KExiv2Iface::KExiv2::ImageOrientation __thiscall KExiv2Iface::KExiv2::getImageOrientation(void) const: Orientation => Exif.Image.Orientation =>  8


kind regards,
Rainer
Comment 7 Marcel Wiesweg 2011-07-31 13:50:51 UTC
Did you double-check you did not set a filter?
Comment 8 Rainer Lay 2011-07-31 16:26:02 UTC
Yep, and I pressed that cool "reset all filters" button.

Did I mention, that I am using 2.0.0rc?
Comment 9 Rainer Lay 2011-07-31 18:07:44 UTC
Is there any reason, why this bug isn't confirmed? The screenshot shows the case clearly, I think.
Comment 10 Marcel Wiesweg 2011-08-08 20:07:34 UTC
As expected, the photo is shown here.

What happens if you copy the file to a different directory? Is there any pattern which files are not shown?
Comment 11 Rainer Lay 2011-08-09 17:53:00 UTC
As of now, I didn't recognize any pattern.
It happens from time to time.
I am using a nikon D70s and a iphone 3. I couldn't remeber this problem with the iphone, but I do not take lot's of pictures with it.
It happens with jpg and with nef format too.

Once "it" happens, I can rename or move these files - they are still not read.

I thought the problem is resolved by pressing f5 "update" - but this doesn't help.
Comment 12 Marcel Wiesweg 2011-08-09 18:44:34 UTC
Is your database file very large? If possible you could send it to me via private mail for examination.
Comment 13 Rainer Lay 2011-08-31 10:02:52 UTC
With 2.0 (not 2.0rc), it is even worse: most of the time, I cannot see any pictures in the album!
Some times I see some, but most of the times not. As of now, I cannot give any hints, when the pictures are visible.
I can recreate the thumbnails for a album, even when the pictures are not visible. And I cannot see the thumbnails later on.
The picture sub menu is not visible (since digikam seems to not see any pictures.

This is really annoying!
Comment 14 caulier.gilles 2011-12-16 11:16:30 UTC
Rainer,

This file stil valid using digiKam 2.3.0 for windows ?

Gilles Caulier
Comment 15 Rainer Lay 2011-12-17 11:36:46 UTC
yep, this file is still valid for digikam 2.3 / windows.
Comment 16 mnaugendre 2011-12-24 07:37:52 UTC
Created attachment 67077 [details]
Screencopy of the content folder

The first 2 groups of pictures _913 and _944 are those ignored by Digikam
Comment 17 mnaugendre 2011-12-24 07:38:49 UTC
Same kind of problem, running Digikam 2.4.1 on Linux/Fedora.

Currently, there are 2 groups of pictures I cannot see in Dk any longer. I have seen them in the past, have worked on them: derawtised and produced a TIFF Darktable, then resizing to a JPEG in Digikam; I've also edited one of them in Gimp.

Then I started to play with the version group, in order to keep only one version in the main window and some groups completeley vanished from Digikam. They still are the same place in the folder, I can open them with any program; but Digikam doesn't seem to see them, and never display them.

There is no active filter. I've tried the F5 refresh function, thumbnail rebuilding, re-reading metadata from pictures, nothing seems to work; these pictures seem to be 'lost' for Digikam.
Comment 18 mnaugendre 2011-12-24 07:40:21 UTC
Created attachment 67079 [details]
Screencopy of the album view in Digikam

The two groups of pictures are nowhere to be seen
Comment 19 mnaugendre 2011-12-24 20:15:27 UTC
I tried to reupload the missing pictures from the folder they are in; Digikam detects they already exist, and ask for "ignorer/écraser" but whatever option I choose, the pictures still don't reappear!

I've made another screencopy showing the album view with the discrepancies in the number of pictures. See next attachment.
Comment 20 mnaugendre 2011-12-24 20:19:54 UTC
Created attachment 67088 [details]
View of the album where pictures are missing

The left panel says there are 42 pictures in this album.
But there are only 35 in the status bar, and in the album title, and no filter.
So 7 pictures are 'hidden' by Digikam.
Comment 21 Marcel Wiesweg 2011-12-26 21:24:36 UTC
If you open one of the images in the folder in the image editor, do you see the images appear in the editor's thumbnail bar?
Comment 22 Rainer Lay 2011-12-27 13:43:39 UTC
you mean "image editor" like showfoto? Yep, I can see a thumbnail.
You mean "image editor" like windows image ...? In the "windows bild und faxanzeige", I cannot see a thumbnail position at all.

If you are wondering, if there is a thumbnail in the exif data, yes there is:


# LANG=en exif.exe  DSc_6348.JPG
EXIF tags in 'DSc_6348.JPG' ('Motorola' byte order):
--------------------+----------------------------------------------------------
Tag                 |Value
--------------------+----------------------------------------------------------
Manufacturer        |NIKON CORPORATION
Model               |NIKON D70s
Orientation         |left - bottom
x-Resolution        |300.00
y-Resolution        |300.00
Resolution Unit     |Inch
Software            |Ver.1.00
Date and Time       |2011:07:23 19:04:30
YCbCr Positioning   |co-sited
Compression         |JPEG compression
x-Resolution        |300.00
y-Resolution        |300.00
Resolution Unit     |Inch
YCbCr Positioning   |co-sited
Exposure Time       |1/125 sec.
FNumber             |f/5.6
Exposure Program    |Normal program
Exif Version        |Exif Version 2.21
Date and Time (origi|2011:07:23 19:04:30
Date and Time (digit|2011:07:23 19:04:30
Components Configura|Y Cb Cr -
Compressed Bits per |1.00
Exposure Bias       |0.00 EV
MaxApertureValue    |4.80 EV (f/5.3)
Metering Mode       |Pattern
Light Source        |0
Flash               |Flash did not fire.
Focal Length        |65.0 mm
Maker Note          |27982 bytes undefined data
User Comment        |
SubsecTime          |90
SubSecTimeOriginal  |90
SubSecTimeDigitized |90
FlashPixVersion     |FlashPix Version 1.0
Color Space         |sRGB
PixelXDimension     |3008
PixelYDimension     |2000
Sensing Method      |One-chip color area sensor
File Source         |DSC
Scene Type          |1
CFA Pattern         |8 bytes undefined data
Custom Rendered     |Normal process
Exposure Mode       |Auto exposure
White Balance       |Auto white balance
Digital Zoom Ratio  |1.00
Focal Length In 35mm|97
Scene Capture Type  |Standard
Gain Control        |Normal
Contrast            |Normal
Saturation          |Normal
Sharpness           |Normal
Subject Distance Ran|Unknown
InteroperabilityInde|R98
InteroperabilityVers|0100
--------------------+----------------------------------------------------------
EXIF data contains a thumbnail (8965 bytes).
Comment 23 Marcel Wiesweg 2011-12-28 10:08:37 UTC
I mean digikam's editor, which appear when you select any photo in the album and click on "Edit" leftmost in the toolbar. All images in the current album are set on the thumbnail bar of the image editor window.
Comment 24 Rainer Lay 2011-12-28 11:02:23 UTC
Nope, in the "Vorschauleiste" (strg-t), the image is not visible too.
Comment 25 mnaugendre 2012-01-01 15:35:59 UTC
Same for me: it's impossible to open a missing picture in the editor, as it doesn't appear in a Digikam album.
I tried "open with ..." from Nautilus; it launches Digikam, but the picture I asked to open is not there.
Comment 26 Andreas Mair 2012-01-04 07:25:46 UTC
I've also the problem of images that disappeared after playing around with grouping/versioning. The problem exists in v2.3, v2.4 and v2.5. I've taken a look at the digikam4.db and found a solution for *my* problem. Maybe it helps you too.

First you need the sqlite3 tool to open digikam4.db (please work on a backed up DB!). I use pic.jpg and pic_v1.jpg as my example images.
$ sqlite3 digikam4.db
sqlite> select * from Images where name="pic.jpg" or name="pic_v1.jpg";
322|1|pic.jpg|1|1|2011-09-12T07:42:27|1632259|60c7b5d04416248f228720ed452c13b9
326|1|pic_v1.jpg|1|1|2011-12-22T13:05:31|475912|988fa2584f52daab90221891abf5b109
sqlite> select * from ImageRelations where (subject=322 and object=326) or (subject=326 and object=322);
322|326|2
326|322|2
sqlite> 

As you can see the images are "linked" in both directions, all working grouped images are only "one-way linked". Deleting one of that ImageRelations line brings back the missing images in Digikam.

It's quite easy to get that double linking in Digikam:
- Select two images
- Open the RMB context menu of the *first* image
- Execute "Group / Group selected here"
- If you can't see both pictures now, click the "expand" button to show them both again.
- If both images aren't selected, select them.
- Open the RMB context menu of the *other* image
- Execute "Group / Group selected here"
- Now both images should have the "versioned" symbol.
- Restart Digikam
- Images are "lost".
Comment 27 Marcel Wiesweg 2012-03-12 13:45:42 UTC
Git commit 09fee5e06d2f2f5a63e60083c917fd059f3c4eb9 by Marcel Wiesweg.
Committed on 12/03/2012 at 14:43.
Pushed by mwiesweg into branch 'master'.

Rewrite addToGroup to handle more cases, especially grouping image b on a where a
is currently grouped on b

M  +2    -1    NEWS
M  +43   -7    libs/database/imageinfo.cpp

http://commits.kde.org/digikam/09fee5e06d2f2f5a63e60083c917fd059f3c4eb9
Comment 28 Rainer Lay 2012-03-12 21:49:20 UTC
I checked the sql stuff from Andreas Mair.
The jpg file from comment #22 is not in the database at all.

So I don't think, that grouping stuff is directly related to my problem.
But I will look forward to test Marcels patch, when it is intergrated in the next win32 release to check, if it solves my problem too.

Kind regards,
Rainer
Comment 29 Rainer Lay 2012-03-12 21:51:04 UTC
I checked the sql stuff from Andreas Mair.
The jpg file from comment #22 is not in the database at all.

So I don't think, that grouping stuff is directly related to my problem.
But I will look forward to test Marcels patch, when it is intergrated in the next win32 release to check, if it solves my problem too.

Kind regards,
Rainer
Comment 30 Marcel Wiesweg 2012-03-13 11:17:01 UTC
Rainer: seems someone hijacked your bug ;-)
I found your db file you sent me and also found out how to unpack the 7z file. The two files have an entry in the Images table, but they are not listed because they do not have an entry in the ImageInformation table. Now I cannot tell you why this would happen, there should be no codepath in ImageScanner to this end. A crash while scanning should not result in such corruption because at least for SQLite, the process will be done in a transaction.
Comment 31 Rainer Lay 2012-03-13 21:51:09 UTC
Marcel: strange.
When I now look for the file ( select * from Images where name="DSC_6330.NEF";), I get no results.
When I do a find . -iname *.jpg. I get ~48K files, while select count(*) from Images; gives ~6k tupels (and yes, I once select scan at startup).
strange.
BTW, could you please change the status to something useful?