Bug 233222

Andi, 

If i'm not too wrong, there is already an entry in bugzilla about this subject. Right ?

Gilles Caulier

Hmm don't know at the moment... I never experienced such issues though.

See bug 210353.

Quentin, which images are affected by your problem?

Sorry, but how do you want me to describe which image?  Do you mean in what view?

File name, file type etc. Or simply upload two images here (or if too big at some other file hoster) that have mixed up thumbnails.

Hi, thanks, however I can't upload the images with incorrect thumbnails as they have correct thumbnails.  It's only in Digikam that they show up incorrect, as in part of how the database must be choosing the thumbnail.  At least that's how I understand it.  If I'm wrong, let me know and I'll sort something out.

Just upload the original files. Digikam generates some signature fore the files to associate them with the thumbnails. We have to check why this fails for your images.

Created attachment 42494 [details]
Photo showing digikam incorrect thumbnail display

Screenshot of how digikam displays incorrect thumbnails.  The four thumbnails are all of the girl on the left, but it has taken a thumbnail from another image (seemingly at random).  Note this is not always the same between the album view and the thumbnail view across the top or bottom after you've double clicked the picture, though it is in this case.  From left to right the images are as follows:
IMG_4033.CR2, IMG_4033.jpg, IMG_4033.JPG, IMG_4033.ppm

Have uploaded the original file as you requested to: http://www.filefactory.com/file/b0gc005/n/IMG_4033.jpg note it has a download link at the bottom, took me a while to find it :)

I've gone and taken some more photo's this afternoon, and now it's gone and replaced that particular images thumbnail (and many many other images, in fact almost all the jpg's) with another thumbnail of the ones I took this afternoon.

Add ppm files to that too, as in they also have incorrect thumbnails.

I assumed as well this was the same problem as 210353 but if it affects images with different file size and different creation date, even JPGs, then it's a different problem.

Same here, Digikam 1.1.0 on Gentoo

Some additional remarks:
1. It does not seem to be a problem with the file itself. If I have the problem with file x.jpg and close digikam, cp x.jpg x-2.jpg, open digikam, then the problem still exists for x.jpg, but not for x-2.jpg
2. Thumbnail is not the only problem. In album overview, these pictures also have displayed file size 0 bytes and 0x0 pixels. But it's also not a problem with Digikam being unable to open the file, because when hovering over the picture with the mouse, the right exif date, camera, etc is displayed.

Perhaps some incorrect information in the internal database? But when starting digikam from console, no error is displayed, even when checking the pics with incorrect information. Oh, and "Extras => Create Fingerprints again" (or whatever stands there in english) does not make a difference for me, same pictures with wrong thumbnails/information.

One last remark: When closing Digikam, I always get this on the console. Don't know if this is in any way related to the bug.

Model deleted: Marble::MarbleModel(0x2853010)
Deleting FileStorageWatcher
Model deleted: Marble::MarbleModel(0x2fc13e0)
Deleting FileStorageWatcher
QSqlDatabasePrivate::removeDatabase: connection 'digikamDatabase-31690112' is still in use, all queries will cease to work.
QSqlDatabasePrivate::removeDatabase: connection 'digikamDatabase-30821504' is still in use, all queries will cease to work.
QSqlDatabasePrivate::removeDatabase: connection 'digikamDatabase-49818352' is still in use, all queries will cease to work.
QSqlDatabasePrivate::removeDatabase: connection 'thumbnailDatabase-31590304' is still in use, all queries will cease to work.
QSqlDatabasePrivate::removeDatabase: connection 'thumbnailDatabase-49818352' is still in use, all queries will cease to work.
QSqlDatabasePrivate::removeDatabase: connection 'digikamDatabase-31590304' is still in use, all queries will cease to work.

Very likely related:
https://bugs.kde.org/show_bug.cgi?id=217422

For further testing, please choose any two perfectly normal pictures a and b to which your problem applies, that is, a and b both show the thumbnail of a.

1) When you restart digikam, is still the wrong thumbnail shown everytime?
2) When you copy these two to a different album, do you still get the wrong thumbnail?
3) When you "touch" these files on the console and then start digikam, still wrong thumbnail?

Ok, I'll try to do this systematically. For this, I'll stick with the picture names you used: Pic A is the picture where the thumbnail comes from. Pics B (C, D, ...) are the ones that use the same thumbnail, although they shouldn't.

First, some general regards:
1. A and B can be in different albums.
2. For all pics B,C,D, the thumbnail from the same pic A is always used.
3. As stated in #13, pics B,C,... display wrong information regarding file size and dimensions. This also goes for Pic A.
4. Now I've also found one pic using its correct thumbnail, but displaying wrong meta information in album view. Wrong info means here: Dimension 0x0px, file size 8192B. See [1] at the end for details. I have only found one pic so far, but that does not necessarily mean its the only one.

Regarding your questions:
1) Yes.
2) Copying pic B via Digikam (Drag'n'drop to different album, select "copy") results in the following: still wrong thumb for original B, correct thumb for new B-2, correct file size for B-2, still wrong picture properties as stated below in [1]. Copying or moving B-2 in Digikam did not change anything, restarting Digikam didn't change anything either.
3) Touching the files and restarting Digikam afterwards helped, this goes for pic B as well as pic B-2.


[1]: Album view, attribute tab on the right side, picture selected
File properties: Correct (name, date, etc) except file size (8192B instead of 664663B)
Picture properties (Bildeigenschaften):
Type: JPEG
Dimensions: Unknown
Color depth: 0 bpp
Color model: Unknown
Photo properties (Fotoeigenschaften): Seem to be correct again

Please open your database with the sqlite3 tool and post the output, for file B from above:
SELECT * FROM IMAGES WHERE name='B.jpg';
The very first number is the imageid:
SELECT * FROM ImageInformation WHERE imageid=<imageid>;
SELECT * FROM ImageMetadata WHERE imageid=<imageid>;

Some updates:

Thumbnail has changed since yesterday, although I've got no idea how or why. The new thumb comes from a pic with correct information, contrary to my "general regard 2" from post #16.

SQL:
select * from Images where name="B";
24661|704|20100304-113715.jpg|1|1|2010-04-11T23:49:42|0|

select * from Images where name="C"; (some other broken pic)
24665|704|20100304-115802.jpg|1|1|2010-04-11T23:49:43|0|

select * from Images where name="A"; (where the thumb comes from this time)
24651|704|20100304-110423.jpg|1|1|2010-05-17T22:22:08|548506|8561f30172a715a01944648ba6c101b3

select * from Images where name="X"; (general regard 4 from #16, correct thumb, wrong information)
24649|704|20100304-105855.jpg|1|1|2010-04-11T23:49:38|8192|3442640930948d35e69223cd4ed15e4a

select * from ImageInformation where imageid=24661; (B)
24661|-1|2010-03-04T11:37:15|2010-03-04T11:37:15|1|0|0|JPG|0|0

select * from ImageMetadata WHERE imageid=24661;
24661|Canon|Canon DIGITAL IXUS 70|5.8 - 17.4 mm|4.9|17.4||0.0166666666666667||0|200|24|0||5||


So, obviously, the file size is 0 for the broken pics, and no "unique" hash exists. For picture X there's a wrong entry in the DB regarding file size.

Perhaps some kind of racing condition? I did not download these pics with digikam, but just copied them into my album directory and started up digikam. When downloading with Digikam, i presume the program will process all new pics sequentially, as copying over usb is (comparatively) slow. Perhaps its sth. else when digikam starts up and finds a bunch of new subdirectories with pictures, parallel processing, problems with database access?

select * from Images where fileSize=0;
24119||20090911-165411.JPG|3|1|2010-04-11T23:47:11|0|
24143||20090922-181612.JPG|3|1|2010-04-11T23:47:27|0|
24169||20090922-191348.JPG|3|1|2010-04-11T23:47:37|0|
24200||20090923-182640.JPG|3|1|2010-04-11T23:47:47|0|
24236||20090928-121616.JPG|3|1|2010-04-11T23:47:59|0|
24240||20090928-121918.JPG|3|1|2010-04-11T23:48:00|0|
24243||20090928-140727.JPG|3|1|2010-04-11T23:48:01|0|
24267||20090928-195526.JPG|3|1|2010-04-11T23:48:10|0|
24268||20090928-195531.JPG|3|1|2010-04-11T23:48:10|0|
24276||20091003-201834.JPG|3|1|2010-04-11T23:48:16|0|
24277||20091003-201842.JPG|3|1|2010-04-11T23:48:16|0|
24500||20100410-202025.JPG|3|1|2010-04-11T23:49:55|0|
24512|688|20091025-140101.jpg|1|1|2010-04-11T23:48:35|0|
24518|688|20091025-141745.jpg|1|1|2010-04-11T23:48:39|0|
24522|688|20091025-142518.jpg|1|1|2010-04-11T23:48:42|0|
24523|688|20091025-142529.jpg|1|1|2010-04-11T23:48:43|0|
24525|688|20091025-143020.jpg|1|1|2010-04-11T23:48:44|0|
24526|688|20091025-143218.jpg|1|1|2010-04-11T23:48:45|0|
24528|688|20091025-143948.jpg|1|1|2010-04-11T23:48:46|0|
24529|688|20091025-144738.jpg|1|1|2010-04-11T23:48:47|0|
24535|688|20091025-150629.jpg|1|1|2010-04-11T23:48:51|0|
24537|688|20091025-150706.jpg|1|1|2010-04-11T23:48:52|0|
24540|688|20091025-150740.jpg|1|1|2010-04-11T23:48:54|0|
24541|688|20091025-151116.jpg|1|1|2010-04-11T23:48:55|0|
24543|688|20091025-151914.jpg|1|1|2010-04-11T23:48:56|0|
24558|691|20091110-205447.jpg|1|1|2010-04-11T23:49:06|0|
24559|691|20091110-205554.jpg|1|1|2010-04-11T23:49:07|0|
24561|691|20091110-205747.jpg|1|1|2010-04-11T23:49:08|0|
24562|691|20091110-205821.jpg|1|1|2010-04-11T23:49:09|0|
24577|692|20091120-200141.jpg|1|1|2010-04-11T23:49:14|0|
24624|698|20091229-133409.jpg|1|1|2010-04-11T23:49:29|0|
24661|704|20100304-113715.jpg|1|1|2010-04-11T23:49:42|0|
24665|704|20100304-115802.jpg|1|1|2010-04-11T23:49:43|0|
24685|704|20100304-153125.jpg|1|1|2010-04-11T23:49:49|0|
24687|704|20100304-165143.jpg|1|1|2010-04-11T23:49:50|0|


select * from Images where fileSize > 0 and fileSize < 9000;
[deleted some pics with correct entries from here]
24117||20090911-165217.JPG|3|1|2010-04-11T23:47:10|8192|ff73e503e5887c4f600027d7a319913a
24120||20090911-165759.JPG|3|1|2010-04-11T23:47:12|8192|11a69b1d6e9e7c08f19b1d61d1a952e6
24146||20090922-182909.JPG|3|1|2010-04-11T23:47:29|8192|cba61275a9623f9ca52bf4cdd4a5c922
24161||20090922-191153.JPG|3|1|2010-04-11T23:47:34|8192|14ab02155ffc5a0753a57775a635d7e0
24164||20090922-191212.JPG|3|1|2010-04-11T23:47:35|8192|26275c250a9fa50db538ee2842ae460c
24179||20090923-092642.JPG|3|1|2010-04-11T23:47:40|8192|c9f8439bfa63da8077aa9848285e69d6
24226||20090928-111127.JPG|3|1|2010-04-11T23:47:55|8192|5c8e456340d83a4e689c420c04ac227c
24230||20090928-113638.JPG|3|1|2010-04-11T23:47:57|8192|9948dae3452562adca1fc6fbbf672225
24233||20090928-121542.JPG|3|1|2010-04-11T23:47:58|8192|9e4a05418ecb92ae3204832f94d578df
24257||20090928-163009.JPG|3|1|2010-04-11T23:48:06|8192|83005354bec8996c52f6eea27264d010
24282||20091021-205151.JPG|3|1|2010-04-11T23:48:19|8192|3baea9d45b3863f94f016044aaec61b1
24283||20091021-205200.JPG|3|1|2010-04-11T23:48:20|8192|fb7cf043754d9b99f0f8edad088686d8
24285||20091021-205233.JPG|3|1|2010-04-11T23:48:21|8192|5f9c385443e9890ac864fab3ce0d38c4
24286||20091021-205240.JPG|3|1|2010-04-11T23:48:22|8192|e28b2f8d0c990e533d6137fab8c3a79e
24288||20091021-205254.JPG|3|1|2010-04-11T23:48:23|8192|358afc84f7ddf34887643ac2d2f2d971
24289||20091021-205410.JPG|3|1|2010-04-11T23:48:24|8192|3ffb1a48082586401bb7f25ebe2758a8
24291||20091021-210739.JPG|3|1|2010-04-11T23:48:25|8192|d86eeefb5e8e5819532a527e7bbe85c8
24292||20091021-210756.JPG|3|1|2010-04-11T23:48:26|8192|2d2024c406d4bb4226e97fa64b3bc1b9
24294||20091021-211059.JPG|3|1|2010-04-11T23:48:27|8192|157b55b4ce4f00def01458d0aa642598
24295||20091021-211206.JPG|3|1|2010-04-11T23:48:28|8192|c54a66b25bdedf9473a8a5c76d9e2fda
24297||20091021-211626.JPG|3|1|2010-04-11T23:48:29|8192|e207486ed8a6028e1eed746362130594
24298||20091021-211633.JPG|3|1|2010-04-11T23:48:30|8192|507c5b2408644cbdb3cc10dd810dcb5c
24300||20091021-211752.JPG|3|1|2010-04-11T23:48:31|8192|d48e5cb1c20f0bdcb4c2dfa6674fb8e5
24301||20091021-211800.JPG|3|1|2010-04-11T23:48:32|8192|3b2add3b512df1aa81a900e9f0159cf4
24486||20100410-140444.JPG|3|1|2010-04-11T23:49:51|8192|9fd3feae01c6c1010b23982e995426d3
24505||20100410-213546.JPG|3|1|2010-04-11T23:49:57|8192|014e4bc26324aae6d16ed4caf81dd736
24509|688|20091025-135348.jpg|1|1|2010-04-11T23:48:33|8192|427bf61ce4831534b746638bcb61ffc4
24510|688|20091025-140046.jpg|1|1|2010-04-11T23:48:34|8192|106a4d5b2755288861349cfba97dd922
24513|688|20091025-140121.jpg|1|1|2010-04-11T23:48:36|8192|960f7b459af6b1bd5bf1231ec6c82c8a
24516|688|20091025-141023.jpg|1|1|2010-04-11T23:48:38|8192|6c82b5c1ce9abf8fc8996f7ea62a8b39
24519|688|20091025-141839.jpg|1|1|2010-04-11T23:48:40|8192|5bd92437de2d3a0d82b420ec1e19035b
24520|688|20091025-141844.jpg|1|1|2010-04-11T23:48:41|8192|f81e0e0d4237216b8d9179f2671675c7
24531|688|20091025-145555.jpg|1|1|2010-04-11T23:48:48|8192|db30a57b9f686ca66a99df0daf6b1ed6
24532|688|20091025-150356.jpg|1|1|2010-04-11T23:48:49|8192|bf4c010b63b18c7f554fa22b3cd8ec9f
24533|688|20091025-150406.jpg|1|1|2010-04-11T23:48:50|8192|dfbe107451ca37b6d45970e29d398e96
24538|688|20091025-150716.jpg|1|1|2010-04-11T23:48:53|8192|2f87e5f1b2fc7913e77c145fc7917638
24545|688|20091025-151927.jpg|1|1|2010-04-11T23:48:57|8192|fccc99e92e588b8a83bd6eb358f185af
24546|688|20091025-151932.jpg|1|1|2010-04-11T23:48:58|8192|2c1d770230df10c03703b8a30b3a9dd2
24552|690|20091031-141332.jpg|1|1|2010-04-11T23:49:02|8192|aba28a504a27f304b80997fb602f3502
24553|690|20091031-141337.jpg|1|1|2010-04-11T23:49:03|8192|7c46cc3d870da85310240fad59bebf22
24594|694|20091128-234019.jpg|1|1|2010-04-11T23:49:19|8192|96eb3a89f245434edd340b6186b72dba
24617|696|20091220-153227.jpg|1|1|2010-04-11T23:49:27|8192|6262cff4650f075f40db8afef5867045
24620|696|20091220-153309.jpg|1|1|2010-04-11T23:49:28|8192|8208937fae66876f7471f672aecd8932
24649|704|20100304-105855.jpg|1|1|2010-04-11T23:49:38|8192|3442640930948d35e69223cd4ed15e4a
24659|704|20100304-112916.jpg|1|1|2010-04-11T23:49:41|8192|7567fb0379a04f45d7235bd6b0adb32d

Are you absolutely sure that digikam was not running in the background while you copied these images?
That would be quite important here because I indeed assume something is happening while scanning. Scanning a file which has a file system entry but is not completely copied yet would nicely explain some problems.

Couldn't this easily be checked by deleting the according image entry from the database and let digikam re-scan it?

@Simon: That will most likely produce a new correct entry in the DB, as it seems that the problem has nothing to do with the images itself. See for example #16, touching the file and restarting digikam does solve the problem for that file.

@Marc: I'm nearly 100% sure that Digikam was not running while copying. But besides... it does not make a difference. At least here Digikam does not scan "live". If I copy (cp, not digikam-copy) a new file to an album directory, Digikam does not react in any way (just tested this here to be sure). To get the new pic into the album, I have to scan for new pictures explicitely (which is by default done on digikam startup here).

Yes KDirWatch is notoriously broken (or crashes, alternatively) so digikam may not react to extern file operations. Nonetheless I dont see any reason why at digikam startup, file scanning should see files with size 0 or 8192 unless they are currently being written and really have this size. In a short test, the cp command seems to use a buffer size of 8192. Looking at the code of uniqueHash generation, it will return a null array if it fails to read any bytes from a file.
The scanning code in digikam is identical for startup scan or scanning while running.

So this looks like a race condition between a cp operation and a simultaneously running file scanner. But I'm unsure how to prevent this. Short research revealed no reliable method to find out if a file is currently being written to.

SVN commit 1130153 by mwiesweg:

Regard a file as modified when its file size has changed, even if the modification
date did not change.

This seems necessary to prevent problems like bug 233222 to persist: Now the file
may still be wrongly scanned during the copy operation, but at least at next
rescan it will be corrected. Moreover it seems reasonable to assume a changed file
when the file size changed (some operations will restore the modification date).

Overhead: For my collections of 30000 images, scanning took 1550 msec without the change
and 1750 including the check, with hot caches. With cold caches, the scan took
9300 msec in both cases, no significant difference here.

Note: At first start with this check, there may be a good number of files rescanned.

CCBUG: 233222
CCMAIL: digikam-devel@kde.org

 M  +10 -0     collectionscanner.cpp  
 M  +5 -6      imagescanner.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1130153

Hm, I'm still not convinced that the cp operation is the reason for the error... still sure that cp operation and sync to HDD were already finished in my case. But as long as Digikams behaviour gets better this way and the false thumbnails are gone, I'm happy :-) Question is: When will Digikam discover that the file size has changed? Ok, at least at next rescan. But what if I open up Digikam, new pics are scanned in, possibly with wrong filesize and thumb again, and then I open one of the new albums and click on a picture. Is this sufficient?

Additional comments and thoughts:
1. Is it in any way possible to find out whether a certain file is currently opened for write operations by another process? I didn't find anything like that in "man 2 stat", but only had a short look. One other solution could be that Digikam opens each _new_ file for writing and immediatly closes it again without modification. As long as only one process may open each file for writing at a time, this would result in the appropriate error code that could be checked. Of course, there are (at least) two preconditions: mtime of the file must not be modified, and it mustn't slow down Digikams scanning operation. And, of course, this cannot be applied to read-only file systems (different error code?).

2. Assume we have a file with 0 bytes length and name x.jpg. Even with the version patched by Marcel this will result in a DB entry without uniqueHash and with wrong thumbnail. Especially the latter is cleary an error and just mustn't happen. What about a special treatment for corrupt jpg files (or whatever other picture format)? I think the thumbnail error could be avoided if every file, even empty ones, get a uniqueHash. Though I don't know how unique this will be for empty files... at least this way they could perhaps point to a special thumbnail indicating "this file is corrupt".

> Question is: When will Digikam discover that
> the file size has changed? Ok, at least at next rescan. But what if I open up
> Digikam, new pics are scanned in, possibly with wrong filesize and thumb again,
> and then I open one of the new albums and click on a picture. Is this
> sufficient?

For now, pressing F5 (Refresh) should do the trick.

> Additional comments and thoughts:
> 1. Is it in any way possible to find out whether a certain file is currently
> opened for write operations by another process? I didn't find anything like
> that in "man 2 stat", but only had a short look. One other solution could be
> that Digikam opens each _new_ file for writing and immediatly closes it again
> without modification. As long as only one process may open each file for
> writing at a time, this would result in the appropriate error code that could
> be checked. Of course, there are (at least) two preconditions: mtime of the
> file must not be modified, and it mustn't slow down Digikams scanning
> operation. And, of course, this cannot be applied to read-only file systems
> (different error code?).

I have also researched this problem on the net, and your suggestion can be found there. I'm not completely convinced that it will work, maybe we try. I dont know about modification date in that case.

> 
> 2. Assume we have a file with 0 bytes length and name x.jpg. Even with the
> version patched by Marcel this will result in a DB entry without uniqueHash and
> with wrong thumbnail. Especially the latter is cleary an error and just mustn't
> happen. What about a special treatment for corrupt jpg files (or whatever other
> picture format)? I think the thumbnail error could be avoided if every file,
> even empty ones, get a uniqueHash. Though I don't know how unique this will be
> for empty files... at least this way they could perhaps point to a special
> thumbnail indicating "this file is corrupt".

Yes, a file of size 0 should be a clear warning sign to the scanning process. We need some mechanisms in place there to handle such problems. (for whatever reason, this problem never came up over the years. I was wondering why but accepted that it worked. Obviously, it does not)

*** Bug 217422 has been marked as a duplicate of this bug. ***

*** Bug 240945 has been marked as a duplicate of this bug. ***

Created attachment 52694 [details]
WrongSize thumbnail incomplete

Hi all, I am from bug #240945, and now using digikam 1.4 with Ubuntu 10.10.

This does not occurs all the time, but let say one in a while.

I edit a picture with GIMP, when I am back from it, I see something like the attachement. The original picture is ok, and for the diff one, the thumbnail is half there, and the size is not correct.

Using the F5 key, does nothing.

My comment #29 still occur with 1.6

*** Bug 272725 has been marked as a duplicate of this bug. ***

Quentin,

This file still valid using digiKam 2.4 ?

Gilles Caulier

New digiKam 4.11.0 is available :

https://www.digikam.org/node/740

Can you reproduce the problem with this release ?

Not reproducible with 4.12.0