Bug 317210 - THUMBDB : delete image removes file, but does not remove thumbnails
Summary: THUMBDB : delete image removes file, but does not remove thumbnails
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Thumbs (show other bugs)
Version: 3.1.0
Platform: Ubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-03-22 22:44 UTC by Marty
Modified: 2017-02-09 20:54 UTC (History)
11 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.5.0
Sentry Crash Report:


Attachments
debug console when deleting, and reattempting delete. (7.95 KB, text/plain)
2013-03-22 22:45 UTC, Marty
Details
open digikam, delete image, fail recognise, close digikam (49.03 KB, text/plain)
2013-03-27 21:33 UTC, Marty
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Marty 2013-03-22 22:44:32 UTC
I delete a photo from thumbnail preview. It does not report any error, but i notice that the thumbnail is still present.

I try to delete it again, and it reports that the image is missing.

So now the image is not present, and i cant see anyway to remove it.

I have a mysql DB, and the files are located on remote nfs server.

Reproducible: Always

Steps to Reproduce:
1. Go to thumbnail view of images.
2. select image and move to wastebasket.
3. The thumbnail is still present.
Actual Results:  
The image is gone on filesystem, but thumbnail is still present in album, even after restart of digikam.

Expected Results:  
The image and the thumbnails are removed from album.
Comment 1 Marty 2013-03-22 22:45:40 UTC
Created attachment 78300 [details]
debug console when deleting, and reattempting delete.
Comment 2 caulier.gilles 2013-03-23 06:32:12 UTC
No problem is relevant of Database which host thumbnails images...

Gilles Caulier
Comment 3 Marty 2013-03-23 11:37:35 UTC
Looking in the db directly, i can still see it. Should it be gone?

mysql> select * from Images where name='20130322_liam_9828.JPG';
+-------+-------+------------------------+--------+----------+---------------------+----------+----------------------------------+
| id    | album | name                   | status | category | modificationDate    | fileSize | uniqueHash                       |
+-------+-------+------------------------+--------+----------+---------------------+----------+----------------------------------+
| 61593 |   876 | 20130322_liam_9828.JPG |      1 |        1 | 2013-03-22 18:04:50 |  4627091 | 93356016d4136a3c7385acb4f7a55c5c |
+-------+-------+------------------------+--------+----------+---------------------+----------+----------------------------------+
1 row in set (0.00 sec)


mysql> select * from UniqueHashes where uniqueHash='93356016d4136a3c7385acb4f7a55c5c';
+----------------------------------+----------+---------+
| uniqueHash                       | fileSize | thumbId |
+----------------------------------+----------+---------+
| 93356016d4136a3c7385acb4f7a55c5c |  4627091 |   76927 |
+----------------------------------+----------+---------+
1 row in set (0.00 sec)
Comment 4 Marty 2013-03-23 11:39:56 UTC
Perhaps its also useful to know this is ubuntu 12.10 quantal with 2.8 previously. I noticed this problem, so upgraded to 3.0 using Philip Johnsson PPA hoping newer build fixed it.
Comment 5 Marcel Wiesweg 2013-03-24 18:22:02 UTC
I dont see debug output from digikam, only kio which is irrelevant. Please enable 50003 in kdebugdialog. Does pressing F5 fix the issue (=> force rescan)?
Comment 6 Marty 2013-03-24 19:08:22 UTC
Oh, you are right. Pressing F5 makes the image from the gallery disapear :)

But, changing albums, and even restarting digikam does not!

Is this expected bahaviour? Do you still want the debug output?
Comment 7 Marcel Wiesweg 2013-03-25 18:30:19 UTC
Yes, please give a full debug output starting the program, delete one image (not recognized), close.
At least on Linux, file watch should work absolutely flawless, we use inotify directly.
Comment 8 Marty 2013-03-27 21:31:42 UTC
OK, so added image "test.JPG" and closed digikam.

Started debug console, and digikam. Deleted image from digikam. Waited 5 seconds for it to do something. Tested UI, it was responsive and ready. Closed digikam.

Attached is the output from console.
Comment 9 Marty 2013-03-27 21:33:33 UTC
Created attachment 78435 [details]
open digikam, delete image, fail recognise, close digikam
Comment 10 Marty 2013-03-27 21:38:12 UTC
Also, a couple of days ago i went to create the above log, and it actually recognised the deletes. I almost fell of my chair. But, now its not working again. There has not been any changes to packages in the last few days. If i see it recognising the deletes again, will capture the console.
Comment 11 Marcel Wiesweg 2013-03-31 20:08:47 UTC
Comment on attachment 78435 [details]
open digikam, delete image, fail recognise, close digikam

Opening inotify looks all right, but the deletion is not recognized. Don't know why at the moment.
Comment 12 DrSlony 2013-04-22 01:56:25 UTC
Same here. Pressing F5 also removes the thumbnail from the strip, but if I happen to be focused on a thumbnail I previously deleted then after pressing F5 focus jumps back to the first image in the strip, so I can't use the "next image" shortcut, which makes this an annoying bug.

digiKam version 3.1.0
Exiv2 can write to Jp2: Yes
Exiv2 can write to Jpeg: Yes
Exiv2 can write to Pgf: Yes
Exiv2 can write to Png: Yes
Exiv2 can write to Tiff: Yes
Exiv2 supports XMP metadata: Yes
LibCImg: 130
LibClapack: external shared library
LibExiv2: 0.23
LibJPEG: 80
LibJasper: 1.900.1
LibKDE: 4.10.2
LibKExiv2: 2.3.0
LibKGeoMap: 2.0.0
LibKdcraw: 2.2.0
LibLCMS: 119
LibPGF: 6.12.27 - external shared library
LibPNG: 1.5.13
LibQt: 4.8.4
LibRaw: 0.15.0-Beta1
LibTIFF: LIBTIFF, Version 4.0.2 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Marble Widget: 0.15.1 (stable version)
Parallelized PGF codec: No
Parallelized demosaicing: No
RawSpeed codec support: No
Database backend: QSQLITE
Kipi-Plugins: 3.1.0
LibKface: 2.0.0
LibKipi: 2.0.0
LibOpenCV: 2.4.3
Libface: 0.2
Comment 13 caulier.gilles 2014-08-29 22:27:45 UTC
*** Bug 269577 has been marked as a duplicate of this bug. ***
Comment 14 caulier.gilles 2014-08-31 13:11:16 UTC
*** Bug 297546 has been marked as a duplicate of this bug. ***
Comment 15 Luca Carlon 2015-08-21 01:32:37 UTC
No update on this issue? It is pretty bothering.
Comment 16 DrSlony 2015-08-21 05:14:50 UTC
Haven't noticed it in recent versions.
Comment 17 Marty 2015-08-21 07:15:43 UTC
I also have not observed this on recent builds. As the original requestor, i think this can be resolved. Unless others can still reproduce.
Comment 18 Luca Carlon 2015-08-21 09:50:24 UTC
I experienced it in git master actually... but I can't reproduce at the moment.
Comment 19 Alister 2015-09-09 14:37:59 UTC
I appear to get this problem too. It been in existence for a long time.
If you try to delete a second time it complains that the file isn't there but then the thumbnail disappears.
Even worse though, when deleting photos that are apparently still there, digikam then deletes a DIFFERENT (real) file and its impossible to restore it.

Camera - Nikon CoolpixS9100 connected in PTP mode.
Comment 20 DrSlony 2015-09-09 15:56:08 UTC
Fixed by 4.12.0 as far as I can tell.
Comment 21 Petr Vorel 2015-10-08 22:06:23 UTC
Not for me. I still experience the described behaviour.
Comment 22 swatilodha27 2016-06-04 11:41:38 UTC
I use digikam5.0.0-beta6

Steps to reproduce:
1)Create a new folder and add some images.
2)Go to the thumbnail view.
3) Move an image to trash.

Actual results
The image is removed from file system and the thumbnail doesn't exist anymore in digiKam.

Expected results
Same as actual results.
Comment 23 caulier.gilles 2016-06-04 11:46:15 UTC
When you said that the thumbnail do not exists, you want mean in thumbnail DB ?

It's true for all DB type ?

I'm surprised because a garbage collector to clean up DB entries about non valid items have never been implemented, as i know...

Perhaps with an item removed from DK interface it will work, but what about to remove file form hard drive by another application while DK session is down. I'm sure that nothing is removed from the DB. But this case is fully relevan of a future garbage collector tool for DB maintenance (i think another bugzilla entry exist about this topic).

Gilles Caulier
Comment 24 swatilodha27 2016-06-04 11:50:48 UTC
(In reply to caulier.gilles from comment #23)

I understand you now. I think I misinterpreted the issue faced by the user. I just checked if thumbnail is removed the DK interface. The file still exists in my Image table. 

I apologize for the mistake.
Comment 25 DrSlony 2016-06-04 22:02:23 UTC
Using 7fb769c27cf48408cf45cad65ba4330940663c05

I imported over 100 photos. Back in the main digiKam window, once importing finished, I went to the new album, clicked on the last thumb, jumped 50 photos back using the "left arrow" key, then I held shift and clicked on the first photo in the album, to select the first >50, then I shift+deleted them. The confirmation popped up, I agreed, and digiKam deleted the photos... only the thumbnails have not disappeared.
https://i.imgur.com/8bHLn3j.png

Clicking F5 reloaded the album and then the deleted photos disappeared.
Comment 26 DrSlony 2016-06-04 22:05:02 UTC
Using the SQLite database.
Comment 27 caulier.gilles 2016-06-04 22:11:06 UTC
SO it's just a refresh problem. Right ?

Gilles Caulier
Comment 28 caulier.gilles 2016-06-04 22:12:29 UTC
Can you run digiKam from a console and post here the trace of operations when you try again a similar deletion of files ?

Gilles Caulier
Comment 29 DrSlony 2016-06-04 22:53:20 UTC
Yes, looks like just a refresh problem.

I tried to reproduce the problem from a console now by just copying an existing album under a new name, starting digiKam and deleting photos from the new album, but the thumbs were correctly removed from digiKam, so I couldn't reproduce. I will run it from a console the next time I import and delete and report back to you.
Comment 30 Greg 2016-06-27 08:42:09 UTC
Thumbnails remain after file deletion in version 5.0.0.0.Beta5 on Fedora Linux 24:
Version     : 5.0.0
Release     : 0.10.beta5.fc24

For example, I have 127 images in a directory. Thumbnails of all original photos appear. After I process each image, I save a new version, e.g., pix-01.jpg saved as pix-01_v1.jpg. Later I remove the original photos, select pix-01.jpg and move it to Trash. 

The original file is moved to Trash, but the thumbnail remains in Album view.
When I try to delete the file a second time, an error message appears:

  File pix-01.jpg does not exist

When I try View -> Refresh, placeholder images (a bare tree at sunset?) appear for thumbnails. Thumbnails and associated metadata seem to remain in any case.

Also, additional directories appear within my current directory in the Album menu (left sidebar). They show different numbers of images. Now the left sidebar shows four subdirectories ("albums") in the album I'm using, but actually there are none.
  
The bug and another one (missing sidebar menu icons) appeared after I upgraded Fedora 23 to 24 and then digiKam from version 4.x to 5.0.0.Beta5).
Comment 31 Greg 2016-06-27 08:42:47 UTC
Thumbnails remain after file deletion in version 5.0.0.0.Beta5 on Fedora Linux 24:
Version     : 5.0.0
Release     : 0.10.beta5.fc24

For example, I have 127 images in a directory. Thumbnails of all original photos appear. After I process each image, I save a new version, e.g., pix-01.jpg saved as pix-01_v1.jpg. Later I remove the original photos, select pix-01.jpg and move it to Trash. 

The original file is moved to Trash, but the thumbnail remains in Album view.
When I try to delete the file a second time, an error message appears:

  File pix-01.jpg does not exist

When I try View -> Refresh, placeholder images (a bare tree at sunset?) appear for thumbnails. Thumbnails and associated metadata seem to remain in any case.

Also, additional directories appear within my current directory in the Album menu (left sidebar). They show different numbers of images. Now the left sidebar shows four subdirectories ("albums") in the album I'm using, but actually there are none.
  
The bug and another one (missing sidebar menu icons) appeared after I upgraded Fedora 23 to 24 and then digiKam from version 4.x to 5.0.0.Beta5).
Comment 32 caulier.gilles 2016-07-23 09:07:03 UTC
Swati,

For this file, outside the fact that a DB garbage collector need to be write to clean up the database (there is one other bug entry about this topic), it must miss a call to a sql query somewhere in thumbnails database interface when file is removed from the collection. 

Gilles Caulier
Comment 33 swatilodha27 2016-07-23 17:49:23 UTC
Using digiKam 5.0.0, I did the following with MySQL DB:

1)Add photos in a new album.
2)In thumbnails view, move multiple images to Trash.

In UI: The images are no more visible in Thumbnails view.

In DB: 
After adding images, they're visible correctly in Images table.
After deletion, images still present in DB but with album column = "NULL". If we restore these images, then NULL entries remains in the table, and new rows are added for those images.
Comment 34 caulier.gilles 2016-07-23 18:06:01 UTC
It's clear. The table in DB must be cleaned properly. That all..

Probably the same dysfunction must exist with sqlite.

Gilles Caulier
Comment 35 caulier.gilles 2016-07-23 18:08:08 UTC
What's happen if you delete an album as well ? All item entries in DB are properly removed ?

Gilles Caulier
Comment 36 swatilodha27 2016-07-25 20:15:05 UTC
I think that what happens now is partially right. 'Cause when an item is "Moved to trash", it is just removed and not permanently deleted. So it is right to exist in table with album=NULL

The issue lies when that item is "permanently" deleted and there's no change in the Images table.
Comment 37 Mario Frank 2017-02-02 09:07:20 UTC
Hi, I just uploaded a patch in https://bugs.kde.org/show_bug.cgi?id=374591 that introduces garbage collection as maintenance stage. This cleans the core, thumbs and recognition db from stale entries. But be advised: The patch is still in testing phase. I did some tests on my own databases and had no problems. But just to be sure, backup your databases before you test.
Comment 38 caulier.gilles 2017-02-04 14:08:09 UTC
New 5.5.0 AppImage is done with garbage database collector patches.
Uploading to GDrive is under progress. It will be online in few minutes at usual
place :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

New database Garbage Collector options are there :

https://www.flickr.com/photos/digikam/32549923912/in/dateposted-public/
https://www.flickr.com/photos/digikam/32549923632/in/dateposted-public/

Gilles Caulier
Comment 39 Mario Frank 2017-02-09 20:54:12 UTC
Git commit 834221c8d1300bbdd4ae78e345cdc0fce3e92be8 by Mario Frank.
Committed on 09/02/2017 at 20:51.
Pushed by mfrank into branch 'master'.

Updated the news. The newly introduced garbage collection gets rid of thumbnails in DB that are not referenced by any image any more.
FIXED-IN: 5.5.0

M  +2    -1    NEWS

https://commits.kde.org/digikam/834221c8d1300bbdd4ae78e345cdc0fce3e92be8