Version: 1.0.0-rc1 (using KDE 4.3.4) OS: Linux Installed from: Ubuntu Packages I had used 1.0.0-beta5 to set the ratings on thousands on images, but when I checked the JPEG files, the metadata had not been updated in most of them. In 1.0.0-rc1, if I change the metadata on a single image, it seems to be written OK, so I don't know if this is a problem that has been fixed. Anyway, the issue now is that I have thousands of image files that are missing their metadata. I got into this situation a year or two ago and sync'd the image metadata to the database--unaware that the metadata was missing--and lost thousands of tags. In order to fix the metadata, I'm going through each album, checking that the metadata (from the database) looks OK, selecting the images in the album, and then clicking the menu item "Image" -> "Write Metadata to Selected Images". This works fine; the metadata is updated correctly in the files. However, if there are a large number of selected image files, the process becomes extremely slow. There appears to be a non-linear relationship between the number of files and the time it takes to write the metadata in RC1: 39 files -> 2 seconds 71 files -> 4 seconds 87 files -> 9 seconds 120 files -> 43 seconds 145 files -> 38 seconds 155 files -> 44 seconds 276 files -> 132 seconds 484 files -> 312 seconds 1,200 files -> I went for lunch. Throughout this time, digiKam uses about 70-90% CPU. I have turned on the option to update the file modification time when the metadata is written. This will make it easier for me to spot when digiKam is not updating the files properly in the future. Turning off this option makes a big difference. For example, I updated 120 files in 43 seconds and the same operation took 38 seconds when I simply repeated it immediately, but when I turned off the file modification time update, the third run took 15 seconds. Similarly, the album with 484 images took 312 seconds when file modification time stamps were updated and 87 seconds when the time stamps were not updated. Yesterday, when I was not really timing things, the end of the 1,200+ file update seemed to hang. However, when I went to the console and looked at the timestamps on the files, I could see that about five files would have their time stamps updated and then there would be a long wait (maybe 30-60 seconds) with no apparent changes and then five more files would be updated and then nothing for another while, and so on until it eventually finished. I can see about 30 images in the thumbnail view at one time. When I select all images in an album of, say, 100 files and update the metadata, the progress bar quickly hits about 20-30% and then pauses while all of the thumbnail images disappear and, one-by-one, slowly appear again. After that the progress bar slowly moves on to 100%. Is digiKam re-scanning an album each time it detects a file time stamp change, even when it is making those changes itself on another thread? That would explain the kind of behaviour I see. Is this likely to have a big effect on the performance of other things like batch processing?
Yes you have right. A process run from KDELibs and use KDirWatch API to check items changed. I think file date time is handle in by it. I have no idea how to change this behavior for this moment. Anyway your investigation are very interesting for the future. we will take a care about... Gilles Caulier
It just occurred to me that while I did not time the 1,200+ file metadata update yesterday, my filesystem did. There were 1,283 files and the difference between the earliest and the latest modification time stamp is 30 min 19 sec. So you can add this to my list of timings: 1,283 files -> 1,819 seconds These were from an older 2MP camera with an average files size of about 800kB. Would it be possible to stop digiKam from reacting to notifications from KDirWatch while it has a background operation in progress? Perhaps digiKam could simply store up these notifications and then apply the appropriate ones when it has finished the first job. It does not sound easy to figure out what "appropriate" would be, though. In the meantime, I'll disable the modification time stamp updates, write the metadata to the files, run "touch" from the command line and then turn on the modification time stamp updates again, as I won't be doing any more large batches of images again for a while.
stopping KDirWatch events will have side-effect, especially if you use batch tool to sync metadata with database. This tool is non modal dialog. It run in background and you can switch to main gui and continue to work. Now imagine than folder can non receive events from KDirWatch when you working on with icon-view... Gilles Caulier
I'm imagining.... It would be much faster, wouldn't it? ;-) I could always just hit the refresh key when I am interested in an update. I have found a slightly easier way around this. If I leave on the modification time stamp updates (which is what I want), I can avoid the long waits by selecting an album, clicking "Album" -> "Write Metadata to Images" and then, once the job starts, selecting a different album folder and waiting. I did that and it took 4.5 mins to update 935 files. The file-watching and scanning only seems to apply to the album currently being viewed, so viewing a different album from the one where the metadata changes are under way seems to avoid the issue. I prefer to wait in the second album, because if I start a second job to update the metadata, the progress of the first job cannot be monitored any more. Once the second job completes, the progress bar is cleared even though I can still hear the disk thrashing away on the first job (and see the time stamps changing on the files from the console). Maybe you could integrate the progress with the KDE notifications widgets that can track several jobs at once (like in the Dolphin file manager).
>Maybe you could integrate the progress with the KDE >notifications widgets that can track several jobs at once (like in >the Dolphin file manager). Already done with batch tool to write metadata, not yet with single album action... Gilles Caulier
Probable explanation for the fact that changing the album helps: A change of modification date will also trigger recreation of the thumbnail.
Debian has a similar report (bug #571811): Package: digikam Version: 2:1.1.0-1 at the moment, digikam is very slow on writing metadata to images. I select round about 20 JPGs between 3 and 7 MB, modify the description and click "apply". The update of the internal DB is fast, but the following write opration to the files is very, very slow, up to 5 minutes. I'm not sure when this issue occured the first time, a few weeks ago everything run fine. There's an open Bug at KDE: https://bugs.kde.org/show_bug.cgi?id=218633. I'm unsure if this is the same issue. I've deselected "update modification time" with no result. iotop doesn't show any unusual, 0 B/s write and read. top reports digikam at 100% CPU. If you need any additional information, please let me know.
Hello, this issue is still present in 1.2.0 (using in Debian unstable and experimental). Interesting, in gwenview the same thing happens. Is it possible, that libexiv causes these problems? Setting and managing of metadata is a key feature of digikam. Is there any chance of getting it back? Nearly distressed, but with kind regards Michael
Yes, this file is fully relevant of Exiv2 library. For Info, I CC Andreas who manage this project. Q : which Exiv2 version you use exactly ? Go to Help/ Components Info for details. Gilles Caulier
Exiv2 0.19 has a known performance issue with images from some Nikon SLRs. It is _very_ slow for these. Images from any other cameras are not affected. Just from reading the descriptions above, comment #7 might be related to this problem, and it should be easy to determine whether that is really the case. If it is, upgrading to exiv2 from SVN trunk will help. For any other (new) performance issue, if you think it is related to exiv2, please narrow the test down to exiv2 and report to the exiv2 bugtracker at dev.exiv2.org preferably. A simple test could be to run a more or less complex update command over a number of images, e.g., something like this: $ time exiv2 -M'set Exif.Image.ImageDescription Some description' *.jpg and compare with the time digiKam takes for a similar operation and with the time a simple cp takes to just copy the files within the same directory. Ideally, they should all be pretty close with cp-time < exiv2-time < digiKam-time Andreas
@9 from Gilles: I'm using Debian unstable's version 0.19. @10 from Andreas: I've checked with some Panasonic Lumix images, this works very well. I'm shooting with a Nikon D90 SLR, which fits into your description. I'm going to open a bug at Debian's libexiv2-6 package for an update to the trunk version of libexiv. If this doesn't help, I'll fill a bug at exiv2.org. Comment #7 was opened by me at bugs.debian.org and forwarded by Fathi :-) As suggested, some timings for manipulating 10 images: Panasonic Lumix: real 0m0.467s user 0m0.128s sys 0m0.204s Nikon D90: real 1m41.809s user 1m35.098s sys 0m0.516s Many thanks! Kind regards, Michael.
This looks like a duplicate of Bug 224094 Forwarded upstream: http://dev.exiv2.org/issues/show/677
exiv2 was recently updated to 0.20, which should not contain this bug any more.
Yes, This file must be fixed using Exiv2 0.20.0. I close it now. Re-open if necessary Gilles Caulier
*** Bug 237104 has been marked as a duplicate of this bug. ***
I think my bug report was hijacked by comment #7. My original problem has nothing to do with Nikon images and Exiv2 problems, I shoot Canon. The issue is that enabling updates to the file modification times causes problems when DigiKam's own changes to images causes DigiKam to rescan the images and generates new thumbnails even though it is unnecessary. Can we wind this bug back to comment #6? Is the problem as described originally and in comments #1-#6 fixed, or was this bug marked as "FIXED" because the unrelated issue with Nikon images and Exiv2 was fixed?