Version: 0.8.1 (using KDE KDE 3.5.8) Installed from: FreeBSD Ports OS: FreeBSD Displaying and scrolling the thumbnails in an album is very slow when the album is residing on NFS. To investigate this problem, I created an album with only two pictures, and made the window small enough to not fit both pictures simultaneously. I then ran ktrace on digikam, and scrolled the window from top to bottom once. I found out this from the ktrace: - it calls access(2) on digikam3.db-journal 27 times (just to find out that the file did not exist) - it reads digikam3.db 27 times - it opens files /var/tmp/sqlite_XXXXXXXXXXXX 18 times - it calls fcntl(2) on digikam3.db with F_SETLCK 108 times When the album is on an NFS filesystem, most of the above operations require an RPC to the server, highlighting the performance problem. It seems to me that digikam just makes a lot of unnecessary work! (This problem might be the same as in Bug 135845.)
Bengt, First, 0.8.1 release is really old. Please, use last stable 0.9.2 release instead. Second, NFS is currently not supported. This is not a problem from digiKam, but from SQlite database. There are a lots of reports about this subject. This problem will be fixed with digiKam 0.10.0 from KDE4 Third, how do you play with digikam3.db database file and NFS. Are you using this issue : http://www.digikam.org/?q=node/219 Gilles Caulier
Yes, I know this is an old version, but I suspect this behaviour is the same on newer versions. No, I did not use any workaround, but just enabled NFS locking with the statd and lockd daemons, which are part of the standard NFS implementation on FreeBSD. It just worked after that! I think this bug report should not be seen as an NFS-only problem. By fixing the obviously redundant work that digikam does, the performance will be much snappier for everybody, especially on slower hardware! (Thanks for a great program, BTW!)
Marcel, What do you think about the ktrace informations from Bengt ? Something can be improved in DB interface ? Gilles
A distinctive feature of SQLite is that it is in-process and working on a single file. At the same time it is a real database that supports transactions and ensures atomicity. What you see above is SQLite working, it needs to use file locking and journaling to fulfill its guarantees. And you see why we do not support keeping the SQLite DB on an NFS share. Good news 1: In 0.10, you will be able to place the db in a different place than your pictures (in theory, needs to be implemented) Good news 2: We now use Qt SQL module, which makes it easier to support MySQL in the future. Pending a lot of problems with SQL incompatibilities.
I get the impression that you exclude the possibility that the problem lies in how digikam interfaces to SQLite, or in the thumbnail display code. If so, why? It would be interesting to redo my experiment, but not with ktrace, but instead running digikam in gdb with tracing on the SQLite calls. I'll see if I have the time to set this up the coming weekend. Bengt
I just found and read the discussion on digikam on NFS from March this year. You discuss storing the db locally, but the pictures on NFS. Would that still enable accessing the album from multiple computers? If not, it defeats a major benefit of having the album on a server. Bengt
I have investigated this further today. The culprit is that the image caption, tag names and rating are not kept in the ImageInfo class. When these values are needed, as they are for redisplaying the thumbnail view, e.g., due to scrolling, the DB is *always* queried, resulting in three DB queries per image. Sometimes this is even done twice per image. I made a quick hack changing the ImageInfo class to cache the values for rating, caption and tag names. The result is that digikam becomes useable over NFS! There is still some time lag to start the display of a selected album, but there is no delay whatsoever in scrolling. Viewing with showimage is also an order of magnitude faster in switching to the next image. There was one culprit due to that e.g., ImageInfo::rating() is declared const. I am a c++ newbie, but as I understand, it means that rating() can't update the instance variables. I wanted getItemRating to be called on demand, but that was not possible without removing "const". Since that had consequences in other places, the hack calls getItemRating when the ImageInfo instance is created. I am willing to work further on this, but would like to get some feedback first! Bengt
Hi Bengt, you have made an impressive and important progress! I don't know about the rating() issue, but still you could attach the patch to this report, so that Marcel or Gilles can comment ... Many thanks, Arnd P.S.: Maybe your findings are even related to http://bugs.kde.org/show_bug.cgi?id=151122 ...
Thanks for the encouragement! At some point I will propose a patch, but I experimented with version 0.8.1, and patches against that is a bit useless. (0.9.2 didn't compile on my box - I didn't want to go throught the hassle of upgrading ports on my FreeBSD 6.2-REL machine. Thats usually a couple of day's work.) I have however made some more thinking to optimise this further. If the db queries can be aggregated so that all image info can be retrieved in *one* SQL query, it will be another big performance gain. I will make some more experimentation with this, but perhaps not until next weekend. Even further improvements might be possible for large albums if the db info for all images are retrieved in a single query, but this is a tradeoff between the time it takes for the intial display, and display of subsequent images when you start scrolling. Now I'm off buying a C++ book so I can improve my C++ skills... I need to understand the implications of using or not using "const" in method declarations in order to make sure I don't propose something that will do more harm than good! Bengt
Bengt you are right and I am fully aware that caching can be improved. This will be improved in 0.10, but not backported to 0.9, this is an area of large changes. I am just currently finding out how to best build the cache in the ImageInfo objects, and in 0.10 ImageInfos as a whole will be shared under the hood (only one object per image across the application) The const problem is solved using const_cast, this seems all right, because lazy caching is one of the fields where people say const_cast can be used. Nevertheless, you are curing symptoms, but the disease is that SQLite over NFS is extremely slow. A database call is relatively slow, slower than looking up in a cache, so we need to cache, but digikam is based on the db, so we cannot support situations where the db is really slow. A second case you mention is "accessing the album from several computers". I understand that as "sharing the database file"? No of course that does not work when the db file is stored locally. One solution would be to support MySQL for such cases. Concurrent access is still a different story, but what you want is to access your pictures and your database from several computers at different times. If we supported MySQL this would solve your problems. In 0.10 we use Qt SQL as database backend, so support for MySQL is no problem in principle. What needs to be done is to test/adapt our SQL code with knowledge of the faint differences in supported SQL (I do not have this knowledge)
Created attachment 21954 [details] Patch adding a simple db cache to ImageInfo class This very simple patch makes digikam 0.8.1 quite usable over NFS. It provides more than an order of magnitude improvement in performance. Sorry for not hacking this in a later version. I anyway thought that it could be useful for someone. It should be straightforward to re-implement in 0.9.x. I will do that when I have upgraded my system, if none beats me to it...
SVN commit 732444 by cgilles: digiKam from trunk (KDE4) : First very important improvement here depending of the new Database interface/schema dedicaced to 0.10.0 release. A new setting in album configuration page have been added to host the database file used by digiKam in a dedicaced place. Before, the SQlite3 database file (digikam3.db) used by digiKam to store all image informations like Tags, Rating, Comments, Date, etc... been store in the root album path. This solution introduce a big problem to use a root album on a network file system like NFS, because SQLite3 do not like it (look here for more information : http://www.sqlite.org/faq.html#q7) The only issue for this problem is to store the databse file in a separate place... This is want mean than with this commit, digiKam can use a remote root album... There is a screenshot of the new Album config page : http://digikam3rdparty.free.fr/Screenshots/digikamKDE4_13.png The next pending important change to do will to be able to drive more than one root album path with digiKam 0.10.0... CCMAIL: digikam-devel@kde.org CCBUGS: 122516 CCBUGS: 150938 BUG: 129437 BUG: 137694 M +72 -58 digikam/albumsettings.cpp M +3 -0 digikam/albumsettings.h M +1 -1 digikam/digikamapp.cpp M +3 -2 digikam/welcomepageview.cpp M +72 -12 utilities/setup/setupgeneral.cpp M +5 -1 utilities/setup/setupgeneral.h WebSVN link: http://websvn.kde.org/?view=rev&revision=732444
This file can be considerated as fixed with KDE4, since network repository support is implemented. Now digiKam database file is not stored on netwrok path but always in local, as it explained to the Collection setup page: http://www.flickr.com/photos/digikam/3038025046/sizes/o/ I close this file now. Gilles Caulier
That sounds great. How do I set this up so I can access my photo album from multiple computers, as that is my purpose with storing the photo album on an NFS server? Bengt
Bengt, It's very easy. Go to digiKam Configuration dialog and Collection settings page: http://www.flickr.com/photos/digikam/3038025046/sizes/o/ For the rest the collection setup is very clear : digiKam database path must be set on local, and all collections paths (local, remote, network, removable, etc...) can be set like you want, one by one. Gilles