Bug 150938

Bengt,

First, 0.8.1 release is really old. Please, use last stable 0.9.2 release instead.

Second, NFS is currently not supported. This is not a problem from digiKam, but from SQlite database. There are a lots of reports about this subject. This problem will be fixed with digiKam 0.10.0 from KDE4

Third, how do you play with digikam3.db database file and NFS. Are you using this issue : http://www.digikam.org/?q=node/219

Gilles Caulier

Yes, I know this is an old version, but I suspect this behaviour
is the same on newer versions.

No, I did not use any workaround, but just enabled NFS locking with
the statd and lockd daemons, which are part of the standard NFS
implementation on FreeBSD. It just worked after that!

I think this bug report should not be seen as an NFS-only problem.
By fixing the obviously redundant work that digikam does, the
performance will be much snappier for everybody, especially on slower
hardware!

(Thanks for a great program, BTW!)

Marcel, 

What do you think about the ktrace informations from Bengt ? Something can be improved in DB interface ?

Gilles

A distinctive feature of SQLite is that it is in-process and working on a single file. At the same time it is a real database that supports transactions and ensures atomicity. What you see above is SQLite working, it needs to use file locking and journaling to fulfill its guarantees. And you see why we do not support keeping the SQLite DB on an NFS share.

Good news 1: In 0.10, you will be able to place the db in a different place than your pictures (in theory, needs to be implemented)
Good news 2: We now use Qt SQL module, which makes it easier to support MySQL in the future. Pending a lot of problems with SQL incompatibilities.

I get the impression that you exclude the possibility that the problem lies in how digikam interfaces to SQLite, or in the thumbnail display code. If so, why?

It would be interesting to redo my experiment, but not with ktrace, but instead running digikam in gdb with tracing on the SQLite calls. I'll see if I have the time to set this up the coming weekend.

Bengt

I just found and read the discussion on digikam on NFS from March this year. You discuss storing the db locally, but the pictures on NFS. Would that still enable accessing the album from multiple computers? If not, it defeats a major benefit of having the album on a server.

Bengt

I have investigated this further today. The culprit is that the image caption, tag names and rating are not kept in the ImageInfo class. When these values are needed, as they are for redisplaying the thumbnail view, e.g., due to scrolling, the DB is *always* queried, resulting in three DB queries per image. Sometimes this is even done twice per image.

I made a quick hack changing the ImageInfo class to cache the values for rating, caption and tag names. The result is that digikam becomes useable over NFS! There is still some time lag to start the display of a selected album, but there is no delay whatsoever in scrolling. Viewing with showimage is also an order of magnitude faster in switching to the next image.

There was one culprit due to that e.g., ImageInfo::rating() is declared const. I am a c++ newbie, but as I understand, it means that rating() can't update the instance variables. I wanted getItemRating to be called on demand, but that was not possible without removing "const". Since that had consequences in other places, the hack calls getItemRating when the ImageInfo instance is created.

I am willing to work further on this, but would like to get some feedback first!

Bengt

Hi Bengt,

you have made an impressive and important progress!
I don't know about the rating() issue, but still you
could attach the patch to this report, so that
Marcel or Gilles  can comment ...

Many thanks, Arnd

P.S.: Maybe your findings are even related to
http://bugs.kde.org/show_bug.cgi?id=151122 ...

Thanks for the encouragement! At some point I will propose a patch, but I experimented with version 0.8.1, and patches against that is a bit useless.
(0.9.2 didn't compile on my box - I didn't want to go throught the hassle of upgrading ports on my FreeBSD 6.2-REL machine. Thats usually a couple of day's work.)

I have however made some more thinking to optimise this further. If the db queries can be aggregated so that all image info can be retrieved in *one* SQL query, it will be another big performance gain. I will make some more experimentation with this, but perhaps not until next weekend. Even further improvements might be possible for large albums if the db info for all images are retrieved in a single query, but this is a tradeoff between the time it takes for the intial display, and display of subsequent images when you start scrolling.

Now I'm off buying a C++ book so I can improve my C++ skills...  I need to understand the implications of using or not using "const" in method declarations in order to make sure I don't propose something that will do more harm than good!

Bengt

Bengt you are right and I am fully aware that caching can be improved. This will be improved in 0.10, but not backported to 0.9, this is an area of large changes. I am just currently finding out how to best build the cache in the ImageInfo objects, and in 0.10 ImageInfos as a whole will be shared under the hood (only one object per image across the application)

The const problem is solved using const_cast, this seems all right, because lazy caching is one of the fields where people say const_cast can be used.

Nevertheless, you are curing symptoms, but the disease is that SQLite over NFS is extremely slow. A database call is relatively slow, slower than looking up in a cache, so we need to cache, but digikam is based on the db, so we cannot support situations where the db is really slow.

A second case you mention is "accessing the album from several computers". I understand that as "sharing the database file"? No of course that does not work when the db file is stored locally.
One solution would be to support MySQL for such cases. Concurrent access is still a different story, but what you want is to access your pictures and your database from several computers at different times. If we supported MySQL this would solve your problems. In 0.10 we use Qt SQL as database backend, so support for MySQL is no problem in principle. What needs to be done is to test/adapt our SQL code with knowledge of the faint differences in supported SQL (I do not have this knowledge)

Created attachment 21954 [details]
Patch adding a simple db cache to ImageInfo class

This very simple patch makes digikam 0.8.1 quite usable over NFS. It provides
more than an order of magnitude improvement in performance. Sorry for not
hacking this in a later version. I anyway thought that it could be useful for
someone. It should be straightforward to re-implement in 0.9.x. I will do that
when I have upgraded my system, if none beats me to it...

SVN commit 732444 by cgilles:

digiKam from trunk (KDE4) : First very important improvement here depending of the new Database interface/schema dedicaced to 0.10.0 release.

A new setting in album configuration page have been added to host the database file used by digiKam in a dedicaced place.
Before, the SQlite3 database file (digikam3.db) used by digiKam to store all image informations like Tags, Rating, Comments, Date, etc... 
been store in the root album path. This solution introduce a big problem to use a root album on a network file system like NFS, because 
SQLite3 do not like it (look here for more information : http://www.sqlite.org/faq.html#q7)

The only issue for this problem is to store the databse file in a separate place...

This is want mean than with this commit, digiKam can use a remote root album...

There is a screenshot of the new Album config page :

http://digikam3rdparty.free.fr/Screenshots/digikamKDE4_13.png

The next pending important change to do will to be able to drive more than one root album path with digiKam 0.10.0... 

CCMAIL: digikam-devel@kde.org

CCBUGS: 122516
CCBUGS: 150938
BUG: 129437
BUG: 137694



 M  +72 -58    digikam/albumsettings.cpp  
 M  +3 -0      digikam/albumsettings.h  
 M  +1 -1      digikam/digikamapp.cpp  
 M  +3 -2      digikam/welcomepageview.cpp  
 M  +72 -12    utilities/setup/setupgeneral.cpp  
 M  +5 -1      utilities/setup/setupgeneral.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=732444

This file can be considerated as fixed with KDE4, since network repository support is implemented.

Now digiKam database file is not stored on netwrok path but always in local, as it explained to the Collection setup page:

http://www.flickr.com/photos/digikam/3038025046/sizes/o/

I close this file now.

Gilles Caulier

That sounds great. How do I set this up so I can access my photo
album from multiple computers, as that is my purpose with storing
the photo album on an NFS server?

Bengt

Bengt,

It's very easy. Go to digiKam Configuration dialog and Collection settings page:

http://www.flickr.com/photos/digikam/3038025046/sizes/o/ 

For the rest the collection setup is very clear : digiKam database path must be set on local, and all collections paths (local, remote, network, removable, etc...) can be set like you want, one by one.

Gilles