Created attachment 24382 [details]
some searches are very slow

the colors in this screenshot represent:

red - those search areas are not working anymore
yellow - the used settings for the search "top rated", it takes up to 15
seconds to display 3 images that have 5 stars
green - those searches / filters work fine

edit: red in the screenshot doesn't mean that they are not working anymore, but that those filters are VERY slow with the settings shown in the yellow dialog

Andi,

Are you compiled yourself digiKam or are you used pre-compiled packages ?

Gilles Caulier

I compiled from SVN (beta4), I don't use the official packages from archlinux because they don't have digikam-0.9.4beta in it...

Andi,

Are you some error messages on the console ?

Gilles Caulier

No, the only messages I get are written to the console during the startup:

Found dcraw version: 8.83
KIPI (loading): WARNING: KIPI::PluginLoader:: createInstanceFromLibrary returned 0 for JPEGLossless (kipiplugin_jpeglossless) with error number 3
KIPI (loading): WARNING: KLibLoader says: libkdcraw.so.2: cannot open shared object file: No such file or directory
KIPI (loading): WARNING: KIPI::PluginLoader:: createInstanceFromLibrary returned 0 for Viewer (kipiplugin_viewer) with error number 3
KIPI (loading): WARNING: KLibLoader says: libkdcraw.so.2: cannot open shared object file: No such file or directory
KIPI (loading): WARNING: KIPI::PluginLoader:: createInstanceFromLibrary returned 0 for SendImages (kipiplugin_sendimages) with error number 3
KIPI (loading): WARNING: KLibLoader says: libkdcraw.so.2: cannot open shared object file: No such file or directory
KIPI (loading): WARNING: KIPI::PluginLoader:: createInstanceFromLibrary returned 0 for SlideShow (kipiplugin_slideshow) with error number 3
KIPI (loading): WARNING: KLibLoader says: libkdcraw.so.2: cannot open shared object file: No such file or directory
KIPI (loading): WARNING: KIPI::PluginLoader:: createInstanceFromLibrary returned 0 for RawConverter (kipiplugin_rawconverter) with error number 3
KIPI (loading): WARNING: KLibLoader says: libkdcraw.so.2: cannot open shared object file: No such file or directory

Andi,

No chance for you... your installation is fully broken. I can't understand how digiKam can start as well.

You must uninstall all (libkipi, libkdcraw, likexiv2, kipi-plugins, and digiKam) and re-compile and re-install properlly.

Gilles Caulier

Strange... but why did it happen since the upgrade of sqlite3? All the other digikam functions seem to work normal.

Ok, I reinstalled everything from the archlinux repository, so now I'm running digikam 0.9.3 again. No error messages in the console, BUT the searching for images with a given rating is still messed up!
It must be an issue with sqlite3-3.5.8....

I checked with another distribution (sidux): I installed sqlite3-3.5.8 and now all searches for a special rating are very slow... so the issue must be the sqlite upgrade...

I checked with the sqlite experts. They pointed out that
"Version 3.5.8 includes some important new performance optimizations", see
http://www.sqlite.org/news.html, and that it might be possible that
these slow things down for some reason.

One suggestion was to use analyze, http://www.sqlite.org/lang_analyze.html

You could try this in the following way:
a) make a copy of digikam3.db
   (which is in the directory below which all images directories are)
b) Then run 
     sqlite3 digikam3.db
   and issue the command
     analyze;
   Quit via
     .quit
  
Does this improve things?
(The change written to the database can be checked by comparing a
string dump: 
   sqlite3 digikam3.db_ORIG
   .output before.dump
   .dump
   .quit
and
    sqlite3 digikam3.db
   .output after.dump
   .dump
   .quit
    
So if this analyze thing does not improve speed, we need
to figure out, which searches  are slow in terms of 
their sql statements.
For this you would need to modify
   digikam/kioslave/digikamsearch.cpp
by adding a
  kdWarning() << "SQL Query " << sqlQuery << endl;
in the line before each of the two calls to m_db.execSql.

Andy, do you think you can do this?

Best, Arnd

Sure... I know some C++, did my diploma thesis in it... so I guess this will be no problem... :-) I will try it tomorrow, now it is too late... I will post my results in here...

Oh forgot to ask: Should I use the code for 0.9.3 or the current SVN repository for KDE3?

Andi,

Please, use the current SVN repository for KDE3. Thanks in advance.
Gilles Caulier

Hi Andi,

well, why not go for svn... ;-)
See http://www.digikam.org/?q=download/svn
in particular below "Install digiKam in your Home Directory")

Best, Arnd

Just to outline the strategy I have in mind to hunt down the problem:
a) isolate one sql statement corresponding to a slow search
b) use the sqlite3 command line tool to reproduce that the search is slow
   (i.e. we would get rid of using digikam itself)
c) then compare on the same machine with the previous version of sqlite3.
d) put the database somewhere to download so that Gilles/Marcel/I can
   try to test/reproduce the problem on our machines
e) report back to the sqlite team

Thanks a lot Andi, that you have a look at this. If there is really
a problem with sqlite, this would strike many digikam users in the near future
if 3.5.8 is installed widely!

OK I'll use SVN, I just asked because I recompiled 0.9.3 against the new sqlite3 some days ago so 'make' will run through a lot faster... :-)

It is impossible for me to compile current SVN tree. When I come to the greycstoration-plugin, the compiler "hangs" and nothing is happening anymore. It just keeps on using 100% CPU and nothing actually happens. After 1 hour I canceled make and replayed CImg 1.2.7, but then compilation fails because some functions are not recognized anymore. But it doesn't hang, so maybe CImg is broken??
Has anybody the same problem compiling greycstoration? Or is there a way to skip it?

Hi Andi,

never had that type of problem.
It should be possible to disable that by
editing digikam/imageplugins/Makefile  and removing
`restoration` from the list of SUBDIRS

Good luck,

Arnd

No, it is still trying to compile greycstoration... I know that compilation of SVN worked for my 10 days ago (with the old CImg version), but the current tree is not compiling... grrrml... :-)

I just want to test the sql queries and now nothing is working anymore. My old digikam is not working because of the new SVN libs, and the current version is not compiling... hmmm... never had such problems with compiling software... this is so typical: you really need to compile a piece of software once only and right then all hell brakes loose... :-)
I will give it another try tomorrow... 

sorry, I was too quick in my previous reply, presumably it is
the one in digikam/libs/Makefile where  `greycstoration`
needs to be kicked out from SUBDIRS.

I am not sure, why things get so complicated for your system,
but I am optimistic that the compile will eventually work -
Thanks a lot for your patience and effort!

Well I removed 'restoration' from digikam/imageplugins/Makefile AND 'greycrestoration' from digikam/libs/Makefile, but then showfoto is complaining that it needs libgreyc... or something like that. If I also remove showfoto from digikam/Makefile, my build system is REALLY getting angry :-)

Well if it is not working for me tomorrow, I will set up a little virtual machine and test if compilation works in there. If it does, well then I don't know what is wrong, but maybe I will be able to get to the actual testing needed for this bugreport :-)

AAAHH!! What's wrong now? I downloaded 0.9.4beta3 from sf.net because SVN is not working, but now I can't see the SQL-Query in the console. Am I doing something wrong? I compiled digikam using

./configure --prefix=/opt/kde --enable-debug=full

and added the line Arnd mentioned above:

        QStringList values;
        QString     errMsg;
        kdWarning() << "SQL Query " << sqlQuery << endl;
        if (!m_db.execSql(sqlQuery, &values, &errMsg))
        {
            error(KIO::ERR_INTERNAL, errMsg);
            return;
        }

But no output is printed to the console. Is this the wrong position or do I have to configure something else? I also changed kdWarning() to kdDebug() but still no output.

Weird with the svn, anyway, you are getting closer.... ;-)
Note that there are two places in digikamsearch.cpp
with a m_db.execSql. Did you add the output to both?

The other possibility coming to my mind is that the
paths have to be set properly and a kdeinit has to be issued
(i.e., it could be that still the old kioslave is active,
but I am not sure on this, which I always considered to be a mess ... ;-):

export DIGIKAMDEST=here_the_path_used_on_configure..... 
export KDEDIRS=$DIGIKAMDEST:/usr
export LD_LIBRARY_PATH=$DIGIKAMDEST/lib:$LD_LIBRARY_PATH
export LDPATH=$DIGIKAMDEST/lib:$LDPATH
export PATH=$DIGIKAMDEST/bin:$PATH
kdeinit

HTH, Arnd

I've added the line in the two places... yes... I also thought that the old kioslave could be in use, I even restarted my laptop, but still no output. But I will try your solution mentioned above...

Hi,

after my compile odyssey I'm happy to present some results... finally.
I was not able to compile the SVN version, I don't know why but greycstoration just ignored my attempts to compile it.
So I downloaded the 0.9.4beta3 from sf.net and tried to compile this version, and IT WORKED!! Yeah!
Here is my little bugreport, hope it is somehow useful to you.

I did not build kipi-plugins and the other related libs by myself because I don't want to mess up my system too much.
I installed most of the packages from the testing repository of archlinux, only libkdcraw had to be build from SVN.
These are the libs and programs I'm using for the tests in this bugreport:
    
    exiv2-0.16
    libkexiv2-0.1.6
    kipi-plugins-0.1.5beta1
    libkipi-0.1.5
    dcraw-1.399
    libkdcraw (current SVN)
    sqlite3-3.5.8
    digikam-0.9.4beta3
    KDE-3.5.9
    
and some build tools used for compiling:
    
    autoconf-2.62
    automake-1.10.1
    gcc-4.3.0
    libtool-2.2
    m4-1.4.10
    make-3.81
    patch-2.5.4
    
Hope all important libs were mentioned above, if not just ask.

So first of all I did the sqlite analyze stuff but it did not fix the problem. I created the two dumps, but it doesn't seem to change a lot in the database structure. I don't know if it is useful but here is the diff:

=========================
before.dump -> after.dump
=========================
> INSERT INTO "Settings" VALUES('Scanned','2008-04-25T00:00:11');
> ANALYZE sqlite_master;
> INSERT INTO "sqlite_stat1" VALUES('Settings','sqlite_autoindex_Settings_1','4 1');
> INSERT INTO "sqlite_stat1" VALUES('Searches','sqlite_autoindex_Searches_1','4 1');
> INSERT INTO "sqlite_stat1" VALUES('ImageProperties','sqlite_autoindex_ImageProperties_1','5080 1 1');
> INSERT INTO "sqlite_stat1" VALUES('ImageTags','tag_index','8708 103');
> INSERT INTO "sqlite_stat1" VALUES('ImageTags','sqlite_autoindex_ImageTags_1','8708 2 1');
> INSERT INTO "sqlite_stat1" VALUES('Images','dir_index','5465 59');
> INSERT INTO "sqlite_stat1" VALUES('Images','sqlite_autoindex_Images_1','5465 2 1');
> INSERT INTO "sqlite_stat1" VALUES('TagsTree','sqlite_autoindex_TagsTree_1','207 3 1');
> INSERT INTO "sqlite_stat1" VALUES('Tags','sqlite_autoindex_Tags_1','92 1 1');
> INSERT INTO "sqlite_stat1" VALUES('Albums','sqlite_autoindex_Albums_1','111 1');
===============================================================================

After that I tried to compile the SVN version of digikam for KDE3, but failed miserably. I don't know why but it just didn't work out.

After compiling the 0.9.4beta3 source code from sf.net everything was up and I added debugging output to the digikamsearch-kioslave like Arnd mentioned before in this bugreport:

==============================================================
    kdWarning() << "SQL Query " << sqlQuery << endl;
    if (!m_db.execSql(sqlQuery, &values))
    {
        error(KIO::ERR_INTERNAL, errMsg);
        return;
    }
==============================================================

and

==============================================================
    kdWarning() << "SQL Query " << sqlQuery << endl;
    if (!m_db.execSql(sqlQuery, &values, &errMsg))
    {
        error(KIO::ERR_INTERNAL, errMsg);
        return;
    }
==============================================================

After installing digikam, no debugging infos were written to the console, even rebooting didn't do the trick, but Arnd's trick with restarting kdeinit did it, so I wrote a little startup script for running digikam with these new settings:

==============================================================
    #!/bin/bash
    
    export DIGIKAMDEST=/opt/kde
    export KDEDIRS=$DIGIKAMDEST:/usr
    export LD_LIBRARY_PATH=$DIGIKAMDEST/lib:$LD_LIBRARY_PATH
    export LDPATH=$DIGIKAMDEST/lib:$LDPATH
    export PATH=$DIGIKAMDEST/bin:$PATH
    kdeinit
    digikam
==============================================================

Now I was able to see the SQL queries that might be responsible for the slowdown. THIS query is my saved search called "top rated" that looks for images with a rating of 5 stars. It is NOT working properly anymore, I have to wait nearly 15 seconds to get a result, although I have only 6 images with a rating of 5:
        
    SELECT Images.id, Images.name, Images.dirid, Images.datetime, Albums.url FROM Images, Albums LEFT JOIN ImageProperties ON Images.id = Imageproperties.imageid WHERE (  (ImageProperties.value >= '5' and ImageProperties.property='Rating')  )  AND (Albums.id=Images.dirid);
        
The next query I wanted to check is the quick filter at the bottom of digikam, set to a rating of 5 stars, executed on a folder with 2000 images in it (recursively). Like I mentioned in my first post, this search is not slow. Unfortunately the quick search does not output a SQL query, so I can not give you any results.

The next step was to fire up sqlite and enter this query manually. So I pasted this query into sqlite and, as expected, it took nearly 15 seconds as well. So this must definitely be the problem!!
        
Hope somebody is still with me, this report is a little longer but I wanted to show all the steps taken to get to these results, so maybe another person might test it on a system with sqlite3-3.5.8
        
Andi

One little note: I have another filter searching for images with the rating of 1 star. Those are images that are not completely tagged, because I was lazy and will do it later... :-)
This search also takes nearly 15 seconds to complete, although the result of this search is 768 images. So it doesn't seem to be important how many images are in the result... if this is of any use to you...

Marcel,

Take a look in Andi report about slowdown DB query. Thanks in advance

Gilles

Andi,

I have just finalized the 0.9.4-beta4 tarball if you want to test with it. Look here :

http://digikam3rdparty.free.fr/0.9.x-releases/digikam-0.9.4-beta4.tar.bz2

Gilles

Andi,

thanks a lot for your heroic effort in tracking down this bug - I think
we are really really getting closer now.
Would it be possible that you put a copy of your digikam3.db at a place
to download (or if it is not too big, i.e. below 5 MB, just mail it directly
to me)?
I would like to install sqlite 3.5.7 and 3.5.8 and compare the speed.
Most likely, we will have to pass this problem, together with your database, 
on to the sqlite team...

Best, Arnd

Hi,

I was able to compile beta4... strange, SVN still is broken, even after deleting the whole folder and checking out the repo again. Anyway beta4 has this "bug", too. I will try to upload my database later.

Andi

Two more queries, maybe this is helpful.
The first query is very fast, it looks in a specific folder for images without a given tag:

SELECT Albums.url||'/'||Images.name FROM Images, Albums LEFT JOIN ImageProperties on Images.id = ImageProperties.imageid WHERE (  (Images.dirid = '39')  AND  (Images.id NOT IN    (SELECT ImageTags.imageid FROM ImageTags JOIN TagsTree on ImageTags.tagid = TagsTree.id     WHERE TagsTree.pid = '25' or ImageTags.tagid = '25' ))  )  AND (Albums.id=Images.dirid)  LIMIT 500;

The next one is incredibly slow, it took nearly 5 minutes!!!! It is a simple search looking for the family name "Saala":

SELECT Images.id, Images.name, Images.dirid, Images.datetime, Albums.url FROM Images, Albums LEFT JOIN ImageProperties ON Images.id = Imageproperties.imageid WHERE ( ( (Images.dirid IN   (SELECT id FROM Albums WHERE url LIKE '%Saala%'))  OR  (Images.name LIKE '%Saala%')  OR  (Images.id IN    (SELECT ImageTags.imageid FROM ImageTags JOIN TagsTree on ImageTags.tagid = TagsTree.id     WHERE TagsTree.pid = (SELECT id FROM Tags WHERE name LIKE 'Saala')     OR ImageTags.tagid = (SELECT id FROM Tags WHERE name LIKE 'Saala') ))  OR  (Images.dirid IN   (SELECT id FROM Albums WHERE caption LIKE '%Saala%'))  OR  (Images.dirid IN   (SELECT id FROM Albums WHERE collection LIKE '%Saala%'))  OR  (Images.caption LIKE '%Saala%')  OR  (ImageProperties.value LIKE '%Saala%' and ImageProperties.property='Rating') ) )  AND (Albums.id=Images.dirid);


So searching for ratings is not the only issue with the new sqlite version. Arnd, I will mail my DB to you (2.5 MB)

Using Andi's query from #c26 and his database, I can reproduce the
problem with sqlite 3.5.8:

date
sqlite3 andi_digikam3.db
SELECT Images.id, Images.name, Images.dirid, Images.datetime, Albums.url FROM Images, Albums LEFT JOIN ImageProperties ON Images.id = Imageproperties.imageid WHERE (  (ImageProperties.value >= '5' and ImageProperties.property='Rating')  )  AND (Albums.id=Images.dirid);
.quit
date

this takes between 10-15s on my machine while with sqlite 3.3.8 it
is way below 1s.
I will try to contact the people from sqlite3.

Andi, again thanks a lot for your work ...

Arnd

After discussion on the #sqlite IRC, I filed a bug in their bug-tracker
http://www.sqlite.org/cvstrac/tktview?tn=3087

Presumably we leave this bug open, until the problem is solved
or a solution/work-around is known.

Should we put a warning somewhere, that we should not use sqlite 3.5.8.
in the meantime?

Arnd,

There are few places for that :

- README
- digikam.org FAQ
- .configure log

Gilles

Just one little note... I found out why I was not able to compile the SVN version: I used

./configure --prefix=/opt/kde --enable-gcc-hidden-visibility --enable-debug=full 

for the beta4 tarball, but only

./configure --prefix=/opt/kde --enable-gcc-hidden-visibility 

for the SVN version. If all optimizations are turned on, my laptop just "hangs"... with full debugging and no optimizations it is working... 

Andi: very interesting - which compiler version do you use ("gcc -v")?
Does the *whole* laptop hang, or just digikam on start?
(and at what place does it hang, i.e. are there any messages on the konsole?)
One of the next steps would be to use a different (=older) gcc version.
Isolating the offending part of the code will be a nice ride ... ;-)

Gilles, could this be related to problems like
http://bugs.kde.org/show_bug.cgi?id=156146 ?
(Even though that sounds a bit different  ...)

Hi Arnd,

like I said in #26 I'm using gcc-4.3.0.
The laptop "hangs" during the compile process, not during startup, although I'm not sure if it actually hangs or if the optimizations just take so long. But after 2 hours compiling one little CPP file I'm losing patience and normally kill the compile process. So this might not be any kind of a bug, maybe my laptop is just too old (Mobile AMD Sempron(tm) Processor 2800+, 1256MB RAM), although I'm pretty sure I once compiled digikam without "--enable-debug=full" and with full optimization...
When I wrote my diploma thesis, I also used CImg in the first place, but I never had to wait that long for gcc to finish its job.

Andi,

Your report #36 is very important.

It's certainly a specific part on digiKam core which failded if no debug options is used to compile digiKam.

Note : no debug option is the same way than to not use ".configure --enable-final". This mode is used to generate final package without debug info and with full optimization.

In KDE3 with autotools, there is a macro to disable optimizations to a specific folder to compile : KDE_OPTIONS = nofinal. It's used for ex. in digikam/libs/jpegutils/Makefile.am with transupp.cpp code which do not support compilation optimizations.

I recommend you to make investiguations where code failed exactly. Certainly in DB interface which is mixed indeep in digikam/digikam folder with GUI (code is separated in KDE4). Just place the KDE macro to the digikam/digikam/Makefile.am and check if it work...

Arnd,

yes, i suspect than #156146 has the same problem.

Marcel, if you have any tips here...

Gilles Caulier

Andi,

If last CImg 1.2.8 included in digiKam core recently do not compile fine for you, it certainly a problem in CImg implementation. I will contact David Tschumperlé (CImg author).

Gilles Caulier

I think sqlite3-3.5.9 has this problem solved, at least the developers say so in the bugreport (posted by Arnd). I will try to get the newest version to check if the problem is still there.

Oh this was a quick compile :-)
Yes, I can confirm that sqlite3-3.5.9 solved the problem, so I guess there is nothing we can do about this problem. Hopefully most distributions will update their packages soon.

Andi,

we must update all Sqlite version depencies everywhere : 

- README file (KDE3 and KDE4)
- .configure.in.in (KDE3)
- .configure.in.bot (KDE3)
- CMakeList.txt (KDE4)

Gilles

SVN commit 815139 by cgilles:

digiKam from KDE3 branch : fix libsqlite depency >= 3.5.9 release
BUGS: 160966


 M  +2 -2      README  
 M  +44 -27    configure.in.bot  
 M  +1 -1      configure.in.in  


WebSVN link: http://websvn.kde.org/?view=rev&revision=815139

SVN commit 815218 by cgilles:

digiKAm from KDE3 branch : sqlite 3.5.9 code source inclusion to digiKam core.
duing a wrong package config provided in sqlite3 package, it's impossible to find the right version installed on computer.
Including sqlite3 directly on digiKam will solve this problem, prevents all future bug reports, and reduce depencies.
CCBUGS: 160966


 M  +1 -1      NEWS  
 M  +0 -1      README  
 M  +0 -13     configure.in.bot  
 M  +0 -20     configure.in.in  
 M  +2 -1      digikam/Makefile.am  
 M  +1 -1      digikam/albumdb.cpp  
 M  +4 -4      kioslave/Makefile.am  
 M  +1 -1      kioslave/sqlitedb.cpp  
 M  +1 -1      libs/Makefile.am  
 A             libs/sqlite3 (directory)  
 A             libs/sqlite3/Makefile.am  
 AM            libs/sqlite3/sqlite3.c   [POSSIBLY UNSAFE: system,gets] [License: GENERATED FILE]
 AM            libs/sqlite3/sqlite3.h   [POSSIBLY UNSAFE: system] [License: GENERATED FILE]


WebSVN link: http://websvn.kde.org/?view=rev&revision=815218

For the KDE4 version we are using the QtSql plugin for SQLite, and do not directly link to libsqlite3. Qt packages its own libsqlite3 (as you did above) or uses the libsqlite3 available on the system, depending on configure options.

Marcel,

To be sure to not have a side effect in KDE4 port, i will backport sqlite 3.5.9 code from KDE3. The advantage is to remove sqlite3 external depencies.

Gilles 

SVN commit 815399 by cgilles:

digiKam from trunk : removing external SQLite shared lib depency
CCBUGS: 160966


 M  +4 -15     CMakeLists.txt  
 M  +1 -2      README  
 M  +1 -0      digikam/CMakeLists.txt  
 M  +1 -1      libs/database/backendsqlite3.cpp  
 A             libs/database/sqlite3 (directory)  
 AM            libs/database/sqlite3/sqlite3.c   [POSSIBLY UNSAFE: system,gets] [License: GENERATED FILE]
 AM            libs/database/sqlite3/sqlite3.h   [POSSIBLY UNSAFE: system] [License: GENERATED FILE]
 M  +4 -2      project/project.kdevelop  


WebSVN link: http://websvn.kde.org/?view=rev&revision=815399

Only IMHO: Nahh,  I'll never buy this argument~! Sorry :(

  digiKam from trunk : removing external SQLite shared lib depency 

> duing a wrong package config provided in sqlite3 package, it's impossible to find the right version installed on computer.
Including sqlite3 directly on digiKam will solve this problem, prevents all future bug reports, and reduce depencies. 


From a distributors perspective this is just wrong. We don't want to pester our users with multiple sqlite engines and we want to issue a single security announcement for sqlite in case of a vulnerability and not for multiple packages. By including libraries instead using shared ones, you increase the memory usage on systems holding mutliple copies of the engine in memory and increase distributors workload. Thanks, no.
 
Simply adding a notice to your release notes, which sqlite engine is fine to use would be just fine.

I have to agree with #50. While this might fix an issue brought in in a recent sqlite update, it is going to cause massive headache to both packagers and users and in long term just bring in more trouble. Please consider reverting this particular change.

Reopening for proper discussion, feel free to close, it's your bug and your code after all.

Jan, the real solution would be to figure out a way so that during
configuration of digikam  (sqlite<3.5.8 or sqlite >3.5.8) is ensured.
Gilles has tried very hard (see the discussion on the mailing list)
to achieve this with autotools, but nothing worked out.
So if you or anyone else provides a patch for this, I am optimistic
that it can be integrated ...

Best, Arnd

Considering that the affected version of sqlite has been available for "just one month" and that most distributions that went ahead and bumped their version to 3.5.8 are likely to ship 3.5.9 in a timely fashion, I'd just ommit the check altogether. Bugs happen all the time and including a known-to-be-safe version in the name of "reducing dependencies" is a way to hell.

I don't see how this is a way to hell, there are surely much worse sins ...

There were already several users reporting problems which were caused by this particular sqlite issue, however finding the real cause in each of these
causes did take some time; also many users might not report such an
issue and just discard digikam ("it is just too slow ...").
This is why this sqlite was included.
Still, IMO the proper solution is the version check. 
Jan, do you have an idea how to do that?

Actually, another (less optimal) option might
be a runtime check of the used sqlite and prevent digikam from starting
with 3.5.8. What do you think Gilles?

Nope, I'm not familiar with autotools, sorry.

Speaking about a runtime check, upstream's ticket mentions adding a workaround "OR 0", so this might be a better action than just refusing to start at all.

> From a distributors perspective this is just wrong. We don't want to pester
> our users with multiple sqlite engines and we want to issue a single security
> announcement for sqlite in case of a vulnerability and not for multiple
> packages. By including libraries instead using shared ones, you increase the
> memory usage on systems holding mutliple copies of the engine in memory and
> increase distributors workload. Thanks, no.
>
> Simply adding a notice to your release notes, which sqlite engine is fine to
> use would be just fine.

+1 to that from a Fedora KDE packager. Including a copy of a library into an application is always wrong (well, let's say in at least 99% of the cases).

>Including a copy of a library into an application is always wrong

I recieve a lots of bug report about an unsuitable digiKam because a shrared lib is bugous is the _hell_.

I prefet to have a program to run as weel. It's really a big puzzle to solve this problem, and really i'm tired about this subject. 

Since few year digiKam already include libsqlite2 library code as well in core, and nobody crying about it. Amarok include libsqlite3 in core application too, Qt code include libpng and libsqlite3... I can continue the list if you want...

Note : for KDE4 port of digiKam, we use Qt4:sqlite plugin to play with database

Gilles Caulier

Can't you just make the autotools require sqlite >= 3.5.9? Then if anyone really wants to compile it with 3.5.7 they can go and patch the source...

I'm sorry if this has already been suggested; I couldn't find the mailing list discussion mentioned...

> Since few year digiKam already include libsqlite2 library code as well in core, and nobody crying about it. Amarok include libsqlite3 in core application too, Qt code include libpng and libsqlite3... I can continue the list if you want... 

Which is in each case annoying and at Gentoo we have quite often bug reports to sort out this library inlusion mess and send patches upstream. In case of Qt you're wrong. It allows to use system libraries. Let me state it again: Fromthe security point of view, senseless included libraries are a hazard. Its a lot more work, a lot more packages are to update and there's always the chance you miss a package leaving users with a vulnerable application. There're very very few reasons to link statically, e.g. on embedded systems or stuff you need at boot time, but there's nothing that justifies it for desktop applications.

I think comment #58 is the best option both for you, not getting duplicates of this bug report as well as us downstream.

Hi,

as mentioned before we tried to restrict the usage of sqlite3 to versions >= 3.5.9, but the current sqlite3 package seems to have problems with its own version number.

sqlite3.pc from current sqlite3-3.5.9:
======================================
# Package Information for pkg-config

prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: SQLite
Description: SQL database engine
Version: 3.5
Libs: -L${libdir} -lsqlite3
Libs.private: -lpthread 
Cflags: -I${includedir}
======================================

So autotools were not able to find the currently installed sqlite3-3.5.9 installation. All those problems with sqlite within the last two months made us decide to include a working copy into digiKam.

> So autotools were not able to find the currently installed sqlite3-3.5.9 installation.

Then just don't check and add a notice to the release. Quick and easy, who doesn't read the announcement is at fault. Including the library is the wrong solution.

Another option would be to ask the SQLite guys to do a 3.5.9.1 release with a fixed .pc file. Remains the question how many application build scripts would choke on that, because of checking for equalness to 3.5.

If I can join in.

In Debian/GNU Linux we have made it policy that packages shouldn't include convenience copies of other libraries, due to the issues with tracking security fixes in all packages that have copies of other libraries.
http://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles

However, I can also sympathise with the need.

Could I suggest that digikam autotools checks for an installed lib >= 3.5.9 and if it doesn't find it then digikam uses its own internal copy, if it does find >= 3.5.9 then it uses the system library.

The distributions can fixup the .pc file in sqlite easily enough, if their upstream doesn't. 

This way end users compiling get an assured working version of digikam and the distributions get an version of digikam using shared libraries and things are the way they should be.

Ditto the convenience copy of cImg library:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=484777

Mark

Further it appears Debian sqlite3 package allready does patch .pc to the correct version:

Mark

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=424235

# Package Information for pkg-config

prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: SQLite
Description: SQL database engine
Version: 3.5.9
Libs: -L${libdir} -lsqlite3
Libs.private: -lpthread
Cflags: -I${includedir}

SVN commit 827804 by cgilles:

digiKam from KDE3 branch : add rule to compile and link with an external libsqlite3 shared lib instead embeded version include in digiKam
Note : embeded version is always compiled by default if nothing is set with .configure script. Packagers, please take a care.
BUGS: 160966


 M  +1 -0      README  
 M  +16 -0     configure.in.bot  
 M  +36 -0     configure.in.in  
 M  +11 -3     digikam/Makefile.am  
 M  +16 -7     kioslave/Makefile.am  
 M  +6 -2      libs/Makefile.am  


WebSVN link: http://websvn.kde.org/?view=rev&revision=827804

Thanks Gilles,  That should work for us.

Mark