Bug 216895 - SQLITE : improve performance and reduce space with some pragma's
Summary: SQLITE : improve performance and reduce space with some pragma's
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Sqlite (show other bugs)
Version: 1.0.0
Platform: Debian testing Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-12-01 07:17 UTC by Scott Crosby
Modified: 2017-02-22 15:10 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.5.0


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Scott Crosby 2009-12-01 07:17:57 UTC
Version:           1.0.0beta6 (using KDE 4.3.2)
OS:                Linux
Installed from:    Debian testing/unstable Packages

You have an awesome program that I enjoy using. Just a couple of things you can do to make it better. The default setup of sqlite uses a 1kb page size. This can lead to a lot of wasted space in the large tuples that digikam uses. Increasing the pagesize to 4kb reduces the total database size by a third. I used 4kb because it matches the normal intel page size and larger page sizes have marginal benefit (<1% space savings). I also suspect this page size comes with some performance benefits.

For existing databases (and recent versions of sqlite3):
   'pragma page_size = 4096 ; vacuum;'

When creating a new database:
   'pragma page_size = 4096;' before the first table.

Also, in order to run in limited environments, sqlite only lets itself use a default of at most of 2000 pages worth of data for caching or internal sorts. I propose increasing this to 8k-16k pages (32-64mb). Modern machines have a lot of RAM. I've not benchmarked if there is any speed improvement, but I cannot see how giving sqlite a few more megabytes of memory can hurt.

Can be run on any existing or new database:
   'pragma default_cache_size=16000;'
Comment 1 caulier.gilles 2009-12-01 10:01:14 UTC
Note : digiKam is under porting to MySQL...

Gilles Caulier
Comment 2 caulier.gilles 2011-12-15 09:01:44 UTC
Scott,

What's about this file using digiKam 2.x serie ?

Gilles Caulier
Comment 3 caulier.gilles 2016-07-16 13:23:24 UTC
Swati,

Did you take a look about this file ?

Gilles Caulier
Comment 4 swatilodha27 2016-08-04 13:22:13 UTC
(In reply to caulier.gilles from comment #3)
> Swati,
> 
> Did you take a look about this file ?

Yes. Seems like a good option. 
The page and the cache size could be modified for SQLite. 

But is there a file already, that contains statement specifically for SQLite? Should the statement be added in XML schema file, under "Create DB" ?
Comment 5 caulier.gilles 2016-08-06 05:53:53 UTC
No specific file exist for sqlite settings.

XML file must be patch. create DB sound like the best pace for that. revision must be increased of course.

Giles Caulier
Comment 6 Mario Frank 2017-02-22 15:10:47 UTC
Git commit 2f8ddd42ef62d7aea9e490cdb05ffcc644810c81 by Mario Frank.
Committed on 22/02/2017 at 15:05.
Pushed by mfrank into branch 'master'.

Merged the current state of the garbage collection branch which improves the database cleanup stage of the maintenance
and improves the reactiveness of the maintenance overall. We ported the way items are processed to a queue based method
that can use the CPUs more effectively and does not create thousands of threads.
Related: bug 283062, bug 374225, bug 351658, bug 362023, bug 329353
FIXED-IN: 5.5.0

M  +17   -12   NEWS

https://commits.kde.org/digikam/2f8ddd42ef62d7aea9e490cdb05ffcc644810c81