Bug 318516 - SCAN : Improve digiKam loading time [patch]
Summary: SCAN : Improve digiKam loading time [patch]
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Unclassified
Component: Database-Scan (show other bugs)
Version: 3.1.0
Platform: Compiled Sources Linux
: NOR wishlist (vote)
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-04-17 18:21 UTC by Yann Le Hir
Modified: 2020-02-17 21:22 UTC (History)
5 users (show)

See Also:
Latest Commit:
Version Fixed In: 7.0.0


Attachments
Change loading sequence. (1.31 KB, patch)
2013-04-17 18:23 UTC, Yann Le Hir
Details
Patch that fixes initialisation of ImageAlbumFilterModel (2.92 KB, patch)
2013-04-17 18:33 UTC, Yann Le Hir
Details
Restore the main view when restoreSession in called by main. (2.33 KB, patch)
2013-04-17 20:17 UTC, Yann LE HIR
Details
Cleaner patch to change the loading sequence and keep the session restore. (2.98 KB, patch)
2013-04-18 21:55 UTC, Yann LE HIR
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Yann Le Hir 2013-04-17 18:21:27 UTC
Right now the loading time for digikam is very high.
I use digikam with something like 16k Albums and 150k pictures.
Loading digikam can take up to 3-4 minutes with a high end CPU. 
Passing the fact that digikam only uses 1 core to load, it should still be much faster.

I have two patches that improves loading time tremendously.

Reproducible: Always

Steps to Reproduce:
1. Start digikam
2.
3.
Comment 1 Yann Le Hir 2013-04-17 18:23:17 UTC
Created attachment 78989 [details]
Change loading sequence.

This patch change the loading sequence, so digikam scan albums before loading the GUI. This way, GUI Objects don't get all first loading SIGNAL that take a lot of time to handle for nothing...
Comment 2 Yann Le Hir 2013-04-17 18:33:25 UTC
Created attachment 78990 [details]
Patch that fixes initialisation of ImageAlbumFilterModel

This patch removes two useless loops.
Indeed the loop call a method with an Album in parameter, but that method doesn't really depend on the Album but more on the AlbumManager.
So I splitting the method in two other methods to avoid code redundance, and call them appropriately.
Comment 3 caulier.gilles 2013-04-17 18:36:14 UTC
Thanks for your patches...

Gilles Caulier
Comment 4 Yann LE HIR 2013-04-17 18:39:25 UTC
Unforseen consequence of the patches, 
digikam doesn't load the last album shown when previously quit.
I'll take a look at this.
Comment 5 Yann LE HIR 2013-04-17 20:17:20 UTC
Created attachment 78993 [details]
Restore the main view when restoreSession in called by main.

Ok, so the view was restored when the AlbumManger emited allAlbumLoaded.
Which is due to my patch never received by the main view.
I joined a patch that put the restore view mechanism when restoreSession is called by main.

This is becoming more intrusive that I would like...
Comment 6 Marcel Wiesweg 2013-04-18 20:01:44 UTC
Ok, with 16000 albums I never tested. I concede that some relevant stuff scales linearly.
With you patches, you cut your loading time by...an order of magnitude? I mean, this was the problem?
Then it's not that intrusive but a very relevant fix. We only need to test thoroughly for regressions.
Comment 7 Yann LE HIR 2013-04-18 21:55:39 UTC
Created attachment 79013 [details]
Cleaner patch  to change the loading sequence and keep the session restore.

I joined a cleaner patch for the loading sequence.

On a clean branch, with digikam compiled in debug mode.
I get an average loading time of 33.36s (with 10 launches)
With the patches I get an average loading time of 2.85s
Almost 12x faster :)
It's near 10x faster in release mode.

I'm pretty sure it was the main issue. The fact that you try to reload 16k times all the filter models is quite a performance killer. Not to mention the AbstractAlbumModel that also took something like 32k events (AboutToBeAdded + Added)...
Without the patches, profiling on the loading of digikam can show up to 103 million calls to Album::next() and a fair amount of Album::title()...
Comment 8 Smit Mehta 2013-04-20 11:48:34 UTC
Hi

I tested it on ~10 albums, and obviously I am not getting that big a speedup. @Marcel, @Gilles, do we need more testing, or shall I apply this to git? I have tested this. It doesn't seem to break anything.

Smit
Comment 9 Yann LE HIR 2013-04-20 11:54:32 UTC
Please do not apply.
This patch is not fixing everything, I did more testing, the scanDalbum with the delay via kioslave etc is still loading after the graphic elements and thus still slowing down everthing.
I have some ideas for a more long term, elegant solution to this issue.
Comment 10 Gowtham Ashok 2013-05-12 07:49:38 UTC
I applied this patch and found that when loading 1000+ pictures, digiKam loads pictures even after the loading window closes, which slows down digiKam considerably. This does not happen in the clean one. Is this the same bug you are facing Yann Le Hir? 

I saw that much time is spent on waiting for SQLite( it's in locked state). I think loading would be faster, if, when loading >1000 files, ~5-10 images are committed to the DB at a time, rather than one at a time.
Comment 11 caulier.gilles 2013-10-30 15:33:37 UTC
Yann

Can you give a feedback about comment #10 from Gowtham, please ?

Marcel,

There is any work from you about to review/test this entry recently ?

Best

Gilles Caulier
Comment 12 caulier.gilles 2014-08-07 07:11:22 UTC
Marcel,

Whats new about this file ?

Gilles Caulier
Comment 13 caulier.gilles 2016-07-23 09:41:23 UTC
To Yann le Hir,

I would to see more investigations about your patches. Can you update these file to be compatible with current implementation from git/master based on Qt5 ?

Thanks in advance

Gilles Caulier
Comment 14 Maik Qualmann 2020-02-17 21:22:28 UTC
I have ported and tested the patches to git/master. However, we now use a delayed timer that gives even better results. I now close the bug report.

Maik