Bug 368468 - digiKam hanging at start up with a collection that has >100000 sub folders
Summary: digiKam hanging at start up with a collection that has >100000 sub folders
Status: REPORTED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Scan (show other bugs)
Version: 5.2.0
Platform: Compiled Sources Linux
: NOR crash
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
: 421673 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-09-08 22:02 UTC by Lars Van Casteren
Modified: 2024-03-28 15:33 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Lars Van Casteren 2016-09-08 22:02:45 UTC
digiKam will scan the collection for pictures but it hangs when trying to display the actual collection tree in the album view.

Reproducible: Always

Steps to Reproduce:
1. Remove all sqlite databases
2. Create a new root collection somewhere.
3. Create the huge hierarchy inside with this little script :

for ((i=1;i<=110000;i++)); do mkdir -p testdir/$i; done

4. Start digiKam in a console through GDB and setup root local collection to the root dir of huge hierarchy.

Actual Results:  
digiKam will scan the collection for pictures but it hangs when trying to display the actual collection tree in the album view.

Expected Results:  
The collection tree (ie the sub folders) should be visualised.

Problem became apparent when adding a directory that contained an OSX Photos library. Logging showed it actually scans and finds all the images but hangs after the discovery process, at that point it's trying to create the album tree but does not complete.
Comment 1 caulier.gilles 2016-09-09 14:55:42 UTC
Originally, this problem have been discovered under OSX.

Gilles Caulier
Comment 2 Maik Qualmann 2016-09-26 17:59:23 UTC
Git commit 38c411cfe17cbc0746151594f889be1363a1023b by Maik Qualmann.
Committed on 26/09/2016 at 17:58.
Pushed by mqualmann into branch 'master'.

optimization ModelCompleter and AlbumWatch

M  +1    -4    libs/album/albumwatch.cpp
M  +36   -38   libs/widgets/common/modelcompleter.cpp
M  +1    -2    libs/widgets/common/modelcompleter.h

http://commits.kde.org/digikam/38c411cfe17cbc0746151594f889be1363a1023b
Comment 3 Maik Qualmann 2016-09-26 18:00:15 UTC
This commit reduce the loading from albums at me from 6:00 minutes to 1:30 minutes at 10.000 albums.

Maik
Comment 4 caulier.gilles 2016-09-26 20:56:19 UTC
Lars, 

A feedback from you using current code from git/master, including Maik patch will be appreciate.

Thanks in advance

Gilles Caulier
Comment 5 Lars Van Casteren 2016-09-27 07:59:27 UTC
Thanks for the effort. 
I am on a (backpack) holiday until 27/10 but i'll try my best to test it somewhere in between. 

Lars
Comment 6 Maik Qualmann 2016-09-27 20:35:16 UTC
Git commit 4ebe5585c7f9c37842c541304cdd0e63912c7cb8 by Maik Qualmann.
Committed on 27/09/2016 at 20:34.
Pushed by mqualmann into branch 'master'.

optimization ImageAlbumFilterModel

M  +55   -13   libs/models/imagealbumfiltermodel.cpp
M  +5    -0    libs/models/imagealbumfiltermodel.h

http://commits.kde.org/digikam/4ebe5585c7f9c37842c541304cdd0e63912c7cb8
Comment 7 Maik Qualmann 2016-09-27 20:36:34 UTC
Now here less than 1 minute for 10.000 albums. But I think 100.000 albums are not really to handle.

Maik
Comment 8 caulier.gilles 2016-11-03 22:03:35 UTC
Lars,

Can you review last commits from Maik using current 5.3.0 AppImage bundle for Linux ?

https://drive.google.com/open?id=0BzeiVr-byqt5Y0tIRWVWelRJenM

Gilles Caulier
Comment 9 caulier.gilles 2016-11-25 16:00:22 UTC
Lars,

Any feedback with current AppImage bundle for Linux ?

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Gilles Caulier
Comment 10 caulier.gilles 2017-04-16 20:29:15 UTC
new 5.6.0 pre-release as bundle is available here :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Please check if this problem still reproducible with these versions.

Thanks in advance

Gilles Caulier
Comment 11 caulier.gilles 2017-06-22 21:43:07 UTC
digiKam 5.6.0 is now released and available as bundle for Linux, MacOS and Windows.

https://www.digikam.org/news/2017-06-21-5.6.0-release-announcement/

Can you check if problem still exists with this version ?

Thanks in advance

Gilles Caulier
Comment 12 caulier.gilles 2017-07-23 18:26:40 UTC
New digiKam 5.7.0 are built with current implementation as pre-release bundles:

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Problem still reproducible ?
Comment 13 caulier.gilles 2017-12-13 22:45:43 UTC
What's about this file using 5.8.0 pre-release buncle :

https://files.kde.org/digikam/

Thanks in advance

Gilles Caulier
Comment 14 caulier.gilles 2018-08-17 21:27:26 UTC
Can you reproduce the dysfunction using digiKam 6.0.0 pre-release bundle available here :

https://files.kde.org/digikam/

Gilles Caulier
Comment 15 Maik Qualmann 2018-08-24 20:02:16 UTC
Git commit f8d8dc6ebdbbb0f75561a6a4dc6a0a95d728ca42 by Maik Qualmann.
Committed on 24/08/2018 at 20:00.
Pushed by mqualmann into branch 'master'.

store the number of childs in the album
this commit reduces the start of digiKam here with 11000 albums from 1:30 minutes to 1:15 minutes
Related: bug 396086

M  +11   -0    core/libs/album/album.cpp
M  +8    -1    core/libs/album/album.h
M  +2    -2    core/libs/models/abstractalbummodel.cpp
M  +0    -14   core/libs/models/abstractalbummodelpriv.h

https://commits.kde.org/digikam/f8d8dc6ebdbbb0f75561a6a4dc6a0a95d728ca42
Comment 16 Maik Qualmann 2018-08-24 21:45:49 UTC
Git commit dcb01e39023564ea538eb06f2cf635a451f713e3 by Maik Qualmann.
Committed on 24/08/2018 at 21:44.
Pushed by mqualmann into branch 'master'.

implement a child album cache hash
this commit reduces the start of digiKam here with 11000 albums from 1:15 minutes to 0:35 minutes
Related: bug 396086

M  +16   -5    core/libs/album/album.cpp
M  +6    -1    core/libs/album/album.h
M  +2    -2    core/libs/models/abstractalbummodel.cpp
M  +0    -23   core/libs/models/abstractalbummodelpriv.h

https://commits.kde.org/digikam/dcb01e39023564ea538eb06f2cf635a451f713e3
Comment 17 Maik Qualmann 2018-08-24 23:21:08 UTC
Git commit 6d16a4f96ac245ed11450326c128cf63ca5a1332 by Maik Qualmann.
Committed on 24/08/2018 at 23:17.
Pushed by mqualmann into branch 'master'.

implement a child album to row cache hash
this commit reduces the start of digiKam here with 11000 albums from 0:35 minutes to 0:20 minutes
the sorting of the albums and entries is now almost without delay.
Related: bug 396086

M  +14   -4    core/libs/album/album.cpp
M  +12   -5    core/libs/album/album.h
M  +1    -16   core/libs/models/abstractalbummodelpriv.h

https://commits.kde.org/digikam/6d16a4f96ac245ed11450326c128cf63ca5a1332
Comment 18 caulier.gilles 2018-12-31 11:50:04 UTC
Can you reproduce the dysfunction using the last digiKam 6.0.0-beta3 just
released ?

https://www.digikam.org/news/2018-12-30-6.0.0-beta3_release_announcement/
Comment 19 caulier.gilles 2019-03-07 05:13:28 UTC
Good news,

After 2 weeks of works, the pre-release 6.1.0 bundles are now reconstructed from scratch with:

- All OpenCV options for CUDA, OPenMP, and OPenCL disabled to prevent crashes
in face management.
- A large upgrade of Qt5 from 5.9.7 to 5.11.3.
- An upgrade to KF5 5.55.
- An upgrade to Ffmpeg 3.3.9
- The fontconfig/freetype integration in the bundle to reduce system
dependencies

Files can be downloaded here :

https://files.kde.org/digikam/

Please test and report.

Gilles Caulier
Comment 20 Maik Qualmann 2020-05-18 20:06:47 UTC
*** Bug 421673 has been marked as a duplicate of this bug. ***
Comment 21 caulier.gilles 2020-07-14 09:41:30 UTC
Hi,

Can you check if this problem still exist with last weekly bundle build of digiKam 7.0.0 available here:

https://files.kde.org/digikam/

Thanks in advance

Gilles Caulier
Comment 22 caulier.gilles 2020-07-30 09:41:16 UTC
digiKam 7.0.0 stable release is now published:

https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/

We need a fresh feedback on this file using this version.

Thanks in advance

Gilles Caulier
Comment 23 Maik Qualmann 2020-10-10 20:24:16 UTC
Git commit d63e171bec0910f036bb3c2b261ab3333af110ee by Maik Qualmann.
Committed on 10/10/2020 at 20:22.
Pushed by mqualmann into branch 'master'.

add experimental QCollatorSortKey cache for fast string sorting
Add quick cache comparison to item and album sorting.
Changing a view with about 30,000 items when sorting
by name or path previously took about 22 seconds.
Now about 2-3 seconds. We will observe how the
memory consumption develops.
Related: bug 396086

M  +4    -8    core/libs/database/models/itemsortsettings.h
M  +3    -6    core/libs/models/albumfiltermodel.cpp
M  +96   -12   core/libs/threadimageio/fileio/loadingcache.cpp
M  +8    -0    core/libs/threadimageio/fileio/loadingcache.h

https://invent.kde.org/graphics/digikam/commit/d63e171bec0910f036bb3c2b261ab3333af110ee
Comment 24 caulier.gilles 2021-03-30 06:53:39 UTC
digiKam 7.2.0 official release is published with more than 360 files closed from bugzilla:

https://www.digikam.org/news/2021-03-22-7.2.0_release_announcement/

Can you reproduce the dysfunction with this version ?

Thanks in advance for your feedback

Gilles Caulier
Comment 25 caulier.gilles 2021-12-15 16:52:09 UTC
Lars,

Stable digiKam 7.4.0 is published. Please check if problem is reproducible.

https://www.digikam.org/download/

Thanks in advance
Comment 26 caulier.gilles 2023-04-20 05:41:47 UTC
@Lars

digiKam 8.0.0 is out. Problem still reproducible ?

Best regards
Gilles Caulier
Comment 27 caulier.gilles 2023-10-12 14:32:54 UTC
@Lars,

What's about this file using current 8.2.0 AppImage Linux bundle ? It's
reproducible ?

https://files.kde.org/digikam/

Note: bundle is now based on Qt 5.15.11 and KDE framework 5.110.

Thanks in advance

Gilles Caulier
Comment 28 caulier.gilles 2024-03-28 15:33:00 UTC
@Lars,

digiKam 8.3.0 stable version is released and available at usual place :

https://www.digikam.org/download/

Can you reproduce the dysfunction on your computer ?

Thanks in advance

Gilles Caulier