Bug 376891 - Digikam becomes unusable with many Metatags
Summary: Digikam becomes unusable with many Metatags
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Searches-Advanced (show other bugs)
Version: 5.7.0
Platform: Neon Linux
: NOR major
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-02-24 15:32 UTC by bzrudie
Modified: 2020-12-25 19:44 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 7.2.0


Attachments
Valgrind log (1.41 MB, application/gzip)
2017-02-25 09:13 UTC, bzrudie
Details

Note You need to log in before you can comment on or make changes to this bug.
Description bzrudie 2017-02-24 15:32:11 UTC
Problem/Motivation
I tried Digikam as a replacement for my current Mac/Aperture setup. I tried on my Linux Box first with some images and everything does well, nice :) So I started the actual migration of around 28.000 images and let Digikam finish all work. Some hours later all thumbs and other stuff were created okay.

The actual problem started when it comes to Metadata. Searching for a tag takes ages, but when it comes to tagging, digikam totally failed :(
I have 5000+ tags in my list all at first level (no nested tags)
Opening the Tag Manager takes around 15seconds, adding a new tag or moving a tag takes around 1 minute and digikam doesn't respond to anything during that.

My first idea: Okay there are missing database indexes, but that doesn't seem the case.
Next idea, IO problem: I moved the digikam SQLiteDB to RAM, but no effect.
Next idea, SQLite problem: Even knowing SQLite is usually really fast I gave MySQL a try - but as to expect - no effect.

So the problem is with digikam itself. As the CPU usage goes up to 100% I think digikam stucks during tags qt-list-building/rendering or whatever.

Any ideas ;)

Thanks
Comment 1 caulier.gilles 2017-02-25 08:31:32 UTC
Where digiKam waste time at startup ? Can you investigate using kcallgrind tool ? This will indicate which code will be called a lots of time in this situation...

Gilles Caulier
Comment 2 bzrudie 2017-02-25 09:13:32 UTC
Created attachment 104212 [details]
Valgrind log
Comment 3 bzrudie 2017-02-25 09:14:13 UTC
(In reply to caulier.gilles from comment #1)
> Where digiKam waste time at startup ? Can you investigate using kcallgrind
> tool ? This will indicate which code will be called a lots of time in this
> situation...
> 
> Gilles Caulier

Hi Gilles,

Digikam starts within a minute or two, so that is no problem for me as I usually will start it once a day. The problem comes up when I try to add/edit/delete a tag, Digikam will freeze for a whole minute until done.

I just installed valgrind and started digikam like this:
valgrind --tool=callgrind digikam

I get lots of segment overflows like :
==5290== brk segment overflow in thread #1: can't grow to 0x4a38000

and I am waiting now 20 minutes for Digikam to start up ;)

I found some info on stackoverflow:

http://stackoverflow.com/questions/35129135/valgrind-reporting-a-segment-overflow

still not sure if that is a problem at all. However, seems I can't even get Digikam to start up running valgrind. I killed the process and attaching the output till here, maybe it's already usefull for debugging.

Thanks!
Comment 4 caulier.gilles 2017-04-16 20:18:17 UTC
new 5.6.0 pre-release as bundle is available here :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Please check if this problem still reproducible with these versions.

Thanks in advance

Gilles Caulier
Comment 5 bzrudie 2017-04-18 14:21:46 UTC
Thanks Gilles for looking into that.

Unfortunately I think it didn't change anything. I'm now using Digikam on my more powerful MacBook Pro with SSD but still have the problem with tags management. Just to make sure I stopped some times to have something to compare if you find something other to try :)
===
- Startup time digikam: 3min 20 seconds (mostly "Reading Database...")
- Startup time "Open Tag Manager" 23 seconds (scrolling through the list of tags is fast BTW)
- Click on "+" to add a new tag, 5 seconds till window open.
- Entering a new tag in that window and click save makes digikam become unresponsive for 1min and 33 seconds, doh! (e.g. digikam completely hangs)
- Deleting the newly created tag takes nearly the same time, 1min and 36s
===

Here are some stats from my database:

digikam version 5.6.0
Images: 
JPG: 32227
total: 32227
: 
Videos: 
AVI: 1
total: 1
: 
Total Items: 32228
Albums: 422
Tags: 5124
: 
Database backend: QSQLITE

Thanks again!
cheers bzrudi
Comment 6 caulier.gilles 2017-04-18 14:34:54 UTC
ok.

Just to be sure :

1/ backup your sqlite database files.
2/ rename it as .old or delete files.
3/ restart digiKam and repopulate the collections from scratch.
4/ Check if problem remain.

Notes :

- here i use more than 150000 items with sqlite and a lots of tags without any special time latency.

- 5.6.0 has a database garbage collector tool in maintenance. Thi scan be a solution to cleanup your original DB.

Gilles Caulier
Comment 7 bzrudie 2017-04-18 17:19:39 UTC
Okay Gilles, did some more testing.
Started from scratch and sadly no change, problem exists. So I checked database and started to simply delete some tags from the Tags relation. E.g. as i have more than 5000 tags I started with:

sqlite> delete from Tags where id > 4500;
no real change...
sqlite> delete from Tags where id > 4000;
no real change...
sqlite> delete from Tags where id > 3000;
better, tag manager now starts within 7 seconds...
sqlite> delete from Tags where id > 1000;
nice, tag manager starts within 1-2 seconds. I would say things work within an acceptable time.
going even further and just leave 500 tags within database things rock!

===
- Startup time digikam: 5000+ tags 3min 20 seconds vs. 500 tags 13 seconds!
- Startup time "Open Tag Manager": 5000+ 23 seconds vs. 500 tags 1 second!
- Click on "+" to add a new tag: 5000+ 5 seconds vs. 500 tags realtime
- Entering a new tag in that window and click save makes digikam become unresponsive for 1min and 33 seconds, doh! vs. 500 tags 2 seconds
- Deleting the newly created tag takes nearly the same time, 1min and 36s vs. 500 tags 2 seconds
===


So things get massiv slower when reaching 2000 or 3000 tags. Feels some indexes are missing!? Maybe I have the time tomorrow for testing out.

cheers bzrudi
Comment 8 caulier.gilles 2017-04-18 17:55:26 UTC
I think the problem can be the auto-completer for tag names in search tree-view.

This one is populated with database contents and is updated at each changes.

This will explain why with a fresh DB, the problem remain...

Gilles
Comment 9 caulier.gilles 2017-06-22 21:43:10 UTC
digiKam 5.6.0 is now released and available as bundle for Linux, MacOS and Windows.

https://www.digikam.org/news/2017-06-21-5.6.0-release-announcement/

Can you check if problem still exists with this version ?

Thanks in advance

Gilles Caulier
Comment 10 caulier.gilles 2017-07-23 18:26:58 UTC
New digiKam 5.7.0 are built with current implementation as pre-release bundles:

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Problem still reproducible ?
Comment 11 bzrudie 2017-08-10 12:31:48 UTC
Hey Gilles,

sadly problem still exists in latest 5.7.0 beta.
Comment 12 caulier.gilles 2018-08-17 21:30:28 UTC
Can you reproduce the dysfunction using digiKam 6.0.0 pre-release bundle
available here :

https://files.kde.org/digikam/
Comment 13 caulier.gilles 2018-12-31 11:51:06 UTC
Can you reproduce the dysfunction using the last digiKam 6.0.0-beta3 just
released ?

https://www.digikam.org/news/2018-12-30-6.0.0-beta3_release_announcement/
Comment 14 caulier.gilles 2019-03-07 05:21:54 UTC
Good news,

After 2 weeks of works, the pre-release 6.1.0 bundles are now reconstructed from scratch with:

- All OpenCV options for CUDA, OPenMP, and OPenCL disabled to prevent crashes
in face management.
- A large upgrade of Qt5 from 5.9.7 to 5.11.3.
- An upgrade to KF5 5.55.
- An upgrade to Ffmpeg 3.3.9
- The fontconfig/freetype integration in the bundle to reduce system
dependencies

Files can be downloaded here :

https://files.kde.org/digikam/

Please test and report.

Gilles Caulier
Comment 15 caulier.gilles 2020-07-14 09:41:38 UTC
Hi,

Can you check if this problem still exist with last weekly bundle build of digiKam 7.0.0 available here:

https://files.kde.org/digikam/

Thanks in advance

Gilles Caulier
Comment 16 caulier.gilles 2020-07-30 09:42:32 UTC
digiKam 7.0.0 stable release is now published:

https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/

We need a fresh feedback on this file using this version.

Thanks in advance

Gilles Caulier
Comment 17 Maik Qualmann 2020-12-25 19:44:09 UTC
The main problem was sorting the tags. We had created a QCollator object for every sorting process of a tag name. Since the QCollator loads a database file every time, this is very slow. We now have a static QCollator. A test with 30,000 tags shows a good speed here. I'm closing the bug now. If there are still problems, please open again.

Maik