Problem/Motivation I tried Digikam as a replacement for my current Mac/Aperture setup. I tried on my Linux Box first with some images and everything does well, nice :) So I started the actual migration of around 28.000 images and let Digikam finish all work. Some hours later all thumbs and other stuff were created okay. The actual problem started when it comes to Metadata. Searching for a tag takes ages, but when it comes to tagging, digikam totally failed :( I have 5000+ tags in my list all at first level (no nested tags) Opening the Tag Manager takes around 15seconds, adding a new tag or moving a tag takes around 1 minute and digikam doesn't respond to anything during that. My first idea: Okay there are missing database indexes, but that doesn't seem the case. Next idea, IO problem: I moved the digikam SQLiteDB to RAM, but no effect. Next idea, SQLite problem: Even knowing SQLite is usually really fast I gave MySQL a try - but as to expect - no effect. So the problem is with digikam itself. As the CPU usage goes up to 100% I think digikam stucks during tags qt-list-building/rendering or whatever. Any ideas ;) Thanks
Where digiKam waste time at startup ? Can you investigate using kcallgrind tool ? This will indicate which code will be called a lots of time in this situation... Gilles Caulier
Created attachment 104212 [details] Valgrind log
(In reply to caulier.gilles from comment #1) > Where digiKam waste time at startup ? Can you investigate using kcallgrind > tool ? This will indicate which code will be called a lots of time in this > situation... > > Gilles Caulier Hi Gilles, Digikam starts within a minute or two, so that is no problem for me as I usually will start it once a day. The problem comes up when I try to add/edit/delete a tag, Digikam will freeze for a whole minute until done. I just installed valgrind and started digikam like this: valgrind --tool=callgrind digikam I get lots of segment overflows like : ==5290== brk segment overflow in thread #1: can't grow to 0x4a38000 and I am waiting now 20 minutes for Digikam to start up ;) I found some info on stackoverflow: http://stackoverflow.com/questions/35129135/valgrind-reporting-a-segment-overflow still not sure if that is a problem at all. However, seems I can't even get Digikam to start up running valgrind. I killed the process and attaching the output till here, maybe it's already usefull for debugging. Thanks!
new 5.6.0 pre-release as bundle is available here : https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM Please check if this problem still reproducible with these versions. Thanks in advance Gilles Caulier
Thanks Gilles for looking into that. Unfortunately I think it didn't change anything. I'm now using Digikam on my more powerful MacBook Pro with SSD but still have the problem with tags management. Just to make sure I stopped some times to have something to compare if you find something other to try :) === - Startup time digikam: 3min 20 seconds (mostly "Reading Database...") - Startup time "Open Tag Manager" 23 seconds (scrolling through the list of tags is fast BTW) - Click on "+" to add a new tag, 5 seconds till window open. - Entering a new tag in that window and click save makes digikam become unresponsive for 1min and 33 seconds, doh! (e.g. digikam completely hangs) - Deleting the newly created tag takes nearly the same time, 1min and 36s === Here are some stats from my database: digikam version 5.6.0 Images: JPG: 32227 total: 32227 : Videos: AVI: 1 total: 1 : Total Items: 32228 Albums: 422 Tags: 5124 : Database backend: QSQLITE Thanks again! cheers bzrudi
ok. Just to be sure : 1/ backup your sqlite database files. 2/ rename it as .old or delete files. 3/ restart digiKam and repopulate the collections from scratch. 4/ Check if problem remain. Notes : - here i use more than 150000 items with sqlite and a lots of tags without any special time latency. - 5.6.0 has a database garbage collector tool in maintenance. Thi scan be a solution to cleanup your original DB. Gilles Caulier
Okay Gilles, did some more testing. Started from scratch and sadly no change, problem exists. So I checked database and started to simply delete some tags from the Tags relation. E.g. as i have more than 5000 tags I started with: sqlite> delete from Tags where id > 4500; no real change... sqlite> delete from Tags where id > 4000; no real change... sqlite> delete from Tags where id > 3000; better, tag manager now starts within 7 seconds... sqlite> delete from Tags where id > 1000; nice, tag manager starts within 1-2 seconds. I would say things work within an acceptable time. going even further and just leave 500 tags within database things rock! === - Startup time digikam: 5000+ tags 3min 20 seconds vs. 500 tags 13 seconds! - Startup time "Open Tag Manager": 5000+ 23 seconds vs. 500 tags 1 second! - Click on "+" to add a new tag: 5000+ 5 seconds vs. 500 tags realtime - Entering a new tag in that window and click save makes digikam become unresponsive for 1min and 33 seconds, doh! vs. 500 tags 2 seconds - Deleting the newly created tag takes nearly the same time, 1min and 36s vs. 500 tags 2 seconds === So things get massiv slower when reaching 2000 or 3000 tags. Feels some indexes are missing!? Maybe I have the time tomorrow for testing out. cheers bzrudi
I think the problem can be the auto-completer for tag names in search tree-view. This one is populated with database contents and is updated at each changes. This will explain why with a fresh DB, the problem remain... Gilles
digiKam 5.6.0 is now released and available as bundle for Linux, MacOS and Windows. https://www.digikam.org/news/2017-06-21-5.6.0-release-announcement/ Can you check if problem still exists with this version ? Thanks in advance Gilles Caulier
New digiKam 5.7.0 are built with current implementation as pre-release bundles: https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM Problem still reproducible ?
Hey Gilles, sadly problem still exists in latest 5.7.0 beta.
Can you reproduce the dysfunction using digiKam 6.0.0 pre-release bundle available here : https://files.kde.org/digikam/
Can you reproduce the dysfunction using the last digiKam 6.0.0-beta3 just released ? https://www.digikam.org/news/2018-12-30-6.0.0-beta3_release_announcement/
Good news, After 2 weeks of works, the pre-release 6.1.0 bundles are now reconstructed from scratch with: - All OpenCV options for CUDA, OPenMP, and OPenCL disabled to prevent crashes in face management. - A large upgrade of Qt5 from 5.9.7 to 5.11.3. - An upgrade to KF5 5.55. - An upgrade to Ffmpeg 3.3.9 - The fontconfig/freetype integration in the bundle to reduce system dependencies Files can be downloaded here : https://files.kde.org/digikam/ Please test and report. Gilles Caulier
Hi, Can you check if this problem still exist with last weekly bundle build of digiKam 7.0.0 available here: https://files.kde.org/digikam/ Thanks in advance Gilles Caulier
digiKam 7.0.0 stable release is now published: https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/ We need a fresh feedback on this file using this version. Thanks in advance Gilles Caulier
The main problem was sorting the tags. We had created a QCollator object for every sorting process of a tag name. Since the QCollator loads a database file every time, this is very slow. We now have a static QCollator. A test with 30,000 tags shows a good speed here. I'm closing the bug now. If there are still problems, please open again. Maik