Bug 391115 - Very slow search, full text search (FTS) proposition for huge speedup.
Summary: Very slow search, full text search (FTS) proposition for huge speedup.
Status: REPORTED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Engine (show other bugs)
Version: 5.8.0
Platform: Other All
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-02-26 18:09 UTC by Michał Karwowski
Modified: 2023-04-25 14:42 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
attachment-32528-0.html (2.61 KB, text/html)
2018-02-26 18:57 UTC, Michał Karwowski
Details
attachment-15315-0.html (1.39 KB, text/html)
2022-01-10 22:26 UTC, Michał Karwowski
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Karwowski 2018-02-26 18:09:32 UTC
Hi.

I have more than 100000 (100k+) pictures in my DigiKam library.
DigiKam seems to be VERY SLOW with such amount of pictures.

Using Sqlite database on RamDisk or using MySql database doesn't realy help when I try to filter (search) pictures.


I notice that the speed issue is caused, by inefficent queries.

I cought SQL queris which looks like:

SELECT * from .. WHERE some_key LIKE '%query%'  (but ofcourse query is done across multiple tables)



Queries to database with 'like' operator are very SLOW BY DESIGN.

Such queries requires FULL TABLE SCAN (reading every record and comparing it with a query).

Nowadays many database backends support builtin full text search engines.

Even Sqlite have such one!

========
Sqlite
========

-----
EXAMPLE SPEED GAIN by using FTS index instead of 'like' operator.
from: https://www.sqlite.org/fts3.html
-----
SELECT count(*) FROM enrondata1 WHERE content MATCH 'linux';  /* 0.03 seconds */
SELECT count(*) FROM enrondata2 WHERE content LIKE '%linux%'; /* 22.5 seconds */



http://www.sqlitetutorial.net/sqlite-full-text-search/
https://sqlite.org/fts5.html

=========
PostgreSQL
=========
https://www.postgresql.org/docs/9.5/static/textsearch.html

=========
MySQL
=========
https://dev.mysql.com/doc/refman/5.7/en/fulltext-search.html


I belive introducing Full text search to DigiKam, could drasticly improve search/filtering performace.

Without this feature, this application is useless.
Queries TAKES MINUTES (on i7, 16GB ram, SSD).

Moreover it's not clear for the user if DigiKam is realy doing anything during search. After minutes results appear.

Later scrollig is as fast as usually, the problem is with searchin/filtering.
Comment 1 caulier.gilles 2018-02-26 18:20:32 UTC
Thanks for this analysis and for the proposal to improve SQL queries.

Just to be sure, which digiKam version do you use exactly ?

Gilles Caulier
Comment 2 Michał Karwowski 2018-02-26 18:57:13 UTC
Created attachment 111026 [details]
attachment-32528-0.html

2018-02-26 19:20 GMT+01:00 <bugzilla_noreply@kde.org>:

> https://bugs.kde.org/show_bug.cgi?id=391115
>
> caulier.gilles@gmail.com changed:
>
>            What    |Removed                     |Added
> ------------------------------------------------------------
> ----------------
>                  CC|                            |caulier.gilles@gmail.com
>
> --- Comment #1 from caulier.gilles@gmail.com ---
> Thanks for this analysis and for the proposal to improve SQL queries.
>
> Just to be sure, which digiKam version do you use exactly ?
>

*I'm using DigiKam 5.8.0 (on Windows 7 x64, currently with MySQL - mariadb
10.2.10 winx64).*

DigiKam seems to be best choice in the future (if search speed would be
improved).
I have tried many applications in different configurations but most of them
have some issues.
Some programs doesn't allow to search for file/folder names or time range
(limitting to time range from NOW only), sometimes accidently disconecting
USB drive couses huge library to disapear for ever).
DigiKam doesn't has such problems. In DigiKam we had speed issues which
disqualifies this application for larger databases.

It seems to be realy good application. I spend many hours to find out, why
searches in digikam are so slow. It is written in C++, and Qt -
technologies which I completly dont undertand :(
But I home introducing full text search (instead of 'like' queries) would
be quite easy for people having that technologies.


best regards,
Michał Karwowski


>
> Gilles Caulier
>
> --
> You are receiving this mail because:
> You reported the bug.
Comment 3 Maik Qualmann 2018-02-26 20:37:39 UTC
I see a big difference in the syntax between SQLite and MySQL when using MATCH. MYSQL requires a FULLTEXT index for the table columns, and SQLite requires FTS5 tables. If I see it correctly, MATCH does not really find parts within a word, although there is the * wildcard, but the search is only from the beginning of the word. I can not imagine that the search takes minutes. I have about 50k images here, a search takes about 1 second. The problem is more because when a lot of images have been found, it takes a little while until the icon model is loaded. Please install DebugView from Microsoft. How long does it take to enter the search text until you see "digikam.database: Search result: XXXX" in the log?

Maik
Comment 4 Michał Karwowski 2018-02-27 00:02:04 UTC
Thanks for 'DebugView' - I didn't know such tool exists.

I made more detail research on the problem.


To be precise, my total media count from DigiKam database stats is about 108012.
(I didn't add all folders to DigiKam)

When I run the application I see in a log:
[8144] digikam.database: Search result: 1512168

But: "1 512 168" is much more than "108 012"!
Strange.. what does it mean?


==========================
==========================
Text filter (available on the right) 
==========================
==========================

The problem with speed appears to be related to "Text filter" which I use.
Filtering pictures with that filter takes from 20 seconds to 3 minutes.

DebugView doesn't show logs for database on "text filter".
Thus now, I believe that's it's not related to database engine,
but rather internal filtering system.

Filter query: 'tajlandia' took 2 minutes 31 seconds (to display results - 8586 items).
Filter query: 'żaba' took 1 minute 25 seconds (to display results - 1862 items).
etc.
Clearing filter also takes lots of time.

I did tests with offline and online albums.
By term offline I mean disconnected USB drive with albums.
By term online I mean connected USB drive with albums.
Measurements I have don after the inilial digikam scan.


==========================
==========================
Search (available on the left) 
==========================
==========================
When I use search I see digikam.database logs on DebugView.
This suggests that for searching database query is made in contrast to filtering.

To be clear I use: digiKam  5.8.0 with mariadb 10.2.10 winx64

........................
Query for: "gosia" (164794 results?)
........................
00001448	21:55:15	[10424] digikam.general: keywords changed to ' "gosia" '	
00001449	21:55:15	[10424] digikam.geoiface: ----	
00001450	21:55:15	[10424] digikam.geoiface: ----	
00001451	21:55:15	[10424] digikam.geoiface: ----	
00001452	21:55:15	[10424] digikam.general: Using  8  CPU core to run threads	
00001453	21:55:15	[10424] digikam.general: Action Thread run  1  new jobs	
00001454	21:55:15	[10424] digikam.database: " ( (  (Albums.relativePath LIKE ?) OR (Images.name LIKE ?) OR (Images.id IN    (SELECT imageid FROM ImageTags     WHERE tagid IN    (SELECT id FROM Tags WHERE name LIKE ?))) OR (Albums.caption LIKE ?) OR (Albums.collection LIKE ?) OR (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?)) OR (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?))  ) ) "	
00001455	21:55:15	[10424] digikam.database: Search query:	
00001458	21:55:16	[10424] digikam.database: Search result: 164794	
# Result is ready but not visible

00001459	21:55:19	[10424] digikam.general: Cancel Main Thread	
00001507	21:55:22	[10424] digikam.general: One job is done	
00001508	21:55:24	[10424] digikam.general: Cancel Main Thread	
00001509	21:55:45	[10424] digikam.geoiface: ----	
00001512	21:55:50	[10424] digikam.general: keywords changed to ' "gosia" '	
00001513	21:55:50	[10424] digikam.general: same keywords as before, ignoring...	
00001514	21:55:58	[10424] digikam.general: keywords changed to ' "ekp" '	
# application is responsive again



........................
Query for: "ekp" (394240 results?)
........................

00001514	21:55:58	[10424] digikam.general: keywords changed to ' "ekp" '	
00001515	21:55:58	[10424] digikam.geoiface: ----	
00001516	21:55:58	[10424] digikam.geoiface: ----	
00001517	21:55:58	[10424] digikam.geoiface: ----	
00001518	21:55:58	[10424] digikam.general: Using  8  CPU core to run threads	
00001519	21:55:58	[10424] digikam.general: Action Thread run  1  new jobs	
00001520	21:55:58	[10424] digikam.database: " ( (  (Albums.relativePath LIKE ?) OR (Images.name LIKE ?) OR (Images.id IN    (SELECT imageid FROM ImageTags     WHERE tagid IN    (SELECT id FROM Tags WHERE name LIKE ?))) OR (Albums.caption LIKE ?) OR (Albums.collection LIKE ?) OR (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?)) OR (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?))  ) ) "	
00001521	21:55:58	[10424] digikam.database: Search query:	
00001524	21:56:00	[10424] digikam.database: Search result: 394240	
00001525	21:56:08	[10424] digikam.general: Cancel Main Thread	
00001526	21:56:18	[10424] digikam.general: One job is done	
00001575	21:56:23	[10424] digikam.geoiface: ----	
00001576	21:56:23	[10424] digikam.general: Cancel Main Thread	
00001582	21:57:17	[10424] digikam.general: keywords changed to ' "ekp" '	
00001582	21:57:17	[10424] digikam.general: keywords changed to ' "ekp" '	
00001583	21:57:17	[10424] digikam.general: same keywords as before, ignoring...	
00001584	21:57:19	[10424] digikam.general: Request to get thumbnail for  ""	
00001585	21:57:19	[10424] digikam.general: Video file  ""  does not exist.	
00001586	21:57:19	[10424] digikam.general: Failed to extract video thumbnail for  ""	
00001587	21:57:19	[10424] digikam.general: Request to get thumbnail for  ""	
00001588	21:57:19	[10424] digikam.general: Video file  ""  does not exist.	
00001589	21:57:19	[10424] digikam.general: Failed to extract video thumbnail for  ""	
00001637	21:57:23	[10424] digikam.general: 
# Interface is responsive again


........................
Query for: "ekpzg" 166474 results?)
........................
00000072	21:17:20	[2136] digikam.general: Cancel Main Thread	
00000074	21:17:30	[2136] digikam.general: One job is done	
00000075	21:17:30	[2136] digikam.general: Cancel Main Thread	
00000076	21:17:38	[2136] digikam.general: Using  8  CPU core to run threads	
00000077	21:17:38	[2136] digikam.general: Action Thread run  1  new jobs	
00000078	21:17:38	[2136] digikam.database: " ( (  (Albums.relativePath LIKE ?) OR (Images.name LIKE ?) OR (Images.id IN    (SELECT imageid FROM ImageTags     WHERE tagid IN    (SELECT id FROM Tags WHERE name LIKE ?))) OR (Albums.caption LIKE ?) OR (Albums.collection LIKE ?) OR (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?)) OR (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?))  ) ) "	
00000079	21:17:38	[2136] digikam.database: Search query:	
00000082	21:17:39	[2136] digikam.database: Search result: 166474	
00000083	21:17:47	[2136] digikam.geoiface: ----	
00000084	21:17:48	[2136] digikam.general: Cancel Main Thread	
00000134	21:18:08	[2136] digikam.general: One job is done	
00000135	21:18:08	[2136] digikam.general: keywords changed to ' "ekpzg" '	
00000136	21:18:08	[2136] digikam.general: same keywords as before, ignoring...	
00000137	21:18:08	[2136] digikam.general: Cancel Main Thread	
00000138	21:18:08	[2136] digikam.geoiface: ----	

........................
Query for "Jurek" - 26488 results
........................
00001637	21:57:23	[10424] digikam.general: keywords changed to ' "Jurek" '	
00001638	21:57:23	[10424] digikam.geoiface: ----	
00001639	21:57:23	[10424] digikam.geoiface: ----	
00001640	21:57:23	[10424] digikam.geoiface: ----	
00001641	21:57:23	[10424] digikam.general: Using  8  CPU core to run threads	
00001642	21:57:23	[10424] digikam.general: Action Thread run  1  new jobs	
00001643	21:57:23	[10424] digikam.database: " ( (  (Albums.relativePath LIKE ?) OR (Images.name LIKE ?) OR (Images.id IN    (SELECT imageid FROM ImageTags     WHERE tagid IN    (SELECT id FROM Tags WHERE name LIKE ?))) OR (Albums.caption LIKE ?) OR (Albums.collection LIKE ?) OR (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?)) OR (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?))  ) ) "	
00001644	21:57:23	[10424] digikam.database: Search query:	
00001647	21:57:24	[10424] digikam.database: Search result: 26488	
00001648	21:57:24	[10424] digikam.general: Cancel Main Thread	
00001650	21:58:10	[10424] digikam.general: One job is done	
00001651	21:58:10	[10424] digikam.general: keywords changed to ' "Jurek" '	
00001652	21:58:10	[10424] digikam.general: same keywords as before, ignoring...	
00001653	21:58:10	[10424] digikam.general: Cancel Main Thread	
00001654	21:58:10	[10424] digikam.geoiface: ----	
00001655	21:58:11	[10424] digikam.general: Request to get thumbnail for  ""	
00001656	21:58:11	[10424] digikam.general: Video file  ""  does not exist.	
00001657	21:58:11	[10424] digikam.general: Failed to extract video thumbnail for  ""	
00001658	21:58:19	[10424] digikam.geoiface: ----	

==============================
==============================
Summary
==============================
==============================

From this analysis it seems, that.
Huge performance issues are not related to SQL queries (which I was wrongly suspecting).
Querying small sets of pictures gives results quite fast.
In contrast to queries which results in thousands of pictures.
Queries are probably quite fast, with 1-second accuracy - queries took 1 second (probably less). 

Problem is rather related to displaying huge amount of pictures.

This is especially visible in when 'text filter' is used.
When using "text filter" -probably internal DigiKam system for filtering is used, which slows application down.

When I use search - which uses database side filtering.
After DigiKam receives results from the database it freezes until a huge subset of pictures is displayed (thousands of pictures).

My mistake with blaming database, but performance issues exist.
Comment 5 Michał Karwowski 2018-02-27 00:09:22 UTC
(In reply to Michał Karwowski from comment #4)
> Thanks for 'DebugView' - I didn't know such tool exists.
> 
> I made more detail research on the problem.
> 
> 
> To be precise, my total media count from DigiKam database stats is about
> 108012.
> (I didn't add all folders to DigiKam)
> 
> When I run the application I see in a log:
> [8144] digikam.database: Search result: 1512168
> 
> But: "1 512 168" is much more than "108 012"!
> Strange.. what does it mean?
> 
> 
> ==========================
> ==========================
> Text filter (available on the right) 
> ==========================
> ==========================
> 
> The problem with speed appears to be related to "Text filter" which I use.
> Filtering pictures with that filter takes from 20 seconds to 3 minutes.
> 
> DebugView doesn't show logs for database on "text filter".
> Thus now, I believe that's it's not related to database engine,
> but rather internal filtering system.
> 
> Filter query: 'tajlandia' took 2 minutes 31 seconds (to display results -
> 8586 items).
> Filter query: 'żaba' took 1 minute 25 seconds (to display results - 1862
> items).
> etc.
> Clearing filter also takes lots of time.
> 
> I did tests with offline and online albums.
> By term offline I mean disconnected USB drive with albums.
> By term online I mean connected USB drive with albums.
> Measurements I have don after the inilial digikam scan.
> 
> 
> ==========================
> ==========================
> Search (available on the left) 
> ==========================
> ==========================
> When I use search I see digikam.database logs on DebugView.
> This suggests that for searching database query is made in contrast to
> filtering.
> 
> To be clear I use: digiKam  5.8.0 with mariadb 10.2.10 winx64
> 
> ........................
> Query for: "gosia" (164794 results?)
> ........................
> 00001448	21:55:15	[10424] digikam.general: keywords changed to ' "gosia" '	
> 00001449	21:55:15	[10424] digikam.geoiface: ----	
> 00001450	21:55:15	[10424] digikam.geoiface: ----	
> 00001451	21:55:15	[10424] digikam.geoiface: ----	
> 00001452	21:55:15	[10424] digikam.general: Using  8  CPU core to run threads	
> 00001453	21:55:15	[10424] digikam.general: Action Thread run  1  new jobs	
> 00001454	21:55:15	[10424] digikam.database: " ( (  (Albums.relativePath LIKE
> ?) OR (Images.name LIKE ?) OR (Images.id IN    (SELECT imageid FROM
> ImageTags     WHERE tagid IN    (SELECT id FROM Tags WHERE name LIKE ?))) OR
> (Albums.caption LIKE ?) OR (Albums.collection LIKE ?) OR (Images.id IN 
> (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?)) OR
> (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment
> LIKE ?))  ) ) "	
> 00001455	21:55:15	[10424] digikam.database: Search query:	
> 00001458	21:55:16	[10424] digikam.database: Search result: 164794	
> # Result is ready but not visible
> 
> 00001459	21:55:19	[10424] digikam.general: Cancel Main Thread	
> 00001507	21:55:22	[10424] digikam.general: One job is done	
> 00001508	21:55:24	[10424] digikam.general: Cancel Main Thread	
> 00001509	21:55:45	[10424] digikam.geoiface: ----	
> 00001512	21:55:50	[10424] digikam.general: keywords changed to ' "gosia" '	
> 00001513	21:55:50	[10424] digikam.general: same keywords as before,
> ignoring...	
> 00001514	21:55:58	[10424] digikam.general: keywords changed to ' "ekp" '	
> # application is responsive again
> 
> 
> 
> ........................
> Query for: "ekp" (394240 results?)
> ........................
> 
> 00001514	21:55:58	[10424] digikam.general: keywords changed to ' "ekp" '	
> 00001515	21:55:58	[10424] digikam.geoiface: ----	
> 00001516	21:55:58	[10424] digikam.geoiface: ----	
> 00001517	21:55:58	[10424] digikam.geoiface: ----	
> 00001518	21:55:58	[10424] digikam.general: Using  8  CPU core to run threads	
> 00001519	21:55:58	[10424] digikam.general: Action Thread run  1  new jobs	
> 00001520	21:55:58	[10424] digikam.database: " ( (  (Albums.relativePath LIKE
> ?) OR (Images.name LIKE ?) OR (Images.id IN    (SELECT imageid FROM
> ImageTags     WHERE tagid IN    (SELECT id FROM Tags WHERE name LIKE ?))) OR
> (Albums.caption LIKE ?) OR (Albums.collection LIKE ?) OR (Images.id IN 
> (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?)) OR
> (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment
> LIKE ?))  ) ) "	
> 00001521	21:55:58	[10424] digikam.database: Search query:	
> 00001524	21:56:00	[10424] digikam.database: Search result: 394240	
> 00001525	21:56:08	[10424] digikam.general: Cancel Main Thread	
> 00001526	21:56:18	[10424] digikam.general: One job is done	
> 00001575	21:56:23	[10424] digikam.geoiface: ----	
> 00001576	21:56:23	[10424] digikam.general: Cancel Main Thread	
> 00001582	21:57:17	[10424] digikam.general: keywords changed to ' "ekp" '	
> 00001582	21:57:17	[10424] digikam.general: keywords changed to ' "ekp" '	
> 00001583	21:57:17	[10424] digikam.general: same keywords as before,
> ignoring...	
> 00001584	21:57:19	[10424] digikam.general: Request to get thumbnail for  ""	
> 00001585	21:57:19	[10424] digikam.general: Video file  ""  does not exist.	
> 00001586	21:57:19	[10424] digikam.general: Failed to extract video thumbnail
> for  ""	
> 00001587	21:57:19	[10424] digikam.general: Request to get thumbnail for  ""	
> 00001588	21:57:19	[10424] digikam.general: Video file  ""  does not exist.	
> 00001589	21:57:19	[10424] digikam.general: Failed to extract video thumbnail
> for  ""	
> 00001637	21:57:23	[10424] digikam.general: 
> # Interface is responsive again
> 
> 
> ........................
> Query for: "ekpzg" 166474 results?)
> ........................
> 00000072	21:17:20	[2136] digikam.general: Cancel Main Thread	
> 00000074	21:17:30	[2136] digikam.general: One job is done	
> 00000075	21:17:30	[2136] digikam.general: Cancel Main Thread	
> 00000076	21:17:38	[2136] digikam.general: Using  8  CPU core to run threads	
> 00000077	21:17:38	[2136] digikam.general: Action Thread run  1  new jobs	
> 00000078	21:17:38	[2136] digikam.database: " ( (  (Albums.relativePath LIKE
> ?) OR (Images.name LIKE ?) OR (Images.id IN    (SELECT imageid FROM
> ImageTags     WHERE tagid IN    (SELECT id FROM Tags WHERE name LIKE ?))) OR
> (Albums.caption LIKE ?) OR (Albums.collection LIKE ?) OR (Images.id IN 
> (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?)) OR
> (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment
> LIKE ?))  ) ) "	
> 00000079	21:17:38	[2136] digikam.database: Search query:	
> 00000082	21:17:39	[2136] digikam.database: Search result: 166474	
> 00000083	21:17:47	[2136] digikam.geoiface: ----	
> 00000084	21:17:48	[2136] digikam.general: Cancel Main Thread	
> 00000134	21:18:08	[2136] digikam.general: One job is done	
> 00000135	21:18:08	[2136] digikam.general: keywords changed to ' "ekpzg" '	
> 00000136	21:18:08	[2136] digikam.general: same keywords as before,
> ignoring...	
> 00000137	21:18:08	[2136] digikam.general: Cancel Main Thread	
> 00000138	21:18:08	[2136] digikam.geoiface: ----	
> 
> ........................
> Query for "Jurek" - 26488 results
> ........................
> 00001637	21:57:23	[10424] digikam.general: keywords changed to ' "Jurek" '	
> 00001638	21:57:23	[10424] digikam.geoiface: ----	
> 00001639	21:57:23	[10424] digikam.geoiface: ----	
> 00001640	21:57:23	[10424] digikam.geoiface: ----	
> 00001641	21:57:23	[10424] digikam.general: Using  8  CPU core to run threads	
> 00001642	21:57:23	[10424] digikam.general: Action Thread run  1  new jobs	
> 00001643	21:57:23	[10424] digikam.database: " ( (  (Albums.relativePath LIKE
> ?) OR (Images.name LIKE ?) OR (Images.id IN    (SELECT imageid FROM
> ImageTags     WHERE tagid IN    (SELECT id FROM Tags WHERE name LIKE ?))) OR
> (Albums.caption LIKE ?) OR (Albums.collection LIKE ?) OR (Images.id IN 
> (SELECT imageid FROM ImageComments   WHERE type=? AND comment LIKE ?)) OR
> (Images.id IN  (SELECT imageid FROM ImageComments   WHERE type=? AND comment
> LIKE ?))  ) ) "	
> 00001644	21:57:23	[10424] digikam.database: Search query:	
> 00001647	21:57:24	[10424] digikam.database: Search result: 26488	
> 00001648	21:57:24	[10424] digikam.general: Cancel Main Thread	
> 00001650	21:58:10	[10424] digikam.general: One job is done	
> 00001651	21:58:10	[10424] digikam.general: keywords changed to ' "Jurek" '	
> 00001652	21:58:10	[10424] digikam.general: same keywords as before,
> ignoring...	
> 00001653	21:58:10	[10424] digikam.general: Cancel Main Thread	
> 00001654	21:58:10	[10424] digikam.geoiface: ----	
> 00001655	21:58:11	[10424] digikam.general: Request to get thumbnail for  ""	
> 00001656	21:58:11	[10424] digikam.general: Video file  ""  does not exist.	
> 00001657	21:58:11	[10424] digikam.general: Failed to extract video thumbnail
> for  ""	
> 00001658	21:58:19	[10424] digikam.geoiface: ----	
> 
> ==============================
> ==============================
> Summary
> ==============================
> ==============================
> 
> From this analysis it seems, that.
> Huge performance issues are not related to SQL queries (which I was wrongly
> suspecting).
> Querying small sets of pictures gives results quite fast.
> In contrast to queries which results in thousands of pictures.
> Queries are probably quite fast, with 1-second accuracy - queries took 1
> second (probably less). 
> 
> Problem is rather related to displaying huge amount of pictures.
> 
> This is especially visible in when 'text filter' is used.
> When using "text filter" -probably internal DigiKam system for filtering is
> used, which slows application down.
> 
> When I use search - which uses database side filtering.
> After DigiKam receives results from the database it freezes until a huge
> subset of pictures is displayed (thousands of pictures).
> 
> My mistake with blaming database, but performance issues exist.

So, Maik Qualmann, as you expected its probably related to loading 'icon model'.

Mayby it would be possible to load icon model od demand (when it would be 'in a view'). For example, Google Picasa somehow is able to show lot's of pictures quickly.

But probably it is a topic for another ticket.
Comment 6 Maik Qualmann 2019-01-03 19:47:14 UTC
Git commit abd32eefb24ac94c6a5343869578eabdd9ec9f9d by Maik Qualmann.
Committed on 03/01/2019 at 19:43.
Pushed by mqualmann into branch 'master'.

use ReadWriteLock for the CollectionManager
Loading a large item view is now about 4x
faster here with MySQL
Related: bug 391840

M  +9    -9    core/libs/database/collection/collectionmanager_album.cpp
M  +43   -39   core/libs/database/collection/collectionmanager_location.cpp
M  +2    -0    core/libs/database/collection/collectionmanager_p.h

https://commits.kde.org/digikam/abd32eefb24ac94c6a5343869578eabdd9ec9f9d
Comment 7 Maik Qualmann 2019-04-22 18:37:51 UTC
Git commit 1f8edbda8bff2a3e201b8870a686f0066e20d590 by Maik Qualmann.
Committed on 22/04/2019 at 18:35.
Pushed by mqualmann into branch 'master'.

use QReadWriteLock for the CollectionManager
Loading a large item view is now about 4x
faster here with MySQL
Related: bug 398921, bug 397901, bug 391840

M  +12   -13   core/libs/database/collection/collectionmanager.cpp
M  +1    -1    core/libs/database/collection/collectionmanager.h
M  +11   -9    core/libs/database/collection/collectionmanager_album.cpp
M  +118  -107  core/libs/database/collection/collectionmanager_location.cpp
M  +1    -1    core/libs/database/collection/collectionmanager_p.cpp
M  +2    -0    core/libs/database/collection/collectionmanager_p.h
M  +1    -1    core/libs/database/coredb/coredbaccess.cpp

https://invent.kde.org/kde/digikam/commit/1f8edbda8bff2a3e201b8870a686f0066e20d590
Comment 8 caulier.gilles 2020-08-04 08:09:32 UTC
digiKam 7.0.0 stable release is now published:

https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/

We need a fresh feedback on this file using this version.

Best Regards

Gilles Caulier
Comment 9 caulier.gilles 2022-01-10 08:11:15 UTC
Hi Michal and happy new year,

Can you check if problem remain with digiKam 7.5.0 pre-release bundle available
here :

https://files.kde.org/digikam/

Thanks in advance

Gilles Caulier
Comment 10 Michał Karwowski 2022-01-10 22:26:26 UTC
Created attachment 145313 [details]
attachment-15315-0.html

Hi, nice that you are writing.

Actually, I'm unable to check it even in next 4 weeks :/
For so many reasons, that it's even hard to explain.
I believe that from mid-February i will have resources, a bit of time etc

/Michał Karwowski


pon., 10 sty 2022 o 09:11 <bugzilla_noreply@kde.org> napisał(a):

> https://bugs.kde.org/show_bug.cgi?id=391115
>
> --- Comment #9 from caulier.gilles@gmail.com ---
> Hi Michal and happy new year,
>
> Can you check if problem remain with digiKam 7.5.0 pre-release bundle
> available
> here :
>
> https://files.kde.org/digikam/
>
> Thanks in advance
>
> Gilles Caulier
>
> --
> You are receiving this mail because:
> You reported the bug.
Comment 11 caulier.gilles 2023-04-25 14:42:02 UTC
@Michał

digiKam 8.0.0 is out. This file still valid ?

Best regards

Gilles Caulier