| Summary: | performance problems with recursive view and filtering | ||
|---|---|---|---|
| Product: | [Applications] digikam | Reporter: | Arnd Baecker <arnd.baecker> |
| Component: | Albums-Filters | Assignee: | Digikam Developers <digikam-bugs-null> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | ||
| Priority: | NOR | ||
| Version: | 0.9.4 | ||
| Target Milestone: | --- | ||
| Platform: | Debian stable | ||
| OS: | Linux | ||
| Latest Commit: | Version Fixed In: | 1.0.0 | |
| Sentry Crash Report: | |||
| Attachments: | hourglass when filtering many items | ||
Created attachment 22845 [details]
hourglass when filtering many items
When many items are subject to filtering, the hour-glass is displayed
from the very beginning.
Note that does not address the situation described in 1.).
Also, providing a text + progress in the statusbar would be
better (however, I did not manage to get the slot signal connection
working...).
While providing some visual feedback is good, it
does of course not address the origin of the problem,
and I really think that digikam should be faster here!
Arnd, is it still so slow? I once talked about an indicator lamp issue, and with the patch provided for that bug I guess we changed something with the emitting of signals... so maybe now the performance is better?? Andi Andi, well, it is still slow when dealing with many images, as described in the original post. I usually try to avoid going from many images to just a few: For example you have filtered for the Word "Sun" and get 10 hits (in an album with 3000 images, recursive view assumed ;-). Now I would like to filter for "Moon". Erasing the "Sun" (i.e. no filtering) and then typing in "Moon" just takes too long. The workaround is to add the "Moon" to obtain "SunMoon" (and no images) and then remove the "Sun" (to give, e.g. 20 hits). So the 10 -> 0 -> 20 hits route is *much* faster than the 10 -> 3000 -> 20. Arnd P.S.: BTW: Andi, you can configure bugs.kde.org settings to not add you to the CC (hurray!) Arnd, thanks. This option is so annoying for me... because I get email reports twice, one from digikam-devel and the other goes to my private mail. Now everything is fine again... :-) About the performance: I think I need to create a very big album to see those effects, even if I take a album like 2007 or so, I get not enough images. But if you search for "Sun" and press CTRL+Backspace, it should delete the search term in one move (or use the little x next to the line-edit). Or press the indicator lamp, this should be quite fast because it is not filtering. If pressing backspace to delete a character, the search will be emitted after 500ms I guess, so you really need to type fast (:-)) or try the way I described above. Still this is not very satisfying. Maybe the easiest way would be to just search on pressing Enter, although I'm not quite sure how to do this, is there a signal from a line-edit widget that tells you that Enter was pressed? Anyway this would at least solve your problem. Just moved some albums into a new "container" album, now I have 3650 images and yes, it is very slow. Well, I do know that deleting the search term in one move is possible, and that's what I would normally use! However, if I do so I get the 3000 images out of which the "Moon" would only be a few; and it is precisely this cutting down from many images to few which takes too long. Still, even with the Enter variant, the basic problem, discussed in 2.) in the original report will remain. ((Ok, after the mid-air-collision I see that you can reproduce the problem ...)) IMO the problem is in AlbumLister::slotFilterItems() in the second part after the comment // This takes linear time - and deleting seems to take longer. Maybe we should not delete the images from the list if there are more images being deleted than will be preserved and built a new list in this case (not sure if this is doable - its been a while that I looked at that code ... ;-) ? A performance lost can also be found in AlbumLister::matchesFilter, because if you don't delete the images (I commented this part of the code), matchesFilter will take quite long. Sure, 3650 images are a lot, so I actually don't know what we can do here right now (or if it is even possible to speed things up). (In reply to comment #6) > Well, I do know that deleting the search term in one move is possible, > and that's what I would normally use! However, if I do so I get the 3000 images > out of which the "Moon" would only be a few; and it is precisely > this cutting down from many images to few which takes too long. Well I tested it, but even your solution "SunMoon" and then remove "Sun" is not faster, it doesn't seem to filter on the already filtered view but on the whole image list. If I create a search (simple or advanced) with the same criteria, it is 10times faster (sure, we ask the db directly). So maybe this is the only solution here. We need to do quick-search also with db query. But how? Since an album is selected, this could be the first search criteria. We already can do recursive album search in KDE3 and KDE4 in the search dialog, so defining a SQL query shouldn't be a problem. And than we need to add the quick-filter categories to it... maybe we can just adopt the "simple search" algorithm for the quick-filter? A problem for sure might also be that matchFilter is called for every single item, not for just the whole imagelist. So we might have 3000 methods calls for 3000 images instead of just one, also we check all the time if any filters are set, but we really need to do this once, not for every single image. I moved this portion of code into slotFilterItems now and it is a little bit faster, but still not satisfying. Also increasing the timer from 100ms to 500ms helped a little, so it doesn't filter on nearly every keystroke. I will try to implement an SQL solution if it is not too time consuming, in general this should be possible. Andi If I remember correctly there have always been problems that actually inserting or removing items from the album icon view is slow. Another possible bottleneck is looking up data from the database. But you cannot guess. Before starting on a solution, I would check with profiling (callgrind?) where the time is really spent. Yes, this is also a good idea. But I need to compile an optimized version first, callgrind doesn't make too much sense on the full debug version of digiKam I guess. Right now I'm experimenting here... It is a bit faster right now, because I removed some filter lines and signals. I think we filter too often and we also use to many signals, this is slowing down. When I'm done I will create a patch, and we can discuss what I messed up ;-) Andi I made a progress, after testing code with callgrind for hours, there is only one conclusion left: We need to filter with custom SQL statements here. I have implemented filtering with text quickfilter right now and my album with 3600 images can be filtered in less than a second. The old way is just not usable, because it queries the database for every single image. But not only one value, it queries up to 8. If every quickfilter is enabled, we make over 24000!! SQL queries. This is way too much. Right now it is only one ;-) But I need to adopt this for the other filters first. Also we filtered the list when the kio-slaves (digikamalbums) delivered the images, that is ok in general, BUT we filtered everything again in a second step, which is not ok. Another problem: We also filtered everything although no quick filter was ever set, and again this happened twice. So by just displaying an album we filtered images, that shouldn't be filtered anyway. This has been removed right now. Gilles, when I'm done with the patch I think we need to talk about some points that I'm unsure right now :-) Andi Andi, And what about 0.10.0. It have the same problem than 0.9.4 ? Gilles Caulier Yes... it is a little bit faster, but still slow. Unfortunately my patch is gone after the harddisk-crash last month. I have no backup of it. Damn. It not only fixed the way we search for text (SQL statements), but it also removed some signals that were sent, although they shouldn't (which slowed down the search, too). Maybe we should leave this and implement something in 0.10 with model-view. This might be faster, because we don't need to query that much (I hope). Andi This bug is fixed for me with the new image model in 1.0. The filter code has been moved to a QSortFilterProxyModel. Filtering is prepared in a thread (results are stored and applied in the main thread), so what remains in the main thread is the work done by QSortFilterProxyModel to set up the mapping from original to filtered contents, and the view relayout. Both is fast for me. The view has been extensively optimized with callgrind, and QSortFilterProxyModel is, as I believe, well optimized by Qt. |
Version: 0.9.4 svn (using KDE KDE 3.5.5) Installed from: Debian stable Packages With the new recursive view of albums and the quick filters in the status bar some performance problems become visible when the number of involved images becomes large. Let me describe a couple of cases for an album with 16289 images (let's call that "2007" in the following); for the tests below "Include Album Sub-Tree" is activated. (of course, the slower your computer, the more clearly the effects will be visible...). 1.) - Start from an album with a small number of images (e.g 24). The status line says "img_xx.jpg (1 of 24)" - Switch to "2007". The status line does not change for quite some time (approx 5s on a reasonably fast machine). There is no visual indication, that something is happening. Question: Can the whole process be made faster? Possible solution: a progress bar should be added (see also below) Attempt at an analysis: After the selection of "2007" dozens of calls to void AlbumIconView::slotImageListerNewItems followed by DigikamView::slotImageSelected() take place. Only at the very end DigikamView::slotDispatchImageSelected() gets called. Is there anything in these steps which could be optimized? 2.) Start from "2007" and activate the rating filter (e.g. >= 5stars). Here a visual indication in terms of an hour glass as mouse cursor is given (however that does not appear immediately). In my case I get 52 pictures, but it takes about 14 seconds! (Sometimes the status line is not updated properly at the end?) Attempt at an analysis: Most of the time is spent in AlbumLister::slotFilterItems() While the first loop in that routine is not extremely fast, it is the second part after the comment // This takes linear time - and deleting seems to take longer. // Set wait cursor for large numbers. If I understand things correctly, this means that in the example 16289 - 52 images get deleted. This looks like the place to be optimized (by a better strategy, I would guess ...). This also explains, why changing between 4 and 5 star filtering (593 vs. 52 results) is faster. In this case, the bottle-neck is the first loop in AlbumLister::slotFilterItems() which calls matchesFilter for each item. ((Note: We might have to be careful with adding too much complexity in the matchesFilter, or use more clever strategies)) Speed improvements to the first loop would be therefore be very helful, also in view of the next point. 3.) Start from "2007" (with all images) and type some text in the text filter. - if you don't type very fast (I did not manage), one will not see the 2nd letter of a word for a while. The reason is that for a new text void AlbumLister::setTextFilter is called, which sets the filterTimer. Then void AlbumLister::slotFilterItems() is called, before the next call to void AlbumLister::setTextFilter becomes active. - Then the main bottle-necks are as in 1.) and 2.). As a short term solution I would suggest to put the waitCursor at the beginning of void AlbumLister::slotFilterItems(), if d->itemList.count() > 500 (etc.) This should give some visual feedback.