Bug 216894

Summary: Slow quadratic runtime generating fingerprints or thumbnails in beta6 during a big import
Product: [Applications] digikam Reporter: Scott Crosby <scrosby>
Component: Import-IconViewAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: normal CC: languitar, marcel.wiesweg
Priority: NOR    
Version: 1.0.0   
Target Milestone: ---   
Platform: Debian testing   
OS: Linux   
Latest Commit: Version Fixed In: 2.0.0
Sentry Crash Report:

Description Scott Crosby 2009-12-01 06:51:58 UTC 
Version:           1.0.0beta6 (using KDE 4.3.2)
OS:                Linux
Installed from:    Debian testing/unstable Packages

Slow quadratic runtime generating fingerprints or thumbnails in beta6 during a big import


I am importing about 50k images by pointing digikam at an already-made
directory tree, and digikam works great! However, in order to find
duplicates digikam must generate fingerprints. When doing this, the
program gets progressively slower. I suspect that this is because the
progress dialog continously grows the list of images, never rolling
the first entries off of the list. As the dialog auto-scrolls to the
bottom on every update, it is virtually impossible to see any entries
EXCEPT the last few. This causes digikam to get progressively slower
and slower. After a few thousand images, it is >5x slower.

If I abort fingerprint generation and try to restart and 'scan' for
images that need fingerprints, digikam remains slow.  However, if I
quit digikam entirely and restart and then have it 'scan', then it
generates fingerprints at full speed.... for a while, until the
quadratic slowdown hits again.
Comment 1 caulier.gilles 2009-12-01 10:04:56 UTC 
So, for you, if i understand properly, slower import is due of list from progress dialog witch take a while to show items. right ?

Gilles Caulier
Comment 2 Scott Crosby 2009-12-01 16:49:40 UTC 
Yes. It progressively slows down, and the list in the dialog gets progressively longer. Memory usage also gets progressively bigger. I suspect that the dialog box is the cause, but I do not know.


Note, I am not doing a digikam 'import'. 

Replication instructions:

  1. Find a directory with a lot of images. (10k-50k)
  2. Make a new collection pointing to that directory [Settings -> Configure DIgikam -> Local Collection -> 'Add Collection']
  3. In the menu, select 'rebuild fingerprints'
  4. Select 'scan'.

Observe how fast it runs, then wait about 5-10 minutes and it is much slower. Quit and restart digikam and repeat steps 3&4, and it is again fast.

In addition, when I run digikam under oprofile, it appears that the dialog box may be responsible for 90% of the CPU utilization.
Comment 3 Marcel Wiesweg 2009-12-17 21:59:35 UTC 
You are quite right, the list including pixmap in the progress dialog grows linearly.
A possible solution would be to remove the scroll bar and remove entries from the beginning of the list.
Comment 4 Johannes Wienke 2009-12-17 22:05:26 UTC 
If this is such a performance killer I would really vote for this. Showing only the last 5 generated fingerprints should really be enough. Who really wants to scroll through the whole list?
Comment 5 caulier.gilles 2009-12-17 22:13:37 UTC 
Agree...

I have make this dialog in case to identify items if something is wrong...

Why not to do something like scan dialog at kphotoalbum startup ?

Gilles
Comment 6 Johannes Wienke 2009-12-17 22:39:14 UTC 
What does it look like? Never used kphotoalbum ;)
Comment 7 caulier.gilles 2009-12-17 23:00:38 UTC 
Dialog is similar than digiKam progress dialog, xcpeted that list view do not exist. There is only a KSqueezedLabel which display current items processed one by one...

Gilles Caulier
Comment 8 Marcel Wiesweg 2011-03-05 21:42:24 UTC 
Fixed since a long time, Gilles changed it as described in #7