Bug 178325 - Rebuild all thumbnails consumes large amounts of memory when processing at the end
Summary: Rebuild all thumbnails consumes large amounts of memory when processing at th...
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Thumbs-Engine (show other bugs)
Version: 0.10.0
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-12-20 22:55 UTC by Marcus D. Hanwell
Modified: 2017-07-28 15:03 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 0.10.0


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Marcus D. Hanwell 2008-12-20 22:55:51 UTC
Version:           0.10.0_beta6 (using Devel)
Compiler:          GNU GCC 4.3.2 
OS:                Linux
Installed from:    Compiled sources

I tried rebuilding all thumbnails on my photo collection today. It took quite a long time and didn't seem to use both cores, but the main issue was right at the end when progress was 100%. It said "Processing..." and consumed more and more memory as I waited for it to complete. It got to the stage where it was causing large amounts of swapping and the system became very unresponsive.

I had to kill digikam in order to regain control of my desktop session. When I restarted digikam it seems like the thumbnails I built have been kept as thumbnails are shown very quickly. I haven't looked at the relevant code yet but I assume it was collecting the thumbnails up in some way. I am comfortable with C++, Qt and KDE programming and would be happy to help debug this issue if you have any ideas of where to start. I assume it is caused by the size of my image collection and so debugging would require hours to get to that point again.

My system is an AMD Athlon X2 dual core, 2GB of RAM and the photos reside on an ext3 RAID5 partition. The thumbs were being generated on a separate partition (/home/myuser/Pictures) where the photos have their own partition (/home/photos). The photo partition has 74GB of used space, mixture of JPEG, NEF and a few AVIs. I will upgrade to the latest beta to check if that helps at all.

New digikam is looking great already. Thanks.
Comment 1 Marcus D. Hanwell 2008-12-21 02:15:01 UTC
I wanted to confirm that after upgrading to 0.10.0_beta7 I observed the same issue. Out of control memory consumption on the processing step. Sat at 100%, it appears to process the same image over and over again - only one file name in view. It is difficult to diagnose further as my system becomes extremely unresponsive as it swaps madly.
Comment 2 Marcel Wiesweg 2008-12-21 16:21:37 UTC
SVN commit 899774 by mwiesweg:

Ensure that the thumbnail that is announced is really the next in our list.
Only in this case remove from list and advance.

CCBUG: 178325

 M  +4 -1      batchthumbsgenerator.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=899774
Comment 3 Marcel Wiesweg 2008-12-21 16:24:43 UTC
I have found a possible problem resulting in an endless loop. Please check again if this fixes a problem.

If you feel that you are again getting to your endless loop, find out the PID of digikam, and attach gdb with the command "gdb att <PID>". If you get out of memory before, it may be necessary to run digikam directly under gdb and interrupt with Ctrl+C.
Comment 4 Marcus D. Hanwell 2008-12-23 22:15:02 UTC
Looks like you fixed it, thank you. I checked with current SVN head and it is indeed working. Completed in 2:33 with no errors. Thanks for getting to this so quickly. I will try to keep an eye on head from now on. Thanks.
Comment 5 caulier.gilles 2008-12-23 22:21:43 UTC
Thanks Marcus for feedback

I close this file now.

Gilles Caulier