Bug 330230 - Memory leak in Image Quality Sorter
Summary: Memory leak in Image Quality Sorter
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Maintenance-Quality (show other bugs)
Version: 4.0.0
Platform: Compiled Sources Linux
: NOR major
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-01-21 04:40 UTC by Ryan Henderson
Modified: 2017-08-12 12:09 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.0.0
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ryan Henderson 2014-01-21 04:40:28 UTC
- Large memory leak running maintenance image quality sorter.   (500Mb / 500 images)
- Tags are not applied.

There are several of these in the valigrind log, here's the largest:
==26674== 307,183,616 bytes in 107 blocks are possibly lost in loss record 42,037 of 42,037
==26674==    at 0x4C2AFE7: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26674==    by 0x73963F0: unsigned char* Digikam::DImgLoader::new_failureTolerant<unsigned char>(unsigned long) (dimgloader.h:183)
==26674==    by 0x737362B: Digikam::DImg::allocateData() (dimg.cpp:315)
==26674==    by 0x737616F: Digikam::DImg::DImg(Digikam::DImg const&, int, int) (dimg.cpp:136)
==26674==    by 0x738C1C8: Digikam::DImg::smoothScaleClipped(int, int, int, int, int, int) const (dimgscale.cpp:233)
==26674==    by 0x738C47B: Digikam::DImg::smoothScaleClipped(QSize const&, QRect const&) const (dimgscale.cpp:194)
==26674==    by 0x738C514: Digikam::DImg::smoothScale(QSize const&, Qt::AspectRatioMode) const (dimgscale.cpp:183)
==26674==    by 0x738C55F: Digikam::DImg::smoothScale(int, int, Qt::AspectRatioMode) const (dimgscale.cpp:188)
==26674==    by 0x74D2D07: Digikam::PreviewLoadingTask::execute() (previewtask.cpp:433)
==26674==    by 0x74D1783: Digikam::PreviewLoadThread::loadSynchronously(Digikam::LoadingDescription const&) (previewloadthread.cpp:114)
==26674==    by 0x50E734: Digikam::ImageQualityTask::run() (imagequalitytask.cpp:96)

Also:
==26674==    at 0x4C2AFE7: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26674==    by 0x73809DD: unsigned int* Digikam::DImgLoader::new_failureTolerant<unsigned int>(unsigned long) (dimgloader.h:183)
==26674==    by 0x7378128: Digikam::DImg::rotate(Digikam::DImg::ANGLE) (dimg.cpp:2298)
==26674==    by 0x7378576: Digikam::DImg::rotateAndFlip(int) (dimg.cpp:2649)
==26674==    by 0x74C581D: Digikam::LoadSaveThread::exifRotate(Digikam::DImg&, QString const&) (loadsavethread.cpp:335)
==26674==    by 0x74D2D47: Digikam::PreviewLoadingTask::execute() (previewtask.cpp:439)
==26674==    by 0x74D1783: Digikam::PreviewLoadThread::loadSynchronously(Digikam::LoadingDescription const&) (previewloadthread.cpp:114)
==26674==    by 0x50E734: Digikam::ImageQualityTask::run() (imagequalitytask.cpp:96)




Reproducible: Always

Steps to Reproduce:
1. Run image quality sorter with library > 500.
2. Mark maintenance option to clear db and re-run for all

Actual Results:  
2. 500Mb memory leak
3. Tags not applied

Expected Results:  
Digikam would validate my photography skills.  or at least not leak memory and apply the good, bad, ok tags.

4.0.0-beta-3 built from git
gitup current as of Jan 20th 2014.
Ubuntu 13.10 (gcc 2.8.1)
libraw from git 0.16.0
Comment 1 caulier.gilles 2014-01-21 06:52:55 UTC
*** Bug 330227 has been marked as a duplicate of this bug. ***
Comment 2 caulier.gilles 2014-01-21 06:57:52 UTC
Which Image Quality Sorter settings do you use exactly ?

Gilles Caulier
Comment 3 caulier.gilles 2014-01-21 09:27:58 UTC
Gowthan, 

This evening from IRC channel, a thread about this entry has give more important information:

[23:10] <rhenders> I'm seeing some pretty big memory leaks in Image Quality sorter in 4.0.0-beta2:  https://gist.github.com/bitflippersanonymous/8530225
[23:11] <rhenders> Built from source on Ubuntu/AMD64
[23:11] <rhenders> I looked for existing bugs but couldn't find any, thought I'd check here before filing
[23:12] <teprrr> uhu, that looks leaky indeed :o
[23:13] <teprrr> problem is here:            d->imgqsort = new ImgQSort(dimg, d->quality, &pick);
[23:13] <rhenders> It grows to 100GB for my collection of 5k pictures.
[23:13] <rhenders> I could build debug and get line numbers, ...    what file is that in?
[23:13] <teprrr> most likely. run() is being called multiple times, and that new gets called all the time then
[23:14] <teprrr> digikam/utilities/maintenance/imagequalitytask.cpp
[23:14] <teprrr> line 110
[23:15] <teprrr> if that isn't destroyed somewhere in the result handler that's most likely the leaker
[23:16] <teprrr> DImg there is the image data itself, which may be heavy :P
[23:16] <rhenders> There are several leaks in that valgrind log.  The last one is the largest, but they all look similar
[23:16] <teprrr> but if you can, provide a valgrind log with the debug-info enabled and report it to the bugs.kde.org :p
[23:16] <teprrr> hmm, related to maintenance tool?
[23:17] <rhenders> I also see leaks when rebuilding thumbnails, but this was the largest.  I'll rebuild in debug and report it
[23:17] <rhenders> (What's the cmake flag to enable debug)
[23:18] <teprrr> -DCMAKE_BUILD_TYPE=Debug
[23:18] <teprrr> or DebugFull
[23:18] <rhenders> Great, thanks
[23:20] <teprrr> not a problem, I gotta go now away for a while. thanks for noticing such a bug :)

Gilles Caulier
Comment 4 Ryan Henderson 2014-01-21 15:52:12 UTC
Just the defaults AFAIK

Enabled
Speed: 1
Detect Everything
Assign Everything
Reject threshold: 10
Pending threshold: 40
Accepted threshold: 60
Blur Weight: 100
Noise Weight: 100
Compress Weight: 100
Comment 5 Gowtham Ashok 2014-01-22 01:58:57 UTC
Thank you teppr and Ryan. You are right. Image quality sorter data was not getting deleted after completion of task. Fixed it.

Please note that Image quality sorter is still under development. The goal is to tag really bad pictures, not to assess the quality of photographs subjectively. 
As of now, we have to manually set the threshold values according to the type of pictures processed.
Presently, we're trying to detect different cases of blurring in an image. We're also trying to detect the type of pictures so that the threshold values are automatically set.
Comment 6 caulier.gilles 2014-01-22 06:56:04 UTC
Git commit df13317e5d9310f7d07943df8b1501d4706f9e17 by Gowtham Ashok.
Committed on 22/01/2014 at 01:46.
Pushed by gowthamashok into branch 'master'.

Fixed imgqsort huge memory leak

M  +6    -2    utilities/maintenance/imagequalitytask.cpp

http://commits.kde.org/digikam/df13317e5d9310f7d07943df8b1501d4706f9e17

diff --git a/utilities/maintenance/imagequalitytask.cpp b/utilities/maintenance/imagequalitytask.cpp
index 1df7a23..eafa8d8 100644
--- a/utilities/maintenance/imagequalitytask.cpp
+++ b/utilities/maintenance/imagequalitytask.cpp
@@ -78,8 +78,11 @@ void ImageQualityTask::slotCancel()
 {
     d->cancel = true;

-    if (d->imgqsort)
+   //TODO: Investigate if commented code is necessary
+   /* if (d->imgqsort)
         d->imgqsort->cancelFilter();
+
+   */
 }

 void ImageQualityTask::run()
@@ -111,8 +114,9 @@ void ImageQualityTask::run()

             ImageInfo info(d->path);
             info.setPickLabel(pick);
+            if(d->imgqsort)
+                    delete d->imgqsort;                 //delete image data after setting label
         }
-
         // Dispatch progress to Progress Manager
         QImage qimg = dimg.smoothScale(22, 22, Qt::KeepAspectRatio).copyQImage();
         emit signalFinished(qimg);