Bug 195610 - Memory leak in image editor
Summary: Memory leak in image editor
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Plugin-Editor-Curves (show other bugs)
Version: 0.10.0
Platform: Ubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-06-08 00:07 UTC by Jean-Marc Liotier
Modified: 2017-08-08 15:07 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.1.0


Attachments
Massif log file (229.37 KB, text/plain)
2010-01-24 18:33 UTC, Johannes Wienke
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jean-Marc Liotier 2009-06-08 00:07:08 UTC
Version:           2.0.10 (using KDE 4.2.2)
OS:                Linux
Installed from:    Ubuntu Packages

I'm using Digikam 2:0.10.0-1ubuntu3 (from Ubuntu). After a while, memory usage is creeping up - until Digikam is killed as the system runs out of swap space, usually after two or three dozen pictures. I suspect that something is leaking memory.

I have been observing this behaviour all week. My usual processing involves cropping, curves and refocus. Hardware is an AMD Athlon XP with 2 GB RAM.

Here are a few outputs of 'ps faux' :

With image editor closed, right after launch :
USER PID   %CPU %MEM VSZ   RSS    STAT START   TIME COMMAND
jim  22096 12.8  5.3 353620 110540 Ssl  22:27   0:14 digikam

With image editor open, right after launch :
USER PID   %CPU %MEM VSZ   RSS    STAT START   TIME COMMAND
jim  22096 12.4 14.5 541760 299168 Ssl  22:27   0:18 digikam

With image editor closed, but after having edited a couple dozen images :
USER PID   %CPU %MEM VSZ   RSS    STAT START   TIME COMMAND
jim  18491  8.2 65.0 1779100 1341752 Ssl  17:36  22:18 digikam

After having edited a couple dozen images, with image editor open :
USER PID   %CPU %MEM VSZ   RSS    STAT START   TIME COMMAND
jim  18491  8.2 67.9 1837908 1400848 Ssl  17:36  22:19 digikam

While running refocus, right before it got killed for lack of memory :
USER PID   %CPU %MEM VSZ   RSS    STAT START   TIME COMMAND
jim  18491  8.0 76.7 2035800 1583432 Dsl  17:36  23:18 digikam
Comment 1 Andi Clemens 2009-06-08 00:55:24 UTC
To track down a possible memory leak, we need to know more details.
What have you done exactly?
Do you use some filters over and over again? Which ones?
I have checked the editor, it doesn't leak when just opening images, so it must be some filter.

Jean-Marc,

please try to reproduce your workflow and tell us which filters you used (it doesn't have to be the exact order of course).

Andi
Comment 2 Jean-Marc Liotier 2009-06-08 01:02:42 UTC
I am working on a set of 230 JPEG images from a Canon Eos 50D. I work sequentially, one image after the other, without leaving the image editor. In most cases, the actions are : crop, curves, refocus - in that order. In maybe 15% of the cases, I correct for white balance or saturation, but it is not typical because this set is mostly outdoor portraits.

The out of memory kills have if I remember well always happened during refocus - so you might want to put that one on the top of your list of suspects !
Comment 3 caulier.gilles 2009-06-08 06:12:37 UTC
To be sure, run digiKam in valgrind as explained here :

http://www.digikam.org/contrib

Note : digiKam will be slower in this case but source code line where memory is leak, will be printed on the console. digiKam must be compiled with debug symbol.

Gilles Caulier
Comment 4 Jean-Marc Liotier 2009-06-08 23:11:06 UTC
In the next days, I'll try using Valgrind. I'll keep you posted.
Comment 5 Andi Clemens 2009-06-12 18:28:52 UTC
There seems to be a memory leak in the refocus plugin:

==6849== 1,936 bytes in 2 blocks are definitely lost in loss record 473 of 491
==6849==    at 0x402441D: operator new[](unsigned int) (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so)
==6849==    by 0x410A3AF: Digikam::RefocusMatrix::init_c_mat(Digikam::CMat*, int) (matrix.cpp:102)
==6849==    by 0x410A45D: Digikam::RefocusMatrix::allocate_c_mat(int) (matrix.cpp:111)
==6849==    by 0x410B214: Digikam::RefocusMatrix::copy_cvec2mat(Digikam::Mat const*, int) (matrix.cpp:360)
==6849==    by 0x410B518: Digikam::RefocusMatrix::compute_g(Digikam::CMat const*, int, double, double, double, bool) (matrix.cpp:431)
==6849==    by 0x410B5E2: Digikam::RefocusMatrix::compute_g_matrix(Digikam::CMat const*, int, double, double, double, bool) (matrix.cpp:461)
==6849==    by 0x4109840: Digikam::DImgRefocus::refocusImage(unsigned char*, int, int, bool, int, double, double, double, double) (dimgrefocus.cpp:153)
==6849==    by 0x4109625: Digikam::DImgRefocus::filterImage() (dimgrefocus.cpp:131)
==6849==    by 0x4107EF6: Digikam::DImgThreadedFilter::startFilterDirectly() (dimgthreadedfilter.cpp:139)
==6849==    by 0x4107FFB: Digikam::DImgThreadedFilter::run() (dimgthreadedfilter.cpp:152)
==6849==    by 0x50692CD: (within /usr/lib/libQtCore.so.4.5.1)
==6849==    by 0x52786BB: start_thread (in /lib/libpthread-2.10.1.so)


It doesn't seem to waste that much memory, but when you never close the editor and work with rather big images, it might get important.

Gilles,
any idea what is wrong? For me it looks like the memory gets freed again, but I might have overlooked something.

Andi
Comment 6 caulier.gilles 2009-06-12 18:53:32 UTC
Ah... i know this memory leak. It small.

It come from Refocus algorithm taken from gimp plugin. It's a pure math code... good luck .I never found why memory is leak here...

Perhaps a better expert than me can found it... Marcel ?

Gilles Caulier
Comment 7 Andi Clemens 2009-06-15 13:56:53 UTC
Another (small) memory leak:
imagecurves.cpp:422

Although we seem to delete it properly in there and in the destructor, it leaks a little bit.
Any idea?

Andi
Comment 8 caulier.gilles 2009-06-15 14:03:50 UTC
Look line 142 :     d->lut->luts      = NULL;

NULL is assigned to luts, but if this one as already values, we have memory leak...

Gilles Caulier
Comment 9 Andi Clemens 2009-07-02 23:16:34 UTC
Jean-Marc,

I have fixed some memory leak in the imagescurves:

commit 1c8d6a6ec0ca24490b0b96a9e7e51cfe42159196
Author: aclemens <aclemens@283d02a7-25f6-0310-bc7c-ecb5cbfe19da>
Date:   Mon Jun 15 13:21:37 2009 +0000

    fix memory leak in ImageCurves

    git-svn-id: https://svn.kde.org/home/kde/trunk/extragear/graphics/digikam@982291 283d02a7-25f6-0310-bc7c-ecb5cbfe19da

libs/curves/imagecurves.cpp
libs/curves/imagecurves.h


Since your workflow also includes curves adjustment, it might be one of the sources for your memory leak problem.
As I said above, I found another one in the refocus plugin, but unfortunately can't see the actual error. The leak seems to be small though, but when you edit over 200 images, it still might be important.

Can you try with the latest SVN version?
Comment 10 Andrej Krutak 2009-09-26 22:15:46 UTC
I'm experiencing this bug too - after editing like 10-20 images, digikam uses 1GB ram for no apparent reason... No idea yet, whether the source is the image editor, or the digikam itself - however it made my computer freeze (for out-of-memory reasons, obviously) a couple of times already, so it's quite annoying...

After trying valgrind, the non-released memory was 13 MB - after doing some common editing of 3 files.. This seems to not be the problem cause (as that should be more like 50 MB or more).

Therefore, either I didn't reproduce the bug with valgrind being used (yet.. digikam is veeeery lazy when valgrinded), or the occupied memory is correctly released when quitting digikam - but is purposelessly allocated after images are closed. 

I hope I'll find the cause - it's pretty uncomfortable to close digikam after every few edits... :)
Comment 11 Andi Clemens 2009-09-28 13:28:31 UTC
Again, what filter did you use?
I can not experience this, so there must be a problem in a specific imageplugin.

What filetypes are you editing?
What dimensions do they have?
Is your system 32bit or 64bit?

Andi
Comment 12 Andi Clemens 2009-09-28 13:33:29 UTC
(In reply to comment #10)
> After trying valgrind, the non-released memory was 13 MB - after doing some
> common editing of 3 files.. This seems to not be the problem cause (as that
> should be more like 50 MB or more).

The 13 MB can come from an unclosed sqlite connection, this typically wastes 8-10 MB on my system...
Comment 13 Andrej Krutak 2009-09-28 14:04:24 UTC
I only use the most common ones in digikam (though things like layers, adding vignetting, clone/heal brush would be nice :)) - curves, crop, hue-value-saturation and less occassionally bw-convertor and resize... that's about that - editing jpegs, 12Mpix (around 4500x3000), 32bit..

currently I'm using the 0.10.0b3 (ubuntu), but the bug seems to be there ever since 0.10.0 (I don't remember having such problems with 0.9.x, but I was mostly using gimp to postprocess photos back then)..

maybe it's qt-related or something - it's hard to tell, when there only are two people who reported this :)
Comment 14 sts 2009-10-29 09:05:11 UTC
here the same.. I have e.g. an album with 300 images.. with image editor I optimized a lot of images with ctrl+shift+b; alt+o and ctrl+s After some time digikam and xorg fills my memory and swap :(
Comment 15 caulier.gilles 2009-10-29 09:13:23 UTC
sts,

Sound like you didn't use editor plugin here, just you show image as well.

Can you run digiKam through valgrind to show memory leak, and reproduce all step in gui until to see memory dysnfuntion appear. Just report all console messages there. Command line to use is below :

valgrind --tool=memcheck --leak-check=full --error-limit=no digikam

Also, give us more info about your linux box...

Gilles Caulier
Comment 16 Andi Clemens 2009-10-29 09:19:17 UTC
I can confirm this for a long time, and it doesn't seem to be a memory leak, at least in the classical term. Valgrind will show nothing, since at digiKam shutdown all memory is freed, but at runtime it is not. I guess this is due to the fact that IE stay open in the background once used?
Every 2 or 3 weeks I look at this bugreport here and try to find the error, but I don't know what is going wrong.
Memory usage goes up, but valgrind tells me that all is fine.
This one is a nasty bug for sure...
Comment 17 Marcel Wiesweg 2009-10-29 17:25:36 UTC
Is the memory listed for the digikam or the xorg process?
There is valgrind's "massif" tool to analyze heap allocation over time. If it detects e.g. a peak in memory usage it will give you backtraces from where significant portions of this memory were allocated.
Comment 18 Johannes Wienke 2010-01-24 17:51:24 UTC
Ok, I'm just trying to figure this out. Most of the memory allocation happens in DImg::allocate which isn't a big surprise.

But I think I got a suspicion what's going on here:
1. Simply viewing images without manipulations is ok, then all memory is cleaned properly.
2. Doing any operation but not saving the image also results in correctly freed memory.
3. Doing any operation and saving the changed image results in more memory use. So there must be something wrong while saving images.
Comment 19 Johannes Wienke 2010-01-24 18:33:48 UTC
Created attachment 40204 [details]
Massif log file

Massif log file while opening several images in the editor, cropping them a bit, saving them and moving on to the next image.
Comment 20 Marcel Wiesweg 2010-01-24 20:22:21 UTC
How large were the edited images (pixels and bit depth)?
Comment 21 Johannes Wienke 2010-01-24 20:27:41 UTC
Tiff files with 8 bit. All one side > 4000px.
Comment 22 Andi Clemens 2010-01-24 20:59:34 UTC
I discovered different things (but unfortunately I can't remember it correctly now). It was something like:

1. Editing TIFFs : ok
2. Just viewing TIFFs: memory usage was duplicated all the time
3. As soon as I edited a TIFF again: mem was freed again
Comment 23 Marcel Wiesweg 2010-01-25 19:42:53 UTC
In the last snapshot from the Massif output I see 3*125 MB of memory. With the size you indicated 125MB can be two full images in memory. Two times from the histogram sidebar - once from the last cropping operation.

I just tried viewing a few TIFFs under Massif. I closed the image editor window at the end and the last heap snapshot also showed a considerable decline in memory usage. I could not identify any leaks here.
Comment 24 Marcel Wiesweg 2010-01-25 19:44:37 UTC
SVN commit 1080155 by mwiesweg:

Reset histogram image data when url is null.
Should probably also be done when switching to other tab.

CCBUG: 195610

 M  +2 -0      imagepropertiescolorstab.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1080155
Comment 25 Johannes Wienke 2010-01-25 20:03:54 UTC
Hm, maybe I should note that I did not use the sidebar at all during the massif snapshot. It was always closed.

Recently, while editing photos not only for testing, I suddenly had digikam filling more than 2 GB RAM, also without using any of the sidebars except the tools itself.
Comment 26 caulier.gilles 2011-11-24 10:26:38 UTC
This entry still valid with 2.x serie ?

Gilles Caulier
Comment 27 caulier.gilles 2011-12-18 17:24:30 UTC
Jean Marc, 

This file still valid using digiKam 2.4 ?

Gilles Caulier
Comment 28 caulier.gilles 2015-07-01 06:04:47 UTC
New digiKam 4.11.0 is available :

https://www.digikam.org/node/740

Can you reproduce the problem with this release ?
Comment 29 caulier.gilles 2016-07-09 18:36:50 UTC
This problem is not reproducible with last 5.0.0.
I close this file now. Don't hesitate to re-open it if necessary.
Gilles Caulier