Version: 0.9.0beta1 (using KDE KDE 3.5.3) Installed from: SuSE RPMs Compiler: gcc 4.1 as well as 4.0.2 OS: Linux Recently I noticed that it seems that imageeditor in 0.9.0 beta1 has a memory leak. When I look at say 50 photos using imageeditor without editing any of the images, just look at them and run through all of these 50 photos my system looses about 300MB of memory. Since I have only a total of 500MB RAM, swap is beeing used at this time more and more. Digikam is getting noticeably more and more slower when loading the next photo in imageeditor since swapping occurs. The strange thing is, that this memory/swap does not seem to be released when I terminate digikam completely. The amount of used swap space (reported by free) remains the very same even a day later this amount is still marked "used". So far only a reboot helped to reclaim the space. So I guess that there must be a memory leak inside the image editor. I found this effect on my suse10.0 (500MB Ram) as well on a suse10.1 (1GB RAM) system. In the latter case it simply take some more photos to run through until on this system swap space is beeing used forever either. Thanks Rainer
Hi krienke, This is certainly a memory leak. Please just re-run digiKam using valgring following the instructions in the HACKING file (at end). Just report the console messages in this thread. Thanks in advance Gilles Caulier
Am Montag, 24. Juli 2006 08:26 schrieb Gilles Caulier: [bugs.kde.org quoted mail] I ran valgrind --tool=memcheck --leak-check=full --error-limit=no /opt/kde3/bin/digikam but there was no information in the output about a big leak as I would have expected because of the observed decrease in free memory: <ran through ~20 photos in image editor, then terminated digikam> ==7042== LEAK SUMMARY: ==7042== definitely lost: 6,349 bytes in 75 blocks. ==7042== indirectly lost: 40,174 bytes in 879 blocks. ==7042== possibly lost: 400 bytes in 3 blocks. ==7042== still reachable: 437,184 bytes in 7,824 blocks. ==7042== suppressed: 0 bytes in 0 blocks. ==7042== Reachable blocks (those to which a pointer was found) are not shown. ==7042== To see them, rerun with: --show-reachable=yes However afterwards I ran free -m then started digikam and in the imageeditor ran through 50 photos, terminated digikam and then again ran free -m: [krienke@bliss:~] 589 > free -m total used free shared buffers cached Mem: 1012 687 324 0 9 571 -/+ buffers/cache: 106 905 Swap: 2055 208 1846 <run of digikam> [krienke@bliss:~] 589 > free -m total used free shared buffers cached Mem: 1012 677 335 0 4 436 -/+ buffers/cache: 236 775 Swap: 2055 377 1677 In this case the usage of swap increased by ~160MB which is the strange behaviour I observed but valgrind does not say anything. I do not understand this. Next I started digikam once more again looked at the very same 50 photos in imageeditor and then terminated digikam again. The free calls are again from before starting digikam and after terminating it: krienke@bliss:~] 587 > free -m total used free shared buffers cached Mem: 1012 688 323 0 10 449 -/+ buffers/cache: 228 783 Swap: 2055 375 1679 <digikam run. Looked at 50 photos in imageeditor> [krienke@bliss:~] 588 > free -m total used free shared buffers cached Mem: 1012 694 317 0 6 390 -/+ buffers/cache: 296 715 Swap: 2055 570 1484 No again another 125 MB of swap has been eaten up. Any idea by what this behaviour might been caused? Thanks Rainer
Perhaps the memory leak is caused by a library used by digiKam image editor. Please check witch libs releases you use on your system : - Exiv2 (metadata management) - lcms (color management) - libjpeg (jpeg files acess) - libpng (png files acess) - libtiff (tiff files acess) Gilles
krienke : 'free' won't tell you anything useful about this problem. The fact that swap space is not recovered is perfectly normal, it won't be until the processes that have been swapped on disk are swapped back into memory. The only thing you should look at is the size of the digikam process while it's running.
Am Montag, 24. Juli 2006 10:28 schrieb Gilles Caulier: > ------- Additional Comments From caulier.gilles free fr 2006-07-24 10:28 > ------- Perhaps the memory leak is caused by a library used by digiKam > image editor. Please check witch libs releases you use on your system : > > - Exiv2 (metadata management) > - lcms (color management) > - libjpeg (jpeg files acess) > - libpng (png files acess) > - libtiff (tiff files acess) > > Gilles Ok. My versions on the suse10.1 system I am sitting at now are: Exiv2: 0.9.1 libcms: 1.15 libjpeg: 6.2.0 libpng: 1.2.8 libtiff: 3.8.2 On my suse10.0 system the libs are: Exiv2: 0.10 libcms: 1.14 libjpeg: 6.2.0 libpng: 1.2.8 libtiff: 3.7.3 Thanks Rainer
I'm use also Suse 10.1 here without memory leak and swap. Note I recommend you to use Exiv2 10.1... Gilles
Am Montag, 24. Juli 2006 11:25 schrieb glaurent@telegraph-road.org: [bugs.kde.org quoted mail] You are right processes are beeing swapped out, but since I did not start any new processes during my tests (except for digikam itself) the swap space should only grow when digikam is started for the first time. However I can repeat this behaviour. So if I terminate digikam then start it again and look at the very same photos then terminate digikam and start it again etc... then swap space usage is increasing with each try. This is not normal. Here is a sequence of free calls after I terminated digikam. After the free call I restarted it and looked at the very same photos again. All in all I did this 10 times (see numbers I added). And after the 10th try the system used 1.4GB swap whereas it had no swap usage initially. Below you can see how swap usage is increasing each time I restarted digikam and looked at the photos: 1 total used free shared buffers cached Mem: 1012 796 215 0 45 519 -/+ buffers/cache: 231 780 Swap: 0 0 0 [krienke@bliss:~] 619 > free -m 2 total used free shared buffers cached Mem: 1012 856 156 0 5 538 -/+ buffers/cache: 312 699 Swap: 2055 0 2055 [krienke@bliss:~] 619 > free -m 3 total used free shared buffers cached Mem: 1012 718 293 0 4 351 -/+ buffers/cache: 363 649 Swap: 2055 251 1803 [krienke@bliss:~] 619 > free -m 4 total used free shared buffers cached Mem: 1012 591 420 0 5 363 -/+ buffers/cache: 222 789 Swap: 2055 363 1691 [krienke@bliss:~] 619 > free -m 5 total used free shared buffers cached Mem: 1012 647 364 0 3 362 -/+ buffers/cache: 282 729 Swap: 2055 538 1516 [krienke@bliss:~] 619 > free -m 6 total used free shared buffers cached Mem: 1012 541 470 0 2 304 -/+ buffers/cache: 235 776 Swap: 2055 692 1362 [krienke@bliss:~] 619 > free -m 7 total used free shared buffers cached Mem: 1012 530 481 0 1 256 -/+ buffers/cache: 272 740 Swap: 2055 927 1127 [krienke@bliss:~] 619 > free -m 8 total used free shared buffers cached Mem: 1012 507 504 0 1 246 -/+ buffers/cache: 259 752 Swap: 2055 1117 937 [krienke@bliss:~] 619 > free -m 9 total used free shared buffers cached Mem: 1012 590 422 0 1 335 -/+ buffers/cache: 252 759 Swap: 2055 1229 825 [krienke@bliss:~] 619 > free -m 10 total used free shared buffers cached Mem: 1012 557 454 0 2 270 -/+ buffers/cache: 285 726 Swap: 2055 1485 569 [krienke@bliss:~] 619 > Next I terminated my KDE session and kdm as well and restarted it and logged in again. The swap space was still used. This should not happen as I think. Am I the only one having this behaviour? Thanks Rainer
Am Montag, 24. Juli 2006 11:50 schrieb Gilles Caulier: [bugs.kde.org quoted mail] Ok I'll try this. What KDE version do you use? I have: [krienke@bliss:~] 631 > rpm -qv kdebase3 qt kdebase3-3.5.3-21.2 qt-4.1.0-29 The RPMS are from SuSEs Build service. Perhaps these non default packages are the problem? Rainer -- --------------------------------------------------------------------------- Rainer Krienke, Universitaet Koblenz, Rechenzentrum, Raum A022 Universitaetsstrasse 1, 56070 Koblenz, Tel: +49 261287 -1312, Fax: -1001312 Mail: krienke@uni-koblenz.de, Web: http://www.uni-koblenz.de/~krienke Get my public PGP key: http://www.uni-koblenz.de/~krienke/mypgp.html ---------------------------------------------------------------------------
on my suse 10.1, i has kdebase3-3.5.1-69.15 installed. Qt 4.1 is not installed. Gilles
Note : a good test under editor to try to reproduce the memory leak is to open the first image of an album where there is a lot of pictures (here 350 JPEG + PNG + RAW files), and toggle on the slideshow tool in loop with a pause of 3s. Here, I have to lets slideshow running during this morning without problem... Gilles
Am Montag, 24. Juli 2006 13:49 schrieb Gilles Caulier: [bugs.kde.org quoted mail] Just tried the slide show feature in the image editor. In this case nothing happened. Swap remained as it was. Next I stopped the slide show and manually went through the same photos (pressing Pages down as fast as the photos are loaded) using the same photos and the same number as for the slide show. The effect was there again. As strange as it is, but for me I can reproduce this effect only by manually pressing the page down key in the image editor *very fast* as fast as your system can load and display the photos in the image editor. This immedeately reproduces this effect for me. Could you give this "method" a try and see if you can reproduce this effect? For me 50 photos each 6Mpix was enough to see the effect. Thanks Rainer
Witch photo format you use to test ? JPEG, RAW, PNG, TIFF ? Gilles
No, i cannot reproduce this problem using PAGE_UP and PAGE_DOWN keys. My folder to test has 36 JPEG file 3Mpx and my computer has 700Mb. In Editor, witch sidebar tab on the right is toogle on ? Gilles
Am Montag, 24. Juli 2006 15:15 schrieb Gilles Caulier: [bugs.kde.org quoted mail] Only jpegs taken with a Nikon D70. > No, i cannot reproduce this problem using PAGE_UP and PAGE_DOWN keys. My > folder to test has 36 JPEG file 3Mpx and my computer has 700Mb. > In Editor, witch sidebar tab on the right is toogle on ? None of the tabs is open because initially I just wanted to look at the photos. Rainer
idem here. tested without sidebar tab toggled. Very strange... Marcel, can you reproduce this problem on your computer ? Gilles
I can get Linux to swap here. 1) I do not think this is a mysterious memory leak. If it was a memory leak, Linux would free the space after all digikam processes (including kioslaves) have been terminated. Otherwise, Linux would be broken, which I do not believe. 2) Does anyone really know when Linux frees the swap space, or rather, how the used-swap statistic has to be interpreted? May be it simply remains marked as used, although the pages are back in RAM again. 3) Digikam is using a lot of memory. But not so much. Valgrind's "Massif" tells me it is about 160MB at the peak when watching JPEGs in IE, and most of this, about 90%, for simple raw RGB image data. We might be able to cut these 5-8 image instances to by two or three or not, reduce cache space, but in the end we need a lot of memory.
Am Dienstag, 25. Juli 2006 15:54 schrieb Marcel Wiesweg: > 1) I do not think this is a mysterious memory leak. If it was a memory > leak, Linux would free the space after all digikam processes (including > kioslaves) have been terminated. Otherwise, Linux would be broken, which I > do not believe. Well I definitively dare to say its a bug but I don't know if its a digikam or a linux bug. This morning I once again started to run digikam browse photos in IE stopped digikam and did it again. The swap usage summed up more and more even though I restarted digikam and always browsed the same photos. If a cache was responsible this could not be true. Finally after 5 digikam restarts and browsing about 100 photos each time my system (suse10.1, 1GB RAM, 2GB swap) had no more swap and then locked up completely! So there really is a bug somewhere. Between the restarts of digikam I ran cat /proc/meminfo (it is documented in /usr/src/linux/Documentation/filesystems/proc.txt). Here is a edited table with the contents of /proc/meninfo of the 5 runs always taken after I terminated digikam: Initially: after 1. run a. 2. run a. 3. run a. 4. run a. 5. run HighFree: 3652 kB 260 kB 264 kB 260 kB 2316 kB 420 kB LowTotal: 905584 kB 905584 kB 905584 kB 905584 kB 905584 kB 905584 kB LowFree: 235768 kB 614700 kB 387984 kB 524304 kB 494688 kB 364808 kB SwapTotal: 2104472 kB 2104472 kB 2104472 kB 2104472 kB 2104472 kB 2104472 kB SwapFree: 1974168 kB 1392244 kB 1021756 kB 684168 kB 347436 kB 352 kB Dirty: 2256 kB 4 kB 132 kB 468 kB 540 kB 220 kB Writeback: 0 kB 0 kB 0 kB 0 kB 0 kB 0 kB Mapped: 195872 kB 118668 kB 86660 kB 128792 kB 129288 kB 135712 kB Slab: 48280 kB 24640 kB 21304 kB 21396 kB 21840 kB 21988 kB CommitLimit: 2622640 kB 2622640 kB 2622640 kB 2622640 kB 2622640 kB 2622640 kB Committed_AS: 1166192 kB 1742828 kB 2402692 kB 3045924 kB 3575108 kB 4110064 kB PageTables: 2852 kB 5264 kB 3480 kB 4008 kB 4216 kB 4208 kB VmallocTotal: 114680 kB 114680 kB 114680 kB 114680 kB 114680 kB 114680 kB VmallocUsed: 49332 kB 49332 kB 49332 kB 49332 kB 49332 kB 49332 kB VmallocChunk: 56820 kB 56820 kB 56820 kB 56820 kB 56820 kB 56820 kB HugePages_Total: 0 0 0 : 0 : 0 : 0 HugePages_Free: 0 0 0 0 0 0 HugePages_Rsvd: 0 0 0 0 0 0 Hugepagesize: 4096 kB 4096 kB 4096 kB 4096 kB 4096 kB 4096 kB The main thing one can see here is the value of Committed_AS. According to the documentation this is the amount of memory presently allocated on the system. So it seems something is eating memory or linux is loosing memory. The mapped parameter shows that very little space is dedicated to libraries via mmap. In another try I pressed the page down in digikam very fast much faster then the images could be loaded waited a while and did it again and did this sevral times. Suddenly my system began to swap as hell, finally used up all swap space and then locked up again. This happened just after browsing 15 photos or so. In another try I ran top while browsing photos in IE. In top I saw a lot of kio_digikampreview processes beeing spawaned. Finally the system used 1,2 GB swap more that I have RAM (digikam was already terminated). Next I ran ps axuw|awk '{SUM+=$5}; END{print SUM}' summing up the values of the VSZ (Processes Virtual memory Size) for all processes. The result was 3759584 (~3.7 GB) and I could still see several kio_digikampreview processes that had not yet terminated. After a while all those kio_digikampreview processes were gone and I again summed up VSZ. Now the result was 1188952 (1,1GB) but cat /proc/meminfo|grep Committed_AS showed still 2181456 kB (2GB). And since I can *crash* my system repeating the whole procedure with browsing in IE I tend to believe the Committed_AS parameter and not the sum of VSZ. But where does the memory go? Finally I switched my system to runlevel 2 leaving only system processes running. Still Committed_As said that 1.9 GB of memory was used. And still free reported a swap usage of 1GB. Perhaps we need someone with more knowledge of kernel memory management? Anyone out there? Rainer
We forgot about Shared Memory! Could you verify that "ipcs -m" shows a _lot_ of abandoned shared memory segements, all with key 0x0, created by digikam, mostly 4MB?
Am Mittwoch, 26. Juli 2006 18:57 schrieb Marcel Wiesweg: > ------- Additional Comments From marcel.wiesweg gmx de 2006-07-26 18:57 > ------- We forgot about Shared Memory! > Could you verify that "ipcs -m" shows a _lot_ of abandoned shared memory > segements, all with key 0x0, created by digikam, mostly 4MB? Yes I guess thats it. I had 333 of these segments on my machine and after each call of digikam and using IE they are getting more. The memory sum of these segments is close to the amount of memory in swap. Running for i in `ipcs -m|grep 4194304|awk '{print $2}'`; do ipcrm -m $i; done removes all those segments and immedeately frees swap space. So this makes sense. Rainer
SVN commit 567460 by mwiesweg: Kill the preview ioslave each time properly so that is frees its Shared Memory. Note: It seems that the usage of Shared Memory is dangerous, as the segments are not freed when the application crashes. BUG: 131277 CCMAIL: digikam-devel@kde.org M +6 -0 imagepreviewjob.cpp M +19 -5 imagepreviewwidget.cpp --- trunk/extragear/graphics/digikam/digikam/imagepreviewjob.cpp #567459:567460 @@ -173,6 +173,12 @@ stream >> width >> height >> depth; preview = QImage(d->shmaddr, width, height, depth, 0, 0, QImage::IgnoreEndian); + + // The buffer supplied to the QImage constructor above must remain valid + // throughout the lifetime of the object. + // This is not true, the shared memory will be freed or reused. + // If we pass the object around, we must do a deep copy. + preview = preview.copy(); } else { --- trunk/extragear/graphics/digikam/digikam/imagepreviewwidget.cpp #567459:567460 @@ -91,11 +91,11 @@ ImagePreviewWidget::~ImagePreviewWidget() { - if (!d->previewJob.isNull()) + if (d->previewJob) d->previewJob->kill(); - + d->blinkPreviewTimer->stop(); - + delete d; } @@ -107,6 +107,9 @@ d->previewBlink = false; d->blinkPreviewTimer->start(200); + if (d->previewJob) + d->previewJob->kill(); + d->previewJob = new ImagePreviewJob(KURL(path), 1024, AlbumSettings::instance()->getExifRotate()); connect(d->previewJob, SIGNAL(signalImagePreview(const KURL&, const QImage&)), @@ -129,9 +132,16 @@ void ImagePreviewWidget::slotGotImagePreview(const KURL&, const QImage& preview) { d->blinkPreviewTimer->stop(); - d->previewJob = 0; d->preview = preview; d->pixmap = QPixmap(contentsRect().size()); + + // It is very important to kill the thumbnail job properly + // so that is frees its shared memory. Otherwise the memory + // will _never_ be freed, see b.k.o. #131277 + if (d->previewJob) + d->previewJob->kill(); + d->previewJob = 0; + updatePixmap(); repaint(false); emit previewComplete(); @@ -140,7 +150,11 @@ void ImagePreviewWidget::slotFailedImagePreview(const KURL&) { d->blinkPreviewTimer->stop(); - d->previewJob = 0; + + if (d->previewJob) + d->previewJob->kill(); + d->previewJob = 0; + d->preview = QImage(); d->pixmap = QPixmap(contentsRect().size()); updatePixmap();