Bug 226676 - memory leak in nepomukfilewatch
Summary: memory leak in nepomukfilewatch
Status: RESOLVED FIXED
Alias: None
Product: nepomuk
Classification: Miscellaneous
Component: general (show other bugs)
Version: unspecified
Platform: unspecified Linux
: NOR normal
Target Milestone: ---
Assignee: Sebastian Trueg
URL:
Keywords:
: 231409 235377 (view as bug list)
Depends on:
Blocks:
 
Reported: 2010-02-13 14:33 UTC by Philipp Hullmann
Modified: 2012-08-06 10:08 UTC (History)
17 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.7.3


Attachments
massif output (19.06 KB, application/zip)
2012-07-10 16:19 UTC, Jure Repinc
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hullmann 2010-02-13 14:33:53 UTC
Version:           unknown (using 4.4.00 (KDE 4.4.0) "release 222", KDE:KDE4:Factory:Desktop / openSUSE_11.2)
Compiler:          gcc
OS:                Linux (x86_64) release 2.6.31.12-0.1-desktop

During initial indexing of my home directory, nepomukservicestub nepomukfilewatch grows to well over 2GByte of memory, rendering the system unusable due to constant swapping.
Comment 1 Martin Fernau 2010-02-18 11:50:27 UTC
I can confirm that this service is eating a lot of memory during indexing. 
At this time I can't say if this returns to normal if the indexing task is finished.
Comment 2 paul.leopardi 2010-03-15 14:21:19 UTC
Can also confirm:

# ps -e -o args -o vsz | grep nepomuk
kdeinit4: nepomukserver [kd 315304
/usr/bin/nepomukservicestub 408108
/usr/bin/nepomukservicestub 186376
/usr/bin/nepomukservicestub 176132
/usr/bin/nepomukservicestub 690664
/usr/bin/nepomukservicestub 2763740
/usr/bin/nepomukservicestub 218376
/usr/bin/nepomukservicestub 254672
grep nepomuk                  7396
/usr/bin/akonadi_nepomuk_co 222776

See also http://forum.kde.org/viewtopic.php?f=154&t=85457&start=10
Comment 3 paul.leopardi 2010-03-15 14:24:55 UTC
# uname -a
Linux linfinit 2.6.31.12-0.1-desktop #1 SMP PREEMPT 2010-01-27 08:20:11 +0100 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/SuSE-release
openSUSE 11.2 (x86_64)
VERSION = 11.2
# rpm -q -f /usr/bin/nepomukservicestub
kdebase4-runtime-4.4.1-192.1.x86_64
Comment 4 Marvin 2010-03-19 17:56:16 UTC
why is this problem not confirmed? can reproduce this here. everytime strigi is indexing it eats up a lot of memory here and my system has to swap.
Comment 5 Sebastian Trueg 2010-03-19 19:38:57 UTC
This bugs has been confirmed by many people and I can reproduce it, too. I suspect that you mean that you want the bug status to change. So I am doing that.
Comment 6 Jamie Smith 2010-03-28 03:39:55 UTC
I actually seemed to stumble across some interesting interactions and behaviour in running a 'find / sudo' after enabling compcache after every bootup.

Access times seem to be updated and the entire filesystem 'ledger' is stored in compressed ram. What I did notice is reduced ram requirements in clementine as versus Amarok and of course in the filewatch daemon.

Probably this is similar in scope to prelinking on Gentoo as far as feel goes, snake-oil-like. It may expose some workarounds for kernel behaviour and / or in Nepomuk behavior.
Comment 7 Jamie Smith 2010-03-29 08:37:29 UTC
(In reply to comment #6)
> I actually seemed to stumble across some interesting interactions and behaviour
> in running a 'find / sudo' after enabling compcache after every bootup.
> 
> Access times seem to be updated and the entire filesystem 'ledger' is stored in
> compressed ram. What I did notice is reduced ram requirements in clementine as
> versus Amarok and of course in the filewatch daemon.
> 
> Probably this is similar in scope to prelinking on Gentoo as far as feel goes,
> snake-oil-like. It may expose some workarounds for kernel behaviour and / or in
> Nepomuk behavior.

Improvements are to be noticed in shared memory mostly. Amarok isn't privy to shared memory on file handles anyway and so it doesn't show much improvement there..not being inclined to patch or dig any deeper into what calls are at issue, I used Amarok as a hardline reference and played around with Nepomuk / Clementine for a bit to assess file-handle economics. I'm still surprised how well compcache functions as a 'phreak'-like application; I haven't felt like I was hacking as such since my Atari ST days.
Comment 8 Tom Kijas 2010-05-09 15:28:40 UTC
I can confirm this bug does affect me too. 
Linux tom-laptop 2.6.33-020633-generic #020633 SMP Thu Feb 25 10:10:03 UTC 2010 x86_64 GNU/Linux
Kubuntu 10.04
Comment 9 Barton 2010-05-10 19:25:14 UTC
I can confirm memory leak of nepomukservicestub too, often with virtuoso-t. It started after upgrading my system to Lucid Lynx

KDE Platform Version: 4.4.2 (KDE 4.4.2)
Qt Version: 4.6.2
Operating System: Linux 2.6.32-22-generic i686
Distribution: Ubuntu 10.04 LTS
Comment 10 Vangelis 2010-05-25 09:48:38 UTC
Can confirm this bug too....
Comment 11 Jose 2010-06-09 17:37:19 UTC
Confirmed on Kubuntu 10.04 with KDE 4.4.2
Comment 12 bugs-kde 2010-06-10 21:07:36 UTC
Might be useful:

http://thread.gmane.org/gmane.comp.kde.devel.core/63824

A mailing list thread where Sebastian asks for help in diagnosing the memory leak.

Sebastian, did you try massif like suggested over there by Andreas Hartmetz?
Comment 13 bugs-kde 2010-06-10 21:15:40 UTC
BTW, Sebastian, could it be related to the dangling metadata graphs that you wrote about (http://trueg.wordpress.com/2010/02/01/dangling-meta-data-graphs-caution-very-technical/)?

On my machine, the query shows that I have 11717 such graphs:

$ nepomukcmd query 'select count(?mg) where { ?mg nrl:coreGraphMetadataFor ?g . OPTIONAL { graph ?g { ?s ?p ?o . } . } . FILTER(!BOUND(?s)) . }'
callret-0 -> "11717"^^<http://www.w3.org/2001/XMLSchema#int>
Total results: 1
Execution time: 00:00:08.58

Is it a large number?
Comment 14 Jamie Smith 2010-07-21 23:01:26 UTC
http://trac.pcbsd.org/changeset/7147

A BSD changeset related to CPU usage. While it's all accounted for, who can say what whale jumps?

Degressing, how is this issue seen differently on BSD?
Comment 15 Sebastian Trueg 2011-01-06 16:25:55 UTC
It seems the memory leak was magically fixed by some changes unrelated to the search for this bug. Too bad we never figured out the real reason. It is still good to know the leak is gone.
Comment 16 Sebastian Trueg 2011-01-06 19:09:55 UTC
*** Bug 231409 has been marked as a duplicate of this bug. ***
Comment 17 Sebastian Trueg 2011-01-06 20:15:23 UTC
*** Bug 235377 has been marked as a duplicate of this bug. ***
Comment 18 Henning Rogge 2011-10-19 08:03:19 UTC
Unfortunately it seems to be back in KDE 4.7 (or in Kubuntu 11.10 to be more correct).

nepomukservicestub just runs my 4GB ram system at work out of memory, had do disable Nepomuk completely to get the memory back.
Comment 19 Sebastian Trueg 2011-10-24 15:49:17 UTC
Git commit 4f7a5d19da26af282f640c913afccad26000a388 by Sebastian Trueg.
Committed on 24/10/2011 at 17:47.
Pushed by trueg into branch 'KDE/4.7'.

Run the MetaDataMover with an event loop.

It is using the exact same approach as the file indexer does: a new
thread is created and started and the MetaDataMover is then
QObject::moveToThread'ed to it.

This fixes mem leaks caused by DBus events that are not cleaned up.

BUG: 226676
FIXED-IN: 4.7.3

M  +70   -63   nepomuk/services/filewatch/metadatamover.cpp
M  +13   -10   nepomuk/services/filewatch/metadatamover.h
M  +9    -10   nepomuk/services/filewatch/nepomukfilewatch.cpp
M  +3    -1    nepomuk/services/filewatch/nepomukfilewatch.h

http://commits.kde.org/kde-runtime/4f7a5d19da26af282f640c913afccad26000a388
Comment 20 Sebastian Trueg 2011-10-24 15:49:21 UTC
Git commit 90afde5b3a488f89be2349085971655e6497f1bc by Sebastian Trueg.
Committed on 24/10/2011 at 17:00.
Pushed by trueg into branch 'master'.

Run the MetaDataMover with an event loop.

It is using the exact same approach as the file indexer does: a new
thread is created and started and the MetaDataMover is then
QObject::moveToThread'ed to it.

This fixes mem leaks caused by DBus events that are not cleaned up.

BUG: 226676

M  +70   -63   services/filewatch/metadatamover.cpp
M  +13   -10   services/filewatch/metadatamover.h
M  +9    -10   services/filewatch/nepomukfilewatch.cpp
M  +3    -1    services/filewatch/nepomukfilewatch.h

http://commits.kde.org/nepomuk-core/90afde5b3a488f89be2349085971655e6497f1bc
Comment 21 Sebastian Trueg 2011-10-24 18:49:41 UTC
Git commit 0f511184ae25364618ba244f6afda5570b02c388 by Sebastian Trueg.
Committed on 24/10/2011 at 17:47.
Pushed by trueg into branch 'master'.

Run the MetaDataMover with an event loop.

It is using the exact same approach as the file indexer does: a new
thread is created and started and the MetaDataMover is then
QObject::moveToThread'ed to it.

This fixes mem leaks caused by DBus events that are not cleaned up.

BUG: 226676

M  +70   -63   nepomuk/services/filewatch/metadatamover.cpp
M  +13   -10   nepomuk/services/filewatch/metadatamover.h
M  +9    -10   nepomuk/services/filewatch/nepomukfilewatch.cpp
M  +3    -1    nepomuk/services/filewatch/nepomukfilewatch.h

http://commits.kde.org/kde-runtime/0f511184ae25364618ba244f6afda5570b02c388
Comment 22 Jure Repinc 2012-07-10 15:16:27 UTC
Looks like this is back in 4.9 (I'm currently using RC1+ from git here).
Comment 23 Jure Repinc 2012-07-10 16:19:03 UTC
Created attachment 72428 [details]
massif output

This is the massif output after running a few minutes
Comment 24 Dennis Schridde 2012-08-03 00:06:18 UTC
(In reply to comment #22)
> Looks like this is back in 4.9 (I'm currently using RC1+ from git here).
I opened bug #304476 against KDE 4.9.0.
Comment 25 Vishesh Handa 2012-08-06 10:08:12 UTC
(In reply to comment #23)
> Created attachment 72428 [details]
> massif output
> 
> This is the massif output after running a few minutes

Thanks a lot. It helped in diagnosing the problem, and I think I found another way I can reduce the memory footprint by about 10 mb.

Could you please test the patch given over here - https://bugs.kde.org/show_bug.cgi?id=304476