Bug 265737 - Nepomuk backup: excessive RAM usage
Summary: Nepomuk backup: excessive RAM usage
Status: RESOLVED DUPLICATE of bug 265510
Alias: None
Product: nepomuk
Classification: Miscellaneous
Component: general (show other bugs)
Version: unspecified
Platform: Unlisted Binaries Linux
: NOR normal
Target Milestone: ---
Assignee: Sebastian Trueg
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-02-07 23:18 UTC by Dominic Lyons
Modified: 2014-04-25 11:52 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
htop showing nepomukservicestub eating memory (46.25 KB, image/png)
2011-05-04 20:13 UTC, p92
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dominic Lyons 2011-02-07 23:18:09 UTC
Version:           unspecified (using KDE 4.6.0) 
OS:                Linux

At the moment a nepumikservices process eats more than 7 GB RAM. This doesn't happen every day but every few days.

I can't imagine how this can happen as nepomuk is restricted to 256 MB in systemsettings and soprano-virtuoso.db is "only" about nearly 900 MB. 


What is noticeable:

* Internet access was broken for more than half an hour - could that have a (crazy) impact? I guess it at least should not...

* After connection was there again Akonadi neither reconnected GMail IMAP nor Kolab disconnected IMAP - but I guess that shouldn't do Nepomuk any harm

* Today Nepomuk backup was scheduled at 21.00. I didn't recognize if excessive RAM usage just started that time... But else than Bug 265510 ("Nepomuk backup eating too much memory when creating backup file") I don't seem to have a big temp-file.
But all that ps can say it seems to be a problem with the backup (see below)


$ ps -AF | grep nepomuk
dominic   2081     1  0 52815  9432   0 18:41 ?        00:00:00 /usr/bin/nepomukserver
dominic   2086  2081 12 190888 139352 3 18:41 ?        00:32:43 /usr/bin/nepomukservicestub nepomukstorage
dominic   2103  2017  0 74585 22008   3 18:41 ?        00:00:06 /usr/bin/akonadi_nepomuk_calendar_feeder --identifier akonadi_nepomuk_calendar_feeder
dominic   2104  2017  0 73921 22088   2 18:41 ?        00:00:06 /usr/bin/akonadi_nepomuk_contact_feeder --identifier akonadi_nepomuk_contact_feeder
dominic   2105  2017  0 110529 32516  2 18:41 ?        00:00:09 /usr/bin/akonadi_nepomuk_email_feeder --identifier akonadi_nepomuk_email_feeder
dominic   2394  2081  0 91383 21996   0 18:42 ?        00:00:09 /usr/bin/nepomukservicestub nepomukqueryservice
dominic   2395  2081  0 109306 26128  3 18:42 ?        00:00:23 /usr/bin/nepomukservicestub nepomukfilewatch
dominic   2400  2081 12 1911954 7253580 3 18:42 ?      00:32:17 /usr/bin/nepomukservicestub nepomukbackupsync
dominic   2402  2081  0 123711 32952  5 18:42 ?        00:00:08 /usr/bin/nepomukservicestub digikamnepomukservice
dominic   2403  2081  0 141079 51976  5 18:42 ?        00:01:38 /usr/bin/nepomukservicestub nepomukstrigiservice
dominic   2404  2081  0 89432 22536   0 18:42 ?        00:00:08 /usr/bin/nepomukservicestub nepomukremovablestorageservice
dominic   5296  2145  0  2595   852   1 23:07 pts/0    00:00:00 grep --color=auto nepomuk


And that's what ksysguard says:

Process 2400 - nepomukservices

Summary
--------------------------------------------------
The process nepomukservices (with pid 2400) is using approximately 6.9 GB of memory.
It is using 6.9 GB privately, and a further 17.3 MB that is, or could be, shared with other programs.
Dividing up the shared memory between all the processes sharing that memory we get a reduced shared memory usage of 346.0 KB. Adding that to the private usage, we get the above mentioned total memory footprint of 6.9 GB.
4.0 KB is swapped out to disk, probably due to a low amount of available memory left.

Library Usage
--------------------------------------------------
The memory usage of a process is found by adding up the memory usage of each of its libraries, plus the process's own heap, stack and any other mappings, plus the stack of its 2 threads. 

Private
--------------------------------------------------
7233680 KB	[heap]
    284 KB	/usr/lib/libQtGui.so.4.7.0
    204 KB	/usr/lib/libkdeui.so.5.6.0
    132 KB	/usr/lib/libkdecore.so.5.6.0
    116 KB	/usr/lib/kde4/nepomukbackupsync.so

Shared
--------------------------------------------------
2648 KB	/usr/lib/libQtGui.so.4.7.0
1760 KB	/usr/lib/libkdeui.so.5.6.0
1560 KB	/usr/lib/libkdecore.so.5.6.0
1396 KB	/usr/lib/libQtCore.so.4.7.0
 704 KB	/usr/lib/libkio.so.5.6.0

Totals
--------------------------------------------------
Private	7235992 KB	(= 188 KB clean + 7235804 KB dirty)
Shared	17724 KB	(= 17724 KB clean + 0 KB dirty)
Rss	7253716 KB	(= Private + Shared)
Pss	7236338 KB	(= Private + Shared/Number of Processes)
Swap	4 KB


Reproducible: Sometimes

Steps to Reproduce:
That's a really good question...
Comment 1 Dominic Lyons 2011-02-08 00:44:17 UTC
> Steps to Reproduce:
> That's a really good question...

Well, maybe I wrote this a bit too quick...

I've started a manual nepomuk backup. It looked really fine for over an hour (seemed to be slow but low RAM usage) until nearly at the end. In the last few minutes a huge amount of heap memory was reserved (again over 7 GB) and was not given free after the backup was finished.

The backup itself is quite small, about 65 MB compressed and less than 750 MB uncompressed.

So I assume this is a duplicate of Bug 265510 or the other way round... 


The same information as before, but a bit shortened:

$ ps -FA  | grep nepomuk
[...]
dominic   2403  2081  0 141079  932   5 Feb07 ?        00:01:38 /usr/bin/nepomukservicestub nepomukstrigiservice
dominic   2404  2081  0 89432   788   3 Feb07 ?        00:00:08 /usr/bin/nepomukservicestub nepomukremovablestorageservice
dominic   5347  2081 38 1912516 7075316 5 Feb07 ?      00:29:18 /usr/bin/nepomukservicestub nepomukbackupsync
[...]


Process 5347 - nepomukservices

Summary
--------------------------------------------------
The process nepomukservices (with pid 5347) is using approximately 6.7 GB of memory.
It is using 6.7 GB privately, and a further 596.0 KB that is, or could be, shared with other programs.
Dividing up the shared memory between all the processes sharing that memory we get a reduced shared memory usage of 2.0 KB. Adding that to the private usage, we get the above mentioned total memory footprint of 6.7 GB.
159.3 MB is swapped out to disk, probably due to a low amount of available memory left.
[...]

Private
--------------------------------------------------
7074816 KB	[heap]
36 KB	/usr/lib/libQtGui.so.4.7.0
[...]

Totals
--------------------------------------------------
Private	7074936 KB	(= 200 KB clean + 7074736 KB dirty)
Shared  596 KB	(= 596 KB clean + 0 KB dirty)
Rss	7075532 KB	(= Private + Shared)
Pss	7074938 KB	(= Private + Shared/Number of Processes)
Swap	163088 KB
Comment 2 Dominic Lyons 2011-03-07 23:09:46 UTC
Still the same in KDE 4.6.1
Comment 3 Nico Kruber 2011-04-21 09:32:56 UTC
same here, virtuoso-t using 760MB RAM although set not to use more than 256MB

I had a lot of files that strigi had to analyse, but strigi is currently disabled - didn't change the RAM usage though :(
Comment 4 p92 2011-05-04 20:13:10 UTC
Created attachment 59626 [details]
htop showing nepomukservicestub eating memory
Comment 5 YeHaa 2011-05-31 21:51:46 UTC
I'm seeing the same thing here.

The process "/usr/bin/nepomukservicestub nepomukbackupsync" uses 1.7GB private memory. The private heap being 1772416KB.

Ok, I've still got enough memory left, but this is kind of ridiculous for something I really hardly ever use.
Comment 6 Dominic Lyons 2011-06-13 23:38:44 UTC
*** This bug has been confirmed by popular vote. ***
Comment 7 Dominic Lyons 2011-06-13 23:56:23 UTC
Having this problem every monday (this time eating more than 8 GB RAM over a long time)  im pretty sure this bug and #265510 are duplicates.

*** This bug has been marked as a duplicate of bug 265510 ***