Bug 291129 - Okular easily occupies huge memory after scrolling in pdf files
Summary: Okular easily occupies huge memory after scrolling in pdf files
Status: RESOLVED FIXED
Alias: None
Product: okular
Classification: Applications
Component: general (show other bugs)
Version: unspecified
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: Okular developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-01-10 07:56 UTC by Jekyll Wu
Modified: 2015-11-21 02:55 UTC (History)
6 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jekyll Wu 2012-01-10 07:56:01 UTC
Version:           unspecified (using Devel) 
OS:                Linux

After scrolling for about 100 or 200 pages in a pdf file, Okular occupies huge memory(1GB or so) easily.

I know this sounds unlikely to happen, but the problem has bothered me for quite some time. And I can reproduce the problem using fresh account and default settings in multiple distributions, so it really looks like a bug.



Reproducible: Always

Steps to Reproduce:
1. open a pdf file which contains enough pages ( > 200 pages).
2. press 'PgDn' and keep scrolling for about 200 pages
3. check the memory usage of Okular in ksysguard


Actual Results:  
Okular occupies huge memory (1GB or so), mostly on heap.

Expected Results:  
Okular occupies memory in an acceptable way.

I tried evince. No problem. 

I'm using okular built from master branch, poppler-0.18.2 and Qt-4.8.

I'm having this problem in Arch, Gentoo and Fedora 16.

Feel free to ask for extra information that may help.
Comment 1 Albert Astals Cid 2012-01-10 20:20:18 UTC
How mant memory do you have?
Comment 2 Jekyll Wu 2012-01-11 01:11:31 UTC
4GB RAM on my machine.
Comment 3 Albert Astals Cid 2012-01-11 06:59:52 UTC
That is the specified behaviour, stuff is left on cache so it is faster to revisit the pages, if that's not the behaviour you want choose a different Memory Usage setting in the performance configuration tab.
Comment 4 igor 2012-02-25 21:29:42 UTC
I'm sorry but your answer about default behaviour sounds not professional, or may be you don't realize the whole situation. 

My test machine use Arch and kde 4.8. I have a 4GB RAM.

Let's go to the "Okular Settings" to "Performance" tab. 
We see "Memory Usege" set to "Normal (default)", and later we have a discription: 
"A good compromise between memory usege and speed gain. Preload next page and boost searches. (For systems with 256MB of memory, typically.)"

And after scrolling book with 400 pages I heve memory consumption of 1380Mb. 
1380 >> 256 =) Ok. I understand that nobody changes this "help text" since KDE3 or somehow...

BUT the real thing is that opening few documents, lets say 3 or ethen 3 instance of the same book and scrolling till the end will end up with more than 4GB in Total. Not smart at all. 
But of couse ist fast. Very fast. It keep whole book, page by page in memory=)))
Comment 5 Albert Astals Cid 2012-02-26 12:00:10 UTC
Ahh, the old "you are not professional" meme.

Ok, i'm not a professional, i'm just a person that has been working for KDE in my free time since 2003. If you want me to be a professional I will take your money deliver a badly coden software and do it late as real professionals i know do, is this a deal?

And yes, the text has not been updated, writing there a number there was not a good idea.

And yes they might be a bug in how we handle memory.

But without you giving me your exact hardware i know i can't fix it, i've been over that code lots of times and looks correct enough to me

Reopen the bug if you want, the end result will be the same, noone will work on it since caching works for us.

On the other hand if you that are the one having the problem have the skills to code a fix drop by the #okular IRC channel and we'll give you a hand in debugging. That would be professional on your side.
Comment 6 Jekyll Wu 2012-02-26 12:42:17 UTC
Well, I almost forgot this report opened by me.

Albert, I'm fine with the technical explanation and the rationale behind it. I just think it is not quite appropriate to be selected as the default mode. That is mainly about user expectation.

There are some applications that are well known for huge memory occupation: FireFox, VirtualBox, etc. Maybe I would also add acroread from Adobe. I would not be surprised if they make my system slow and would be careful about free memory when starting/using them.  But I just have never expected okular to be one of them. I guess other users might hold similar expectation and would be annoyed when they are hitten by this default mode.  They would then think okular is working in a buggy way and open bug report for it. Or worse, they would think okular is just "so crazy" and stop using it.

So my suggestion is change the existing description so that users understand its impact better, and/or use "low" as the default mode.
Comment 7 igor 2012-02-26 13:55:41 UTC
I didn’t meant that when i talked about "professional".
I think that Jekyll Wu better explained my point of view. 

I checked behavior for two position in okular "default" and "aggressive" and didn't found any difference. In both case it took about 1.4Gb for scrolled 400pages pdf. For the case of "Low memory" okular tooks about 250Mb for same activity. 
And the main thing -- I didn't found big difference between this two regimes. 
At "Low" a see only very short and fast rendering of pages while scrolling.
Comment 8 Albert Astals Cid 2012-02-26 21:42:39 UTC
As said, you guys are free to debug the memory caching and total/free memory calculation code in document.cpp and find any bugs there (i'm of course not saying that code is bug free).

Okular uses memory if there unused one (why would you prefer memory doing nothing?) and frees memory if it is using more memory than the one that is actually free (thus we do not exhaust memory or monopolize it if someone starts to need it)
Comment 9 igor 2012-02-26 21:52:55 UTC
As I said before, I started 3 times exactly the same file, scroll over 400pages, and each gets about 1400mb, so ALL of my 4GB was used by 3 instance of okular.

When as you say "and frees memory if it is using more memory than the one that is
actually free". If any new application is going to be run, first of all it should ask okular to free some memory, or even go swapping to be able to run. 
Sorry, here is only theory, I didn't try to fill all of my memory with okular and after it trying to run anything else. 

By the way, I didn't see such problem in okular 4.4.5 and in 4.6.5. May be in 4.7.2 where was also no problem (not sure).
Comment 10 Albert Astals Cid 2012-02-26 22:03:44 UTC
Yes, i read what you wrote, did you read what I wrote?

* There might be bugs, obviously
* If they are bugs i can't fix them since it works for me, so you are on your own
Comment 11 igor 2012-02-26 22:40:36 UTC
Can you at least check this idea:
old "Low" is missing and not possible to choose
old "Normal" is now "Low"
old "Aggressive" is now "Normal"
and for new "Aggressive" is again "Aggressive"

As I see no difference in behavior between "Aggressive" and "Normal", 
and you don't gonna look for bugs, may be at least just remove third choice and change text description.
Comment 12 Albert Astals Cid 2012-02-26 22:57:07 UTC
The code hasn't changed for years.
Comment 13 igor 2012-02-26 23:16:27 UTC
I didn't know that.
In this case I see only two possible reasons: Qt or Distribution.
Comment 14 Benni Hill 2012-06-19 20:21:21 UTC
Hi,

I think I found a bug in DocumentPrivate::getFreeMemory().
I was programming on Gwenview which has a copied version of this function in its code.

When writing the output of getFreeMemory() to stdout via qDebug() I sometimes get values like 18446744073694777344 which suggest that there is an integer underflow. Here is an excerpt from cat /proc/meminfo shortly after that happened:
MemTotal:        2824516 kB
MemFree:          277580 kB
Buffers:           10292 kB
Cached:           169928 kB
SwapCached:        32984 kB
Active:          1733572 kB
Inactive:         546212 kB
Active(anon):    1663072 kB
Inactive(anon):   468348 kB
Active(file):      70500 kB
Inactive(file):    77864 kB
Unevictable:       11316 kB
Mlocked:           11316 kB
SwapTotal:       7814140 kB
SwapFree:        7334064 kB

When I understand the function properly it is calculation the free memory as:
MemFree + Buffers + Cached + SwapFree - SwapTotal
which in this case would result in a negative value of -22 276 kB which can't be returned as qulonglong.
Comment 15 Albert Astals Cid 2012-06-19 22:37:48 UTC
Git commit 444e6b7b19bee5285f39a44007cf00f6e0b235d2 by Albert Astals Cid.
Committed on 20/06/2012 at 00:33.
Pushed by aacid into branch 'master'.

Fix underflow in getFreeMemory()

It actually serves three purposes:
a) Make sure all the values are there (this should be always true, but doesn't hurt making sure) because if SwapFree was there but SwapTotal was not, it'd be a mess
b) add up things in order so we don't underflow, currently the code did process stuff as it came in the file, and it happens that SwapTotal appears before SwapFree in /proc/meminfo so it actually did  "MemFree:" + "Buffers:" + "Cached:" - "SwapTotal:" + "SwapFree:", which can underflow if "MemFree:" + "Buffers:" + "Cached:" < "SwapTotal:"
c) Do not underflow at all, so if  "MemFree:" + "Buffers:" + "Cached:" +  "SwapFree:" < "SwapTotal:" we return 0 correctly not a zillion of free memory

Aurélien you might want to update gwenviews copy of this code (there's a few other bugfixes we did a while ago you didn't update either)

CCMAIL: agateau@kde.org

M  +22   -7    core/document.cpp

http://commits.kde.org/okular/444e6b7b19bee5285f39a44007cf00f6e0b235d2
Comment 16 Albert Astals Cid 2012-06-19 22:38:28 UTC
Git commit 5fb937a46aaf4e531ada816692b1e47fc95b20b0 by Albert Astals Cid.
Committed on 20/06/2012 at 00:33.
Pushed by aacid into branch 'KDE/4.8'.

Fix underflow in getFreeMemory()

It actually serves three purposes:
a) Make sure all the values are there (this should be always true, but doesn't hurt making sure) because if SwapFree was there but SwapTotal was not, it'd be a mess
b) add up things in order so we don't underflow, currently the code did process stuff as it came in the file, and it happens that SwapTotal appears before SwapFree in /proc/meminfo so it actually did  "MemFree:" + "Buffers:" + "Cached:" - "SwapTotal:" + "SwapFree:", which can underflow if "MemFree:" + "Buffers:" + "Cached:" < "SwapTotal:"
c) Do not underflow at all, so if  "MemFree:" + "Buffers:" + "Cached:" +  "SwapFree:" < "SwapTotal:" we return 0 correctly not a zillion of free memory

Aurélien you might want to update gwenviews copy of this code (there's a few other bugfixes we did a while ago you didn't update either)

CCMAIL: agateau@kde.org
(cherry picked from commit 444e6b7b19bee5285f39a44007cf00f6e0b235d2)

M  +22   -7    core/document.cpp

http://commits.kde.org/okular/5fb937a46aaf4e531ada816692b1e47fc95b20b0
Comment 17 Benni Hill 2012-06-20 19:14:43 UTC
(In reply to comment #16)

Albert,
I didn't understand why swapped memory is considered for free memory calculation in Linux. After having a look at the FreeBSD and Windows part of getFreeMemory(), which are just returning the free physical memory as it seems, I'm totally confused. Could you explain this briefly, please?
Comment 18 Albert Astals Cid 2012-06-23 11:00:01 UTC
The swapped memory is "substracted" from the free memory because we don't want to prevent swapping in of any program that might be swapped out to be slow or to end up putting okular stuff in the swap.

But to be honest i did not write that code and i'm not sure if that's the right thing to do, all I just did was to fix the technical problems.

And as always everyone says you should not do memry managemtn in the program by yourself so for a lot of people "we are doing it wrong", if you have a much better suggestion and a real good rationale behind it i'll be happy to hear it
Comment 19 Benni Hill 2012-06-26 00:32:39 UTC
(In reply to comment #18)
> The swapped memory is "substracted" from the free memory because we don't
> want to prevent swapping in of any program that might be swapped out to be
> slow or to end up putting okular stuff in the swap.

I understand, but in this case SwapCached should also be considered then. According to [1] SwapCached is "Memory that once was swapped out, is swapped back in but still also is in the swapfile (if memory is needed it doesn't need to be swapped out AGAIN because it is already in the swapfile. This saves I/O)". So SwapCached could be added to memoryFree because it's already in physical memory.

But I would prefer to change the behavior of getFreeMemory:
Instead of a program that is swapping in, the memory could also been taken away by a program that is just starting up. To detect this okular is already polling the memory usage every 2 s (if I get this right). So why can't we detect programs that are swapped in, this way? I would propose to just poll the physical memory usage (perhaps faster) and check if (freePhysicalMemory < fraction*totalPhysicalMemory) with fraction ~0.5 or so. If thats the case then we free some memory.

And finally: Shouldn't getFreeMemory behave consistently on all platforms? (BTW: Is Mac OS X missing?)


Another thing related to memory usage:
In DocumentPrivate::cleanupPixmapMemory when set to Greedy:
memoryLimit = qMin( qMax( freeMemory, getTotalMemory()/2 ), freeMemory+freeSwap );

Let's assume freeSwap = 10 GB, freeMemory = 1 GB and totalMemory = 8 GB, then memoryLimit would be totalMemory/2 (= 4 GB) and would result in swapping? Or do I understand this wrong?



[1]: https://lwn.net/Articles/28345/
Comment 20 Benni Hill 2012-06-26 00:43:40 UTC
(In reply to comment #18)
> And as always everyone says you should not do memry managemtn in the program
> by yourself so for a lot of people "we are doing it wrong",

I don't know how to do it right. Don't use memory? ;-)
Comment 21 Albert Astals Cid 2012-06-27 16:52:12 UTC
If you know how to implement the MacOSX version we'll take the patch ;-)

About the greedy stuff, Yes, it would result in us kicking out of the memory some oher program to the swap and using the memory, that's why it's called greedy ;-)
Comment 22 Benni Hill 2012-06-27 17:39:47 UTC
OK, but then there should be a big warning like "might make your system slow and unresponsive".
And in doc/index.docbook this should also be less misunderstanding:
"Use Greedy profile to preload all pages without risk of system memory overfull..."
Comment 23 Albert Astals Cid 2012-06-27 17:42:55 UTC
You are deviating the topic :D

If you feel the doc is wrong, please open a bug about that, and no, i don't think there is a need for a big warning about greedy.
Comment 24 Massimiliano 2013-06-30 13:58:57 UTC
I had the same issue: default "normal" setting used to be too greedy and sometimes freezed my system. This could be a little annoying for less skilled users who typically don't proofread program settings before using them.

Anyway, setting the radio button to "minimal" does the trick for me.
Comment 25 zellimzel 2015-11-21 02:55:54 UTC
*** Removed by KDE Sysadmin due to violation of abuse policies ***