Bug 368838 - plasmashell memory leak when slideshow is used for wallpaper/media frame/photo widget with QSG_RENDER_LOOP=basic
Summary: plasmashell memory leak when slideshow is used for wallpaper/media frame/phot...
Status: RESOLVED UPSTREAM
Alias: None
Product: plasmashell
Classification: Plasma
Component: general (show other bugs)
Version: 5.9.3
Platform: Arch Linux Linux
: NOR major
Target Milestone: 1.0
Assignee: David Edmundson
URL: https://codereview.qt-project.org/#/c...
Keywords: usability
: 377795 388580 390238 391063 391377 392174 392181 392505 392524 392568 392599 392668 393248 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-09-15 09:45 UTC by Antonio Orefice
Modified: 2018-12-31 17:35 UTC (History)
52 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Memory informations (489.47 KB, application/octet-stream)
2016-09-15 09:47 UTC, Antonio Orefice
Details
plasmoidviewer: memory maps, memory status and "valgrind --tool=memcheck --leak-check=full" log (47.04 KB, application/gzip)
2016-11-25 14:04 UTC, Antonio Orefice
Details
log with debug symbols enabled for qt5-declarative and qt5-base (17.57 KB, application/gzip)
2016-11-25 14:39 UTC, Antonio Orefice
Details
time-series chart demonstrating plasmashell memory leak (61.27 KB, image/png)
2018-02-16 21:21 UTC, Matt Whitlock
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Antonio Orefice 2016-09-15 09:45:55 UTC
On an always on system, my plasmashell memory usage grows and grows.
Last time i checked before restarting it, it was eating almost 900MB of ram.

Reproducible: Always

Steps to Reproduce:
1. measure the memory usage of plasmashell just after startup
2. use it (open apps, display notifications, move widgets, show menu, clock, calendar)
3. leave it as it was in the fresh state (close apps, close notifications restore panel position etc)
3. measure the memory usage again
4. leave it alone for some time (30 minutes?)
5. check the ram usage again

Actual Results:  
The ram usage increased a lot.


Expected Results:  
The ram usage to be almost back to the startup state.

I attached pmap,ksysguard and /proc/$(pidof plasmashell)/status  before and after usage.
Where usage has been:
Show the 'kickoff menu' and walk through applications
send notifications
show notifications
show calendar
hover the mouse on the taskbar items
right click on the desktop to show his configuration window
right click on the desktop to show the activities window
move a desktop widget
resize a desktop widget to the entire screen size
click on the panel configuration icon
show the available widgets
--
Revert all of the changes so that apparently it is a newly started session

Before all the described operations, Resident+shared mem was little less than 300MB
After, it reached more than  630MB (!)
...and it stays that way (or still grows) till a restart.
Comment 1 Antonio Orefice 2016-09-15 09:47:18 UTC
Created attachment 101094 [details]
Memory informations
Comment 2 Antonio Orefice 2016-09-20 14:43:31 UTC
I just played with nuoveau, and found that it consumes so much less memory.
With a quick test (show kickoff menu, clock, panel configuration window, moving a plasmoid):
nvidia blob reached 400MB
nuoveau stayed at about 250MB

tried nvidia 367.35 and 370.28
Comment 3 Antonio Orefice 2016-09-20 14:47:45 UTC
I forgot to add that running a compositor or not running a compositor makes 0 difference
Comment 4 David Edmundson 2016-09-20 16:28:11 UTC
What version of plasma-framework?
Comment 5 Antonio Orefice 2016-09-21 07:15:07 UTC
First time i tried was with framework 5.25. It persists with 5.26
qt5 is 5.7.0
Comment 6 Antonio Orefice 2016-09-22 13:52:48 UTC
@David Edmundson :
Are you able to reproduce with Nvidia proprietary drivers?
Comment 7 Antonio Orefice 2016-09-22 14:06:24 UTC
New info:
with raster backend, apart from having a broken desktop, it seems plasma still allows me to interact with panels, so i interacted with them by using the menu, popup some tooltips, showed the calendar and so on.
Well, with export QMLSCENE_DEVICE=softwarecontext i've a ram usage that starts with 228MB and ends to 276MB
Using classic opengl, doing the exact same things, it starts at 228MB and reaches 420MB.
Comment 8 Antonio Orefice 2016-09-22 15:06:51 UTC
Since it seems nouveau is not affected, I reported the very same to nvidia
https://devtalk.nvidia.com/default/topic/966824/linux/huge-memory-allocation-on-plasmashell-with-proprietary-driver-nouveau-is-fine/post/4981667/?offset=5#4981701
In there, at posted a script using xdotool that did nothing else that showing the plasma configuration window; then the get wallpapers window, close both and reiterate.
After 20 minutes of that, my system started swapping badly and plasma was eating almost 6GB of ram.
Comment 9 David Edmundson 2016-09-22 15:14:54 UTC
Oh, then it's not the issue that I fixed in 5.26 which was purely related to thumbnails.

The wallpaper dialog doesn't do any custom GL, so if that's leaking only on NVidia it strongly implies the problem is higher in the stack than Plasma.
Comment 10 David Edmundson 2016-09-22 15:15:35 UTC
please also try with 

QSG_RENDER_LOOP=basic plasmashell

and report on what that does
Comment 11 Antonio Orefice 2016-09-22 15:59:37 UTC
I made a 3 minute run of that xdotool script (open and close wallpaper settings multiple times).
Nvidia proprietary driver + QSG_RENDER_LOOP=basic
Start: 286MB
End: 486MB
Nvidia proprietary driver + unset QSG_RENDER_LOOP
Start: 281MB
End: 533MB
Nouveau + unset QSG_RENDER_LOOP
Start 289MB
End 436MB

So, in the end it seems that NVIDIA blob is using more ram, but not that much, and plasma/qt are leaking more memory just because qt uses threads with nvidia proprietary driver, right?
Comment 12 Antonio Orefice 2016-11-16 12:24:42 UTC
For what is worth, the bug is present even in plasmoidviewer.
Just play with it and see how his memory usage grows with ease.
Comment 13 Antonio Orefice 2016-11-25 14:04:29 UTC
Created attachment 102436 [details]
plasmoidviewer: memory maps, memory status and "valgrind --tool=memcheck --leak-check=full" log

As i found that plasmoidviewer leaks memory too, i was able to make a valgrind log out of it (too slow for the whole plasmashell).
See that VmRSS jumped from 392504kB to 453376kB, all i did was moving, rotate and resize it.
Comment 14 Antonio Orefice 2016-11-25 14:39:29 UTC
Created attachment 102437 [details]
log with debug symbols enabled for qt5-declarative and qt5-base

Same as before, but with debug symbols for (what i think) relevant libraries.
Comment 15 Rob Boudreau 2016-11-27 16:14:04 UTC
I can confirm the memory problem, though whether it's a leak or just memory not being released I don't know. Plasmashell doesn't seem to grow overnight or when the machine is idle, but any interaction with the desktop will increase plasmashell's resident memory. At start it's usually 173Mb, then depending on what I'm doing, will increase in 4-5Mb chunks or more. After just a few hours it can be as much as 400-500Mb. It will continue to grow if I let it, I've had it over a gig before, but now I usually kill plasmashell and restart it once or twice a day.

System specs:
    KDE Plasma Version: 5.8.4
KDE Frameworks Version: 5.28.0
            Qt Version: 5.7.0
        Kernel Version: 4.8.10-1-ARCH

            Processors: 4xIntel Core i5-3470 CPU @ 3.20GHz
                Memory: 8 GiB
                 Video: NVIDIA GF108 [GeForce GT 630]
                Driver: NVIDIA 375.20

The system is Arch with all the latest updates from the stable repos. I've tried the 'QSG_RENDER_LOOP=basic' with no effect. Tried changing rendering backends (OpenGL 3.1, 2.0 and xrender), switching to the nouveau driver, no noticeable difference. Kinfo reports almost all the used memory is held in the heap.

Any other information I can add that may be helpful in tracking this down, just ask.
Comment 16 Antonio Orefice 2017-03-08 12:31:58 UTC
from 290MB to 930MB in 5 minutes:
https://www.youtube.com/edit?o=U&video_id=K3nbY0xNykA
Comment 17 Antonio Orefice 2017-03-08 13:53:39 UTC
Sorry, wrong link:
https://www.youtube.com/watch?v=K3nbY0xNykA
Comment 18 draffin 2017-07-30 12:39:22 UTC
I have the same problem : the plasmashell's memory keeps growing until the system crashes. It usually takes 2 or 3 hours to fill my 8GO of RAM. I have to log out frequently to avoid the problem.

System specs:
    KDE Plasma Version: 5.10.4
KDE Frameworks Version: 5.36.0
            Qt Version: 5.9.1
        Kernel Version: 4.12.3-1-ARCH

            Processors: 4xIntel Core i3-3120M CPU @ 2.50GHz
                Memory: 7.7 GiB
Comment 19 David Edmundson 2017-07-30 22:36:13 UTC

*** This bug has been marked as a duplicate of bug 381000 ***
Comment 20 Antonio Orefice 2017-07-31 06:58:36 UTC
I'm sorry, but how can this be a duplicate of bug 381000?

* My wallpaper is static, i've just one image
* I reported this bug on 2016-09-15, while 381000 is from 2017-06-09 and seems to be caused by a commit done *after* i reported this.

(What is wrong is comment 18, which probably is referring to 381000.)

If the source cause is the same, i'd be happy to be wrong!
Comment 21 draffin 2017-07-31 08:22:43 UTC
I have also a static background but ... I had a photo widget on my desktop, configured as ... slideshow ! I've shut it and it works fine.
Comment 22 Antonio Orefice 2017-07-31 12:17:22 UTC
Yes, by static i mean not slideshow :P
Comment 23 electr0nux 2017-12-01 04:02:36 UTC
I`m on plasma 5.11.1 and when I open "plasma discover" memory usage goes from 450MB to 7.8GB/8GB, eventually filling up my swap too rendering my system non-responsive.
RAM usage keeps on increasing until I find some way to close the plasma discover.
Comment 24 Nate Graham 2018-01-05 23:03:42 UTC
*** Bug 388580 has been marked as a duplicate of this bug. ***
Comment 25 Nate Graham 2018-01-05 23:06:46 UTC
Marking as Major since slideshow options are pretty popular and present a large surface area for user dissatisfaction when they eat up too much RAM and bog down the system.
Comment 26 Matthew Woehlke 2018-01-05 23:44:50 UTC
I'm not sure if Bug 388580 is a duplicate or not; I have only started noticing problems recently, and I'm using the radeon driver.

(Random data point: immediately after start-up, plasmashell was 4G/400M (virt/res), and is up to 6G/2G after about 8 hours.)
Comment 27 Fabian Kislat 2018-01-10 21:53:11 UTC
I have the same problem using the following versions:

KDE Plasma: 5.11.4
KDE Frameworks: 5.41.0
Qt: 5.9.3
Kernel: 4.12.12-gentoo

I'm using the nouveau driver, so I doubt it is driver related.

In order to test if this is background image related, I watched the memory used by plasmashell while cycling through background images using the context menu option. Every time I change the image I can see a large increase in RSS.

For me this issue is fairly new. I can't pin it down exactly, but I would think it must be related to my last update, which included the updates to frameworks 5.41 from 5.40 and Qt 5.9.3 from 5.7.x (not sure what exact version I used previously).
Comment 28 Sebastian 2018-01-16 06:31:55 UTC
I can still confirm that this Bug exsists on actual KDE Neon. Changing to slideshow increases CPU-Usage immediatley, going back to single image und restart plasmashell solves the problem.

The same happends on Archlinux.
Comment 29 Luis 2018-01-17 08:05:07 UTC
It happens on Kubuntu 17.10 . I see plasmasheel eating all RAM of my system (16GiB or total RAM).
Comment 30 Luis 2018-01-17 11:24:46 UTC
Also, I noticed that screenlock could hang if is using slideshow wallpaper
Comment 31 Matt Whitlock 2018-01-22 16:29:14 UTC
I am experiencing this bug on a single monitor with the AMDGPU driver, Qt 5.9.3, and Plasma 5.11.4, running on Linux 4.14.13-gentoo. This bug has been affecting me for a few months now, plausibly since I upgraded to the Qt 5.9.x series from 5.7.1 at the end of September 2017.

I have the slideshow wallpaper set to change every 29 minutes and 1 second, and I can get 1-2 weeks of use out of my desktop session until I run out of available memory and my system starts thrashing and I have to take steps (see below) to recover and continue. Memory usage seems to grow even while I am away from my computer (i.e., while it's sitting at the KDE Screen Locker with the monitor powered down).

Of particular interest on my system: the majority of the memory usage isn't attributed to the plasmashell process, as reported by 'ps' and 'top'. Bizarrely, the memory usage appears to be accountable to *no process*. Because of this, it took me a long time to track it down to Plasma. I even reported a kernel bug[1], as the memory seemed to be inexplicably in use by the kernel itself: adding up the memory counters in /proc/*/status across all running processes yields a total memory usage that does not account for the complete exhaustion of MemAvailable in /proc/meminfo.

How I know plasmashell is the culprit:

1. My system started to become sluggish, which happens as the kernel starts evicting pages of executable code from RAM and having to load them back in from disk. (If not immediately addressed, this condition worsens until I have no choice but to invoke the OOM killer using Magic SysRq to reclaim a little bit of memory so I can proceed with the next steps.)
2. I switched to a text-mode VT and ran 'echo 2 > /proc/sys/vm/drop_caches' as root. This elicited a short pause before the prompt reappeared, and then 'free' reported a small amount of additional available memory.
3. I killed (by SIGTERM) my plasmashell process. Some memory was immediately freed (as expected), but about 3 GB was still unaccounted for. That is, 'free' reported much more memory as "used" than the sum of all processes in my system.
4. I ran 'echo 2 > /proc/sys/vm/drop_caches' again. This time the prompt took noticeably longer to reappear.
5. Now 'free' reported a sane number for "used", which was fully attributable to the set of remaining processes running on my system.

My usual procedure for recovering from the low-memory scenario is to log out of my KDE session entirely and then do the drop_caches trick to get the kernel to release the apparently unaccounted-for memory. But this time I left my session running and only killed plasmashell. That was enough to unstick the memory, and that's how I found my way to this bug report.

I am a low-level C/C++ developer, but I have very little experience with OpenGL and no experience with Qt. My current (only somewhat informed) working theory is that the plasmashell process is allocating wallpapers in video memory but never freeing them and that this memory is not accounted to the process in the usual ways, as it is nominally not system RAM. However, because VRAM is limited, eventually the kernel shifts the disused wallpapers out of VRAM and into system RAM, where they sit, not accounted to any process. I do not understand why killing the plasmashell process does not immediately cause the kernel to free this memory, though. I have to tell the kernel to drop its inode+dentry caches to get the memory back.

As I only get into this situation every week or two and have to address the situation immediately when it occurs, it's hard for me to try things and report back quickly, but I would gladly welcome any suggestions and will supply any information requested. Since I run Gentoo, I am able to patch sources and rebuild packages easily.

At the moment, I am going to try disabling the slideshow wallpaper and see if plasmashell no longer causes this unaccountable memory usage. I will report back in a few weeks with my findings.

__________

[1] https://bugzilla.kernel.org/show_bug.cgi?id=197783 - Incidentally, the rogue memory pages are swappable.
Comment 32 Nate Graham 2018-01-22 21:27:25 UTC
Thanks a lot for the investigation and analysis, Matt! If you do happen to pinpoint the issue, you're welcome to submit a patch using http://phabricator.kde.org/. More information is available at https://community.kde.org/Get_Involved/development.

If you need a hand with anything, don't hesitate to let me know.
Comment 33 Diego Alcántara 2018-01-23 12:23:17 UTC
I'm also affected by this bug in the PC at home, but not at work, both updated to plasmashell 5.11.5, kde 5.41.0, Kubuntu 17.10
This weekend I experimented a bit at home: when I changed the second monitor desktop from "Slideshow" (20 secs between image change) to "Hunyango" the bug stopped. When I restored back to "Slideshow" the bug reappered.

"free" shows that a lot of memory is used in "shared"
Comment 34 David Edmundson 2018-02-16 10:22:19 UTC
So this whole thing has got quite messy.

There was this bug:
https://bugreports.qt.io/browse/QTBUG-61754

This as reported I'm 99% sure is fixed.

We've also got a bug about high CPU, to do with stopped animation drivers not getting removed.


If you're running < Qt5.10 or < 5.9.3 upgrade now so you have them and don't confuse this report.


Please also report if you're using the basic or threaded render loop. You can find out by running QSG_INFO=1 plasmashell and reading the output.

Something like: "10:19:03.848 qt.scenegraph.general: QSGRenderLoop::instance threaded render"

narrow that down and it'll help.

Also please download the sample QML file in https://bugreports.qt.io/browse/QTBUG-61754 and report on whether that gives you issues.
Comment 35 Antonio Orefice 2018-02-16 11:54:04 UTC
I repeat, the bug i reported (this one) was not about wallpaper/images photo widgets and so on.
if the underlying cause is the same, i don't know.
Comment 36 Matt Whitlock 2018-02-16 21:21:45 UTC
Created attachment 110735 [details]
time-series chart demonstrating plasmashell memory leak

Do we need to open a new bug report for the slideshow wallpaper issue? I am running Qt 5.9.4 and Plasma 5.12.1 now, and it most definitely is still an issue.

Please see the attached chart, which visualizes the output of...

 • free | grep '^Mem:'
 • grep '^VmRSS:' /proc/"$(pgrep plasmashell)"/status

...with samples taken at 1-second intervals while manipulating plasmashell.

Particularly noteworthy is that the actual plasmashell process's VmRSS hardly changed the whole time. The system is evidently not accounting the memory usage to the plasmashell process. Nevertheless, given enough time, plasmashell brings the system to its knees. (I actually had to use AltGr+SysRq+F to manually invoke the OOM killer just so I could terminate plasmashell, as my session had become unresponsive.)

I would reasonably expect to see the gradually increasing "Buff/Cache" line and the gradually decreasing "Free" line, as the block cache should be caching the JPEG files as they are loaded from disk. (This is no problem, although frankly plasmashell ought to be calling posix_fadvise(2) with POSIX_FADV_DONTNEED on these files.)

What I would NOT expect to see is the "Used" line increasing and the "Available" line decreasing. This suggests that memory is being consumed by something other than the block cache.

My best guess is that plasmashell is not properly disposing of its textures. It takes a while for them to fill up VRAM, but then they start spilling into system RAM. I could test this hypothesis if I knew how to view how much VRAM is used/free.
Comment 37 David Edmundson 2018-02-16 21:23:50 UTC
Please see my requests on #34
Comment 38 Matt Whitlock 2018-02-16 21:47:50 UTC
(In reply to David Edmundson from comment #37)
> Please see my requests on #34

(In reply to David Edmundson from comment #34)
> If you're running < Qt5.10 or < 5.9.3 upgrade now so you have them and don't
> confuse this report.

(Quoting myself from comment #36)
> I am running Qt 5.9.4 and Plasma 5.12.1 now

> Please also report if you're using the basic or threaded render loop. You
> can find out by running QSG_INFO=1 plasmashell and reading the output.

qt.scenegraph.general: QSG: basic render loop
qt.scenegraph.general: Using sg animation driver
qt.scenegraph.general: texture atlas dimensions: 4096x2048
qt.scenegraph.general: R/G/B/A Buffers:    8 8 8 8
qt.scenegraph.general: Depth Buffer:       24
qt.scenegraph.general: Stencil Buffer:     8
qt.scenegraph.general: Samples:            -1
qt.scenegraph.general: GL_VENDOR:          X.Org
qt.scenegraph.general: GL_RENDERER:        AMD Radeon (TM) R7 300 Series (BONAIRE / DRM 3.23.0 / 4.15.3-gentoo, LLVM 5.0.1)
qt.scenegraph.general: GL_VERSION:         3.0 Mesa 17.3.3
qt.scenegraph.general: GL_EXTENSIONS:      *(snip)*
qt.scenegraph.general: Max Texture Size:  16384
qt.scenegraph.general: Debug context:     false

> Also please download the sample QML file in
> https://bugreports.qt.io/browse/QTBUG-61754 and report on whether that gives
> you issues.

I observed no issues when I ran the sample QML file in qmlscene, but those sample images are tiny and would take a very long time to fill up VRAM, so it may not be a useful test.
Comment 39 Antonio Orefice 2018-02-19 12:34:58 UTC
Me, the OP,
reported the issue on QT 5.7.x times, so the bug here: https://bugreports.qt.io/browse/QTBUG-61754 does not apply at all.

Is there any chance to bring the title of this back to the original, the 'thread' has derailed...
It originally was: "[plasmashell] [Bug 368838] plasmashell keeps eating ram (NVIDIA BLOB), but never release it."
The "NVIDIA BLOB" part is irrelevant.
Comment 40 Matt Whitlock 2018-02-19 17:04:14 UTC
Sorry your bug report has been "derailed," Antonio, though it's very possible that the symptoms we're each observing may have the same root cause.

I just installed radeontop and watched my card's VRAM usage climb rapidly while running the slideshow wallpaper in plasmashell. Stopping the slideshow did not release the VRAM. Killing plasmashell did.
Comment 41 TYY331 2018-02-20 05:16:44 UTC
The memory leak also occurs while displaying application's icons.
Displaying application's icons in the launcher constantly increases plasma's memory usage.
Steps to reproduce.
1. Display application's icon (application menu, application launcher and application dashboard trigger the memory leak)

Bear in mind since icons are low size files the memory leak per icon is quite low, the problem becomes evident while displaying a lot of icons over time, that's why you have to rapidly navigate the menus due to time constrains, normal use would take a couple of days.

The problem is that there is no workaround since application's icons are always present and can't be disabled.
Comment 42 Nate Graham 2018-02-21 05:18:22 UTC
That Qt bug was reportedly fixed in 5.9.2. If that's the cause, then anyone using a later Qt version should be fine.

Is there anyone using Qt 5.9.2 or beyond who can reproduce this issue?
Comment 43 Matt Whitlock 2018-02-21 05:31:24 UTC
(In reply to Nate Graham from comment #42)
> Is there anyone using Qt 5.9.2 or beyond who can reproduce this issue?

As I mentioned in comment #36 and comment #38, I am running Qt 5.9.4, and I can still easily exhaust all system memory by running a slideshow wallpaper in plasmashell.

I don't know about the application icons thing. I do recall that I used to see the plasmashell process taking hundreds of megabytes of RAM and sometimes even over a gigabyte. I haven't observed that in a while. Nowadays its major memory leak appears to be in VRAM.
Comment 44 gertlink_nospam 2018-02-21 06:25:56 UTC
My current system is QT 5.10.0 and Plasma 5.12.1.
The "memory leak" error is still present, since version QT 5.9.1

If I activate the slideshow in the picture frame applet, I miss about 150 MB of memory after every picture change, which is no longer released.
Comment 45 Antonio Orefice 2018-02-21 07:48:40 UTC
qt5 5.10.0 here.
Memory increases over time as described in the first post, nothing changed.
Comment 46 Matthew Woehlke 2018-02-23 15:30:21 UTC
I am also still seeing the issue with Qt 5.9.4, plasma 5.11.5 (full list of my current packages posted to Bug 388580 so as not to clutter this report).
Comment 47 Nate Graham 2018-02-24 13:47:23 UTC
*** Bug 387341 has been marked as a duplicate of this bug. ***
Comment 48 Erin Yuki Schlarb 2018-02-26 12:26:33 UTC
Moving discussion here from my bug report at 388580…

Summary of my findings form that other thread (Radeon R9 M280X, Linux 5.9 – 5.15, Mesa any version, Qt 5.9.3):

  1. The issue doesn't occur with the "AMD GPU Pro" userland GL stack (in return you get s***loads of other issues) – only MESA (including ancient versions) is affected
  2. Memory usage of the offending process will be accounted correctly with running it with `LIBGL_ALWAYS_SOFTWARE=true`, but the leaking issue will persist
  3. The leak can be greatly accelerated by enabling the "Media-Frame" Plasmoid
  4. I am absolutely not able to compile a working Qt + KDE stack to bisect this issue
  5. (Setting `QSL_RENDER_LOOP=threaded` seems to make the issue go away ← Still investigating this currently!)


@Matt Whitlock

> Have you tried running radeontop?

`radeontop` doesn't work on my system (Debian unstable with current 4.15.0 kernel):

    amdgpu DRM driver is used, but amdgpu VRAM size reporting is not enabled
    amdgpu DRM driver is used, but amdgpu VRAM usage reporting is not enabled
    Failed to get VRAM usage, kernel likely too old

Do you think there are any insights to be gained by compiling a custom kernel for this?



One other thing I noticed (not sure if its related to this issue) is that when enabling DisplayCore (DC) for the `amdgpu` driver (kernel flag `amdgpu.dc=1`), I get lots of these messages in `dmesg`:

    swiotlb: coherent allocation failed for device 0000:00:01.0 size=2097152
    CPU: 0 PID: 9635 Comm: kworker/u8:15 Tainted: G           O     4.15.0-1-amd64 #1 Debian 4.15.4-1
    Hardware name: LENOVO 80NY/Allsparks 5B, BIOS CECN43WW 09/15/2015
    + A long stack trace with calls such as: `amdgpu_gem_object_create`, `ttm_tt_bind` and `swiotlb_alloc_coherent` (i.e.: memory related)

This may be related to the accounting issues…
Comment 49 Erin Yuki Schlarb 2018-02-26 12:49:36 UTC
Hmm… At least for me there seems to be a pretty obvious corrolation here:

    QSG_RENDER_LOOP=basic    → Memory leak
    QSG_RENDER_LOOP=threaded → No issues, no leaking

Can other people confirm this (both with MESA and with other/propritary GL libraries)?
Comment 50 Marco Martin 2018-02-26 16:35:29 UTC
can this be tried again with a more recent Qt? either the latest 5.9 or 5.10
making sure both of these patches are in:

https://codereview.qt-project.org/#/c/200715/
https://codereview.qt-project.org/#/c/202781/
Comment 51 gertlink_nospam 2018-02-26 16:49:21 UTC
QSG_RENDER_LOOP=threaded solves the problem here.

Graphics:  Card-1: Intel 2nd Generation Core Processor Family Integrated Graphics Controller
           Card-2: NVIDIA GF117M [GeForce 610M/710M/810M/820M / GT 620M/625M/630M/720M]
           Display Server: x11 (X.Org 1.19.6 ) driver: intel Resolution: 1366x768@60.05hz
           OpenGL: renderer: Mesa DRI Intel Sandybridge Mobile version: 3.3 Mesa 17.3.5
Comment 52 gertlink_nospam 2018-02-26 16:52:03 UTC
Plasma 5.12.1
QT 5.10.1
Comment 53 Antonio Orefice 2018-02-26 16:56:49 UTC
QSG_RENDER_LOOP makes no difference to me (i'm not testing media frame widgets or slideshow wallpaper, see first post).
Comment 54 Erin Yuki Schlarb 2018-02-26 20:24:22 UTC
@Marco Martin: I've tested with the [Qt binary 5.9.4 binary release](https://download.qt.io/official_releases/qt/5.9/5.9.4/) by setting the correct `PATH`, `LD_LIBRARY_PATH`, `QML2_IMPORT_PATH` & `QT_PLUGIN_PATH` and starting the system `plasmashell` binary with `QCG_RENDER_LOOP=basic`.

I'm happy to report that there was no apparent memory leaking over the course of over an hour. 😃
Comment 55 Matt Whitlock 2018-02-26 20:26:24 UTC
(In reply to Alexander Schlarb from comment #54)
> `QCG_RENDER_LOOP=basic`.

It's supposed to be QSG_RENDER_LOOP. ("SG" for "scene graph.")
Comment 56 Erin Yuki Schlarb 2018-02-26 22:17:59 UTC
That's a typo, the actual command had been issued with `QSG_RENDER_LOOP=basic`.
Comment 57 Matt Whitlock 2018-02-27 01:14:54 UTC
(In reply to Marco Martin from comment #50)
> can this be tried again with a more recent Qt? either the latest 5.9 or 5.10
> making sure both of these patches are in:
> 
> https://codereview.qt-project.org/#/c/200715/
> https://codereview.qt-project.org/#/c/202781/

The second patch set (202781) is a reversion of the first patch set (200715) plus one extra line [sgrc->endSync();]. I confirm that the dev-qt/qtdeclarative-5.9.4 that is built on my Gentoo system does contain that added line.

(In reply to Alexander Schlarb from comment #48)
>   2. Memory usage of the offending process will be accounted correctly with
> running it with `LIBGL_ALWAYS_SOFTWARE=true`, but the leaking issue will
> persist

I tried this and do confirm that the leak persists and is accounted to the plasmashell process. In this case, radeontop shows unchanging VRAM usage, which is not surprising, given that the rendering is happening all in software. (The transitions between wallpaper images are a lot jerkier, and CPU usage is through the roof.)

It's worth noting that triggering this bug is not always so simple as starting up a plasmashell and running a slideshow. For me, it can take many minutes of running an aggressive slideshow before plasmashell starts to leak memory. Maybe this suggests that a race condition is the initial triggering event that leads into the state in which plasmashell leaks textures.

Also, I'm not sure if this is a placebo effect, but it may be that mousing over the launcher icons and task manager buttons in the panel (to cause the sliding tooltips effect) may increase the likelihood of starting the memory leak. Once it starts leaking, it's dramatic and fast when you're running a slideshow wallpaper with a 1-second interval.

(In reply to gertlink_nospam from comment #51)
> QSG_RENDER_LOOP=threaded solves the problem here.

After about 10 minutes of trying, I have been unable to trigger the leak when running plasmashell with the threaded render loop. I'll leave the slideshow wallpaper enabled (with a less aggressive, 29-minute interval), using this threaded render loop, and I'll continue to monitor for excessive VRAM usage.
Comment 58 boonhead.nl 2018-02-27 08:37:39 UTC
Holy cow that worked! (pkill plasmashell; QSG_RENDER_LOOP=threaded kstart plasmashell)
Now my system is running smooth as f*! :D
Even closing opened applications returns my total system memory usage back to a mere 678 KB!

I've been experiencing this problem since the upgrade from KDE Neon 5.8 LTS to 5.12 LTS. It might not be relevant, but my theme has lots of semi-transparent elements and my background changes every 30 seconds.
I could not survive an hour before even my swap ran out. A movie would hang 30 minutes in.

For the time being, I have modified file /etc/xdg/autostart/org.kde.plasmashell.desktop to contain:
Exec=QSG_RENDER_LOOP=threaded plasmashell --shut-up

And deleted file /etc/xdg/autostart/plasmashell.desktop since it was an almost exact duplicate, the only difference being that it included the "--shut-up" parameter of which I can't find any documentation but is mentioned a lot with "100% CPU usaged".
Comment 59 Antonio Orefice 2018-02-27 10:27:37 UTC
I'm glad you guys found a workaround for your particular case.
However, mine (see first post), is still present.
I just made a video showing how plasma can reach about 1GB of ram usage from 300MB in a bunch of seconds.
Something is better than what i observed in comment #16 and #17, but leaks are still present.
see:
https://www.youtube.com/watch?v=DviiWnq2Rp8
Comment 60 Matt Whitlock 2018-02-27 10:39:05 UTC
(In reply to Antonio Orefice from comment #59)
> I just made a video showing how plasma can reach about 1GB of ram usage from
> 300MB in a bunch of seconds.

I too can reproduce a rather serious memory leak just by scrolling the "Add Widget" panel up and down repeatedly, as you demonstrated in your video. This seems to be a separate issue from the scene graph basic render loop texture leak. This one doesn't cause VRAM usage to increase but rather causes the plasmashell process's VmRSS to increase.
Comment 61 Antonio Orefice 2018-02-27 10:44:02 UTC
Exactly.
Exactly what i reported when i opened this bug, see comment #1.
I don't want to be harsh, but this 'bug report' is becoming a mess since it has been arbitrarily joined/defaced with something unrelated.
Comment 62 Erin Yuki Schlarb 2018-02-27 11:41:17 UTC
@Antonio Orefice: Sorry 'bout that. There seemed to be a connection between these two issues that apparently isn't there.

Since the QSG issue has fixed in Qt 5.9.4 and 5.10.0, I nominate that we close 387341 (that is about a VRAM memory leak with QSG_RENDER_LOOP=basic and MESA drivers, as we now know) as resolved upstream and return the discussion of this thread to the RSS memory leak with the nVidia binary drivers.

I cannot, btw, reproduce your issue: There is a short increase in memory usage when attempting any of the actions in the video, but the memory is free'd quickly again.
Comment 63 Matt Whitlock 2018-02-27 11:45:49 UTC
(In reply to Alexander Schlarb from comment #62)
> the RSS memory leak with the nVidia binary
> drivers.

Um, I'm using the open-source AMDGPU driver, and I can easily drive up plasmashell's VmRss by doing what Antonio did in his demo video.
Comment 64 Erin Yuki Schlarb 2018-02-27 12:34:46 UTC
I feel like an idiot. I just tried running plasmashell with qt 5.9.4 “again” and found that it crashes on startup. Looking through my command-history it seems that all the search paths were slightly wrong *facepalm*.
Just forget everything I said about QSG_RENDER_LOOP=basic working in newer Qt…
Comment 65 Christoph Feck 2018-02-28 23:56:26 UTC
*** Bug 390238 has been marked as a duplicate of this bug. ***
Comment 66 Christoph Feck 2018-03-01 13:00:51 UTC
*** Bug 390436 has been marked as a duplicate of this bug. ***
Comment 67 Rex Dieter 2018-03-04 15:24:28 UTC
*** Bug 391377 has been marked as a duplicate of this bug. ***
Comment 68 David Edmundson 2018-03-16 03:10:50 UTC
There's a branch of plasmashell davidedmundson/wallpaperrewrite

Could someone reliably with this bug build and try it please.

It's not written to specifically fix anything, there is no bug in plasma-side code, but it does remove a lot of the complex intertwining of image loading and animation scripts that we previously had.
Comment 69 Kishore Gopalakrishnan 2018-03-18 17:37:07 UTC
(In reply to David Edmundson from comment #68)
> There's a branch of plasmashell davidedmundson/wallpaperrewrite
> 
> Could someone reliably with this bug build and try it please.
> 
> It's not written to specifically fix anything, there is no bug in
> plasma-side code, but it does remove a lot of the complex intertwining of
> image loading and animation scripts that we previously had.

I can still reproduce this bug with that branch. While the rate at which the memory usage increases is silghtly slower, the memory usage still increases to around 600 to 700 MiB quickly. I have tested with both the media frame and slideshow wallpaper (this was the one which showed the slower increase with your branch). The memory leak is still present in both cases (I tested with the image set to change at 3 sec intervals).

Unrelated to this issue, the option to blur the part not covered by the wallpaper when it is 'centered' seems to be broken in your branch. The blur is applied to the entire background, instead.
Comment 70 David Edmundson 2018-03-18 19:14:08 UTC
Not too surprising. Thanks for reporting back.
Comment 71 Nate Graham 2018-03-22 16:05:01 UTC
*** Bug 392174 has been marked as a duplicate of this bug. ***
Comment 72 Nate Graham 2018-03-22 17:57:35 UTC
So the consensus is that QSG_RENDER_LOOP=threaded doesn't fix the issue?

It seems like the discussion of this issue should take place in a Qt bug report, since it's not a KDE issue. https://bugreports.qt.io/browse/QTBUG-61754 has been closed as fixed; do we need to file a new bug or re-open that one?
Comment 73 Matt Whitlock 2018-03-22 18:05:11 UTC
(In reply to Nate Graham from comment #72)
> So the consensus is that QSG_RENDER_LOOP=threaded doesn't fix the issue?

There are two issues. QSG_RENDER_LOOP=threaded *does* avoid the VRAM memory leak but *does not* avoid the in-process memory leak.
Comment 74 Nate Graham 2018-03-22 19:02:32 UTC
(In reply to Matt Whitlock from comment #73)
> There are two issues. QSG_RENDER_LOOP=threaded *does* avoid the VRAM memory
> leak

Thanks Matt. So should Qt use the threaded render loop by default, then? Or should we? Or something else?


> but *does not* avoid the in-process memory leak.

So perhaps we need a Qt bug to track this. Was https://bugreports.qt.io/browse/QTBUG-61754 tracking that such that it should be re-opened, or do we need a new one?
Comment 75 David Edmundson 2018-03-22 19:25:08 UTC
*** Bug 392181 has been marked as a duplicate of this bug. ***
Comment 76 Matt Whitlock 2018-03-22 20:06:56 UTC
(In reply to Nate Graham from comment #74)
> So should Qt use the threaded render loop by default, then? Or
> should we? Or something else?

Presumably the VRAM memory leak in the basic render loop should be found and fixed. I would stop short of suggesting that plasmashell switch to the threaded render loop by default in the meantime, as I do not know whether the threaded render loop has drawbacks.

> So perhaps we need a Qt bug to track this. Was
> https://bugreports.qt.io/browse/QTBUG-61754 tracking that such that it
> should be re-opened, or do we need a new one?

I do not know the cause of the in-process memory leak. It may be that QTBUG-61754 is not really fixed, or it may be a fault in plasmashell, or it may be something else.
Comment 77 Erin Yuki Schlarb 2018-03-22 22:26:41 UTC
Please just keep in mind that the current first-use experience is absolutely abysimal for everybody using the described configuration: You log into an up-to-date system with plasmashell and everything will appear to work fine. A few minutes/hours (depending on configuration) your system will inexplicietly become slow and finally either your KDE desktop or (more likely) your entire system will crash/hang. This will happen every time you use KDE, but not on whatever DE/OS you came from and there won't be any obvious reason.

Likely you won't be using KDE for a long time after having such an experience a couple of times. (The people here bothering to dig up such a hard to grasp issue are the absolute minority, please don't forget that.)
Yes, I'm very much advocating that `QSG_RENDER_LOOP=threaded` be made default at least on Desktop MESA platforms.

> Presumably the VRAM memory leak in the basic render loop should be found and fixed.

Is Qt actively working on that? Or will they just stand by on this issue “'cause the threaded one is better anyways”?
I guess what I'm asking is this (and absolutely correct me if I got a wrong impression of the situation!): Is there anything important that Qt needs QSG_RENDER_LOOP=basic for, that QSG_RENDER_LOOP=threaded cannot do? Because presumably the basic loop was originally written first and then threaded got added for getting some extra performance at the cost of complexity, so it was decided to only enable it on (GL) platforms known to be “good”. So, assuming that my assumption on how this developed is correct, I'm asking you if there is any reason for them (or anybody) to keep the basic render loop in good shape, when the threaded one gets the job done anyway on all platforms they consider important and their long run goal is phase out usage of the basic loop anyways?
Comment 78 Nate Graham 2018-03-22 22:40:44 UTC
We're all aware of the issue's implications, and are trying to fix it.

I'm also wondering if there's any disadvantage to always using the threaded render loop. Alexander, would you be able to run with it for a while and report any regressions, if any?
Comment 79 David Edmundson 2018-03-23 00:16:59 UTC
> So should Qt use the threaded render loop by default, then? Or should we? Or something else?

Is decided by Qt based on your graphic card/drivers.

Maybe that selection code in Qt can be tweaked, but we most definitely shouldn't force it from our side.

Especially not this way round (forcing basic would be maybe ok). If you do this on a driver that doesn't support threaded GL you're going to have random crashes at any point.
Comment 80 Nate Graham 2018-03-23 03:31:33 UTC
Basically, we need to find a path forward here. This bug is getting pretty long and squirrelly, especialyl considering that it's not our fault. Do we need a few more Qt bugs, David?
Comment 81 Erin Yuki Schlarb 2018-03-23 13:38:09 UTC
@Nate Graham:
> I'm also wondering if there's any disadvantage to always using the threaded render loop. Alexander, would you be able to run with it for a while and report any regressions, if any?
I've been running my entire KDE Desktop session with `QSG_RENDER_LOOP=threaded` set in `.pam_environment` for the last month and it only had an effect on Plasma's (invisible) memory usage. Everything else is working as before (i.e.: KWin still has its share of issues –particularily on Wayland – but it's not worse than before either).

As a matter of fact with this bug it's so bad I wouldn't be using Plasma at all (like I did for 2 months before finding this solution) if the issue wouldn't have been fully resolved by this.



@David Edmundson:
> Is decided by Qt based on your graphic card/drivers.
I did some digging in the Qt source code on why this is disabled and it mostly seems to be because of a bug in an old (pre-2014) version of `libxcb` that caused issues for MESA. I filled a bug with Qt to suggest reevaluating that blacklist entry – we'll see what they say:
https://bugreports.qt.io/browse/QTBUG-67277

> Maybe that selection code in Qt can be tweaked, but we most definitely shouldn't force it from our side.
If Qt decides that it's safe to remove that blacklist entry for 5.11+ or maybe just 5.12+, then forcing this from Plasmashell's side for older should IMHO be considered (with an appropriate version guard and comment so that's its easy to remove once support for these versions is dropped of course).
Comment 82 Anders Lund 2018-03-26 11:27:33 UTC
fredag den 23. marts 2018 14.38.09 CEST skrev du:
> https://bugs.kde.org/show_bug.cgi?id=368838
> 
> --- Comment #81 from Alexander Schlarb <alexander-kde@ninetailed.ninja> ---
> 
> @Nate Graham:
> > I'm also wondering if there's any disadvantage to always using the
> > threaded render loop. Alexander, would you be able to run with it for a
> > while and report any regressions, if any?
> I've been running my entire KDE Desktop session with
> `QSG_RENDER_LOOP=threaded` set in `.pam_environment` for the last month and
> it only had an effect on Plasma's (invisible) memory usage. Everything else
> is working as before (i.e.: KWin still has its share of issues
> –particularily on Wayland – but it's not worse than before either).

Thanks for the tip, I'm trying that, so far witout any problems :)

---
Kindly,
Anders
Comment 83 David Edmundson 2018-03-26 20:42:39 UTC
*** Bug 391063 has been marked as a duplicate of this bug. ***
Comment 84 David Edmundson 2018-03-26 22:21:12 UTC
Now managed to reproduce locally...but not even slightly consistently.

Can someone who has it reliably check if reverting eeb320bbd8763f3e72f79369cc3908e999a0da3c in QtDeclarative fixes it.

My theory from some other user's output in 381000 was that we have a situation in plasma where Qt's basic loop always things some other window needs updates and we never hit endSync which releases the textures; but then that theory was drowned in confusion from people talking about other bugs and it got lost.
Comment 85 David Edmundson 2018-03-26 22:26:52 UTC
Lucky timing, just gave up and hit submit.

Had

diff --git a/src/quick/scenegraph/qsgrenderloop.cpp b/src/quick/scenegraph/qsgrenderloop.cpp
index ca1fb9ba5..a8beac369 100644
--- a/src/quick/scenegraph/qsgrenderloop.cpp
+++ b/src/quick/scenegraph/qsgrenderloop.cpp
@@ -418,6 +418,8 @@ void QSGGuiThreadRenderLoop::renderWindow(QQuickWindow *window)
     emit window->afterAnimating();
 
     cd->syncSceneGraph();
+
+    qDebug() << "maybe ending sync" << lastDirtyWindow;
     if (lastDirtyWindow)
         rc->endSync();



all the while not leaking this was printing with true on the end...then randomly just as I started doing something else I'm seeing this always false.. and it's clear from the code that at that point it stops releasing anything.

If we find the cause of that, we'll get this fixed.
Comment 86 David Edmundson 2018-03-26 23:26:37 UTC
Steps to reproduce:

 - set to QSG_RENDER_LOOP=basic
 - create a slideshow
 - open a taskmanager tooltip
 - then we leak

Tooltip is not rendered because it's not visible.
But it's re-used so still exists.
QSGGuiThreadRenderLoop::renderWindow() returns false for that window without doing anything.
But it's still marked as updatePending, so we never think we've rendered all the windows.

This patch in Qt will fix it; though arugably isn't quite at the right cause.

diff --git a/src/quick/scenegraph/qsgrenderloop.cpp b/src/quick/scenegraph/qsgrenderloop.cpp
index ca1fb9ba5..51b4775f2 100644
--- a/src/quick/scenegraph/qsgrenderloop.cpp
+++ b/src/quick/scenegraph/qsgrenderloop.cpp
@@ -384,7 +384,8 @@ void QSGGuiThreadRenderLoop::renderWindow(QQuickWindow *window)
     bool lastDirtyWindow = true;
     auto i = m_windows.constBegin();
     while (i != m_windows.constEnd()) {
-        if (i.value().updatePending) {
+        QQuickWindowPrivate *cd = QQuickWindowPrivate::get(i.key());
+        if (i.value().updatePending && cd->isRenderable()) {
             lastDirtyWindow = false;
             break;
         }
Comment 87 Gauthier 2018-03-27 09:27:57 UTC
Hi,

I have memory leak that occurs without having the slide-show but all other bugs on memory leak seem to be linked to this one so I'm posting here.

Since updating to plasma 5.12.x on neon LTS, a memory leak started to appear on plasmashell. This is not happening on my other machine on neon user and both desktop have same config (same background, widgets, etc.). Main difference between the two is QT 5.9.3 on neon LTS whereas QT 5.10 on neon user.

So I feel this is a different issue to the one reported here, and probably fixed by QT 5.10 and so ideally should be reported as an inconsistency in the update to plasma 5.12 in neon LTS...but I don't know where to report this .
Comment 88 Nate Graham 2018-03-27 13:42:33 UTC
(In reply to Gauthier from comment #87)
> Hi,
> 
> I have memory leak that occurs without having the slide-show

> So I feel this is a different issue to the one reported here

Please don't report unrelated issues in a pre-existing bug report; file a new one instead. See https://community.kde.org/Get_Involved/Bug_Reporting#One_issue_per_bug_report

Also, if the issue is already resolved in a later Qt version, there is no need to report a bug at all.
Comment 89 David Edmundson 2018-03-27 23:39:46 UTC
Please test: https://codereview.qt-project.org/#/c/224684/
Comment 90 Hannes Schweizer 2018-03-31 12:17:33 UTC
Setup:
dev-qt/qtdeclarative-5.9.4
kde-plasma/plasma-desktop-5.12.3
QSG_RENDER_LOOP unchanged
OpenGL vendor string: X.Org
OpenGL renderer string: AMD CAYMAN (DRM 2.50.0 / 4.15.4-gentoo, LLVM 5.0.1)
OpenGL core profile version string: 4.3 (Core Profile) Mesa 18.0.0

Before applying your patch a 1s interval slideshow while playing around with plasma reproducibly led to a mem increase of rougly 50-100MB/s.
After applying the patch and restarting plasma => mem consumption looks stable for over an hour now.

You sir, deserve a medal!
Comment 91 Patrick Silva 2018-04-01 11:47:36 UTC
*** Bug 392599 has been marked as a duplicate of this bug. ***
Comment 92 David Edmundson 2018-04-02 22:12:47 UTC
*** Bug 392568 has been marked as a duplicate of this bug. ***
Comment 93 David Edmundson 2018-04-09 16:21:48 UTC
*** Bug 392524 has been marked as a duplicate of this bug. ***
Comment 94 David Edmundson 2018-04-09 16:23:04 UTC
FYI. Merged now into Qt 5.11.0. Due to be released in a few months
Comment 95 Jesse 2018-04-09 17:12:09 UTC
I tried the patch but the memory leak bug reported in #392568, which was marked as a duplicated of this one, still persists for me. So either I'm missing something or that bug is not really a duplicated.
Comment 96 David Edmundson 2018-04-11 00:43:04 UTC
*** Bug 392668 has been marked as a duplicate of this bug. ***
Comment 97 Zakhar 2018-04-11 00:46:01 UTC
(In reply to Jesse from comment #95)
> I tried the patch but the memory leak bug reported in #392568, which was
> marked as a duplicated of this one, still persists for me. So either I'm
> missing something or that bug is not really a duplicated.

My bug 392668 which was also marked duplicate will likely also persist, but we will have to see.
Comment 98 David Edmundson 2018-04-18 12:47:54 UTC
*** Bug 393248 has been marked as a duplicate of this bug. ***
Comment 99 Jason 2018-04-19 01:12:23 UTC
how do I go about changing "QSG_RENDER_LOOP" to threaded? FYI, when I run from a terminal "QSG_INFO=1 plasmashell" I get nothing back.
Comment 100 Erick Osorio 2018-04-19 04:10:07 UTC
(In reply to David Edmundson from comment #89)
> Please test: https://codereview.qt-project.org/#/c/224684/

David, do will your patch be in QT 5.10?
Comment 101 Nate Graham 2018-04-19 04:38:05 UTC
Qt 5.10 is closed now, so it will be in Qt 5.11. Ask your distros to backport it (as Kubuntu already has into the Qt 5.9.5 they'll ship with Kubuntu 18.04: https://bugs.launchpad.net/ubuntu-rtm/+source/qtdeclarative-opensource-src/+bug/1761016).
Comment 102 DeMus 2018-04-28 15:01:26 UTC
KDE 5.12.4, Frameworks 5.45.0, Qt 5.10.1.
When using the slide-show memory is filling up pretty fast.
Comment 103 Nate Graham 2018-04-28 15:03:22 UTC
See my previous comment: this has been fixed in *Qt 5.11*. Ask your distros to backport the patch to their Qt 5.9.x and 5.10.x packages.
Comment 104 Ovidiu Chi 2018-04-29 10:34:42 UTC
Nate, David, will the fix be backported to KDE Neon?
I'm asking here since Neon is the showcase of KDE software.
Thanks!
Comment 105 David Edmundson 2018-04-30 13:02:22 UTC
*** Bug 377795 has been marked as a duplicate of this bug. ***
Comment 106 Ovidiu Chi 2018-05-16 12:50:07 UTC
Thought to give an update.
I'm using Kubuntu 18.04.
The original issue reported here by Antonio seems fixed. Video here: https://vimeo.com/270079798
The memory leak due to wallpaper changes isn't. Video here: https://vimeo.com/270080920
Comment 107 David Edmundson 2018-05-16 13:22:47 UTC
>Nate, David, will the fix be backported to KDE Neon?

I don't deal with any downstreams. You'll have to ask them.

As for your video, Qt 5.11 isn't out yet, are you sure Kubuntu has it?
Comment 108 Ovidiu Chi 2018-05-16 13:40:14 UTC
(In reply to David Edmundson from comment #107)

> As for your video, Qt 5.11 isn't out yet, are you sure Kubuntu has it?

Kubuntu has Qt 5.9.5, with a backport of your fix as stated here: https://bugs.launchpad.net/ubuntu/+source/qtdeclarative-opensource-src/+bug/1761016

But Nate is best able to answer your question.
Comment 109 Nate Graham 2018-05-16 16:12:20 UTC
The bug was not that plasmashell uses memory, but rather that the usage grew infinitely and the memory was never released. In Kubuntu 18.04, I'm happily using a slideshow wallpaper and yes, while plasmashell consumes memory in the process, the memory usage doesn't grow infinitely and it doesn't hold onto that memory forever like it used to.
Comment 110 Ovidiu Chi 2018-05-16 17:52:56 UTC
(In reply to Nate Graham from comment #109)

I hate to be the "bad news" guy, but this looks exactly like what you describe - an ever growing, never-releasing memory problem.
Here is a slideshow video: https://vimeo.com/270145715

The rate of increase seems lower than before the fix, but it's there. With the 4k pics it looks like about 30MB per image switch.
Try going from 3seconds to 2seconds or vice-versa. Changing the interval at least once seems to matter.
Three of the 4k pictures are approx. 1.5MB, the eagle is 350KB, the blue desert is 900KB, the mountain valley is 2.7 MB.
The Ubuntu ones were in the 500KB vicinity.
The problem is not as apparent with small image files.

P.S. A few tests later, plasmashell is at 4.4GB resident memory.
Comment 111 Matt Whitlock 2018-05-16 19:07:31 UTC
(In reply to Ovidiu Chi from comment #110)

Plasmashell *does* still have a memory leak, but it's not the one from this bug. Actually, it *is* the one from the original report in Comment #0, but this thread morphed into being about the other leak, which has been fixed. The original, in-process (non-VRAM) memory leak still exists and can be triggered by making plasmashell repaint a lot of images, such as by aggressively scrolling the list of available widgets.
Comment 112 David Edmundson 2018-05-16 21:23:44 UTC
@Ovidiu, A packager stating something doesn't necessarily mean much. Can you point me to the packaging code. Also do you have https://codereview.qt-project.org/#/c/202781/  which is for a different leak.

@Matt
How can you tell what is leaking from looking at comment #0? and what do you mean by an "in-process leak". All leaks are in-process.
Comment 113 Matt Whitlock 2018-05-16 23:20:45 UTC
(In reply to David Edmundson from comment #112)
> @Matt
> How can you tell what is leaking from looking at comment #0?

There have been at least two distinct memory leaks discussed in this thread. The original report by Antonio Orefice is about a memory leak that occurs when Plasma renders images. Antonio posted a link to a video demonstrating the leak in Comment #59. I am able to reproduce his leak by the same method: force Plasma to paint a lot of images in a short amount of time. It will also leak just the same eventually in normal use, but it takes longer. The leak occurs even without opening the "Add Widget" panel and scrolling the list up and down like mad, but that's one quick way to reproduce it.

The other leak was due to the logic error that you described in Comment #86. That leak has been fixed by https://codereview.qt-project.org/#/c/202781/.

> and what do you mean by an "in-process leak". All leaks are in-process.

Firstly, that's not true; it's entirely possible to leak pixmaps, and those are in the X server's process, not the X client's. But in this case, what I meant was that the leak that was fixed by calling endSync() was a leak of graphics memory, not a leak of system memory, so it wasn't accounted to the plasmashell process.
Comment 114 Ovidiu Chi 2018-05-16 23:57:22 UTC
(In reply to Matt Whitlock from comment #113)

Matt, I'm not knowledgeable enough to say which memory leak this is, or the precise nature of it, but scrolling the list of widgets doesn't do much for my system as you can see in the first video in my comment #106. When I was using KDE Neon I could get the memory usage to 1GB in a few seconds by doing just that.
Keep in mind that Kubuntu 18.04 has backported the Qt patch slated for release with Qt 5.11.
As it stands, in practice, Kubuntu's memory seems fine as long as I don't test a new wallpaper to use, or fiddle with the slideshow feature.
Comment 115 Ovidiu Chi 2018-05-16 23:59:17 UTC
(In reply to David Edmundson from comment #112)
> @Ovidiu, A packager stating something doesn't necessarily mean much. Can you
> point me to the packaging code. Also do you have
> https://codereview.qt-project.org/#/c/202781/  which is for a different leak.

David, I'm not a packager, nor a Qt/C++ dev so I couldn't point you at much except for user facing stuff.
The link looks like a commit. If you're asking if my system has that included, I don't know how to check precisely (I assume it would be in binary form anyway). But you mentioned at some point it should be in Qt >= 5.9.3. Kubuntu 18.04 has 5.9.5.
I was hoping someone could reproduce what I showed in the videos, with the known fixes applied: either switch wallpapers manually for a couple of minutes or start a slideshow with 4k images and change from 3s to 2s in the middle of it. Memory should increase continuously and not be released.
Comment 116 Matt Whitlock 2018-05-17 00:31:05 UTC
(In reply to Ovidiu Chi from comment #114)
> Matt, I'm not knowledgeable enough to say which memory leak this is, or the
> precise nature of it, but scrolling the list of widgets doesn't do much for
> my system as you can see in the first video in my comment #106. When I was
> using KDE Neon I could get the memory usage to 1GB in a few seconds by doing
> just that.

The leak demonstrated by rapidly scrolling the list of available widgets occurs *much* more slowly on my system now than it used to. I too could previously get plasmashell up over 1 GB in under a minute. I tried again just now and was able to bloat it up from just under 200 MB to over 300 MB with about five minutes of scrolling the widgets list. So it's *better* but still not totally solved.

The leak demonstrated by the slideshow is fully gone for me. I am running a slideshow of 2560x1600 images every 2 seconds with QSG_RENDER_LOOP=basic as I write this, and both plasmashell's memory usage and the VRAM usage (as reported by radeontop) are remaining in a narrow range.
Comment 117 Ovidiu Chi 2018-05-17 12:40:48 UTC
> I am running a slideshow of 2560x1600 images every 2 seconds with QSG_RENDER_LOOP=basic

@Matt, You'd have to start the slideshow then, after a little while(a few seconds), change the duration (e.g. from 3s to 2s).

I did some more tests.
I set `export QSG_RENDER_LOOP=threaded` in `~/.config/plasma-workspace/env/env.sh` and logged back in.
I cannot reproduce the problem anymore. The memory either doesn't increase or rises very little and gets freed afterwards. Everything looks stable.

I then set `export QSG_RENDER_LOOP=basic` in the same file and I could reproduce again.
So it seems Kubuntu 18.04 is set to `basic` by default. I entered `QSG_INFO=1 plasmashell` in the terminal to find out the value, but nothing is displayed.

I don't know if the behavior is the same on all graphic cards.
If someone is able to test this out, great, otherwise it will be another tweak I need to write down.

System details:
CPU: Intel i7-6700HQ (8) @ 3.500GHz 
GPU: Intel HD Graphics 530 
GPU: NVIDIA GeForce GTX 960M 

OpenGL renderer string: Mesa DRI Intel(R) HD Graphics 530 (Skylake GT2) 
I have not installed the Nvidia drivers.

I've set GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nouveau.modeset=0" in `/etc/default/grub` so I can log in, shut down and suspend the laptop properly.
Comment 118 David Edmundson 2018-05-17 12:48:56 UTC
>. I entered `QSG_INFO=1 plasmashell` in the terminal to find out the value, but nothing is displayed.

You need to quit the old plasmashell beforehand.

>So it seems Kubuntu 18.04 is set to `basic` by default.

Qt swtiched based on graphic driver. Some don't support GL use in threads.


-----

As for Kubuntu. Looking at http://archive.ubuntu.com/ubuntu/pool/universe/q/qtdeclarative-opensource-src/qtdeclarative-opensource-src_5.9.5-0ubuntu1.debian.tar.xz  from https://packages.ubuntu.com/source/bionic/qtdeclarative-opensource-src  I do not see my patch applied.
Comment 119 Nate Graham 2018-05-17 14:27:33 UTC
Which fix is it lacking, David?

https://codereview.qt-project.org/#/c/224684/
or
https://codereview.qt-project.org/#/c/202781/?

Or both? :/

Either way, I'll go get this fixed...
Comment 120 Matt Whitlock 2018-05-17 14:29:07 UTC
(In reply to Ovidiu Chi from comment #117)
> > I am running a slideshow of 2560x1600 images every 2 seconds with QSG_RENDER_LOOP=basic
> 
> @Matt, You'd have to start the slideshow then, after a little while(a few
> seconds), change the duration (e.g. from 3s to 2s).

Okay, I have restarted plasmashell with QSG_RENDER_LOOP=basic QSG_INFO=1, and I do indeed see this in the output:

qt.scenegraph.general: QSG: basic render loop

I set a slideshow at an interval of 3 seconds and started monitoring plasmashell's memory usage and VRAM usage. It has been a few minutes, and I see no extreme rise in memory usage. (Plasmashell, since it hosts a garbage-collected runtime environment, does grow in memory usage for a little while after startup until it is "warm.")

Something I forgot to mention before: I do make sure to show some tooltips in the task switcher, pop up the applications menu, pop up the calendar, etc., to be certain that there are some previously rendered but now hidden windows. This was the trigger of the logic bug originally.

Now I change the slideshow interval from 3 seconds to 2 seconds. I observe no extreme increase in plasmashell's memory usage or VRAM usage. Plasmashell's usage still gently rises over time, due to the other leak, which is still unaddressed, but the major VRAM leak is not present.


(In reply to David Edmundson from comment #118)
> Qt [switches] based on graphic driver. Some don't support GL use in threads.

You would think that Mesa/Gallium/RadeonSI (AMDGPU) would be modern enough for Qt to use the threaded renderer by default, but "basic" is still the default on my system. I am going to leave the QSG_RENDER_LOOP=threaded in my session environment, as I experience no problems with it whatsoever.
Comment 121 David Edmundson 2018-05-17 15:26:49 UTC
@Nate

I can't see https://codereview.qt-project.org/#/c/224684/ in the debian packaging tarball. It's in the debian packaging git repo though.
Comment 122 Antonio Orefice 2018-05-19 08:54:55 UTC
So, i open another one, it is simpler.
Could you guys please test? It is a plasma js script that will use all of my memory just re-aligning folderview icons over and over
Here:
https://bugs.kde.org/show_bug.cgi?id=394436
Comment 123 Rik Mills 2018-05-23 00:36:51 UTC
@Nate @David

Appears the mentioned fix is in debian packaging repo for their master branch, but not in the ubuntu/bionic branch used for Ubuntu 5.9.x uploads. At a guess, as the launchpad bug was marked as fixed, I would guess that was an oversight, or it was thought the fix was in 5.9.5 tars themselves.

Letting Qt guys know.
Comment 124 Darin Miller 2018-05-23 01:10:16 UTC
Memory leak still present with Kubuntu 18.04 updated as follows: 
  Plasma 5.12.5
  FW 5.44.0 
  QT 5.9.5 
  HW: Toshiba Satellite laptop, Intel graphics

Both the media frame widget and desktop background are affected.  Setting the update interval to 10s (or less) and parking a memory widget on the desktop was the method used for monitoring.

Note: pkill plasmashell; QSG_RENDER_LOOP=threaded kstart plasmashell also stopped  the leak.
Comment 125 Nate Graham 2018-05-23 16:57:43 UTC
Thanks Rik. I'd love to know what you hear. It would be nice to finally get this fixed in Kubuntu once and for all.
Comment 126 David Edmundson 2018-05-28 16:07:34 UTC
*** Bug 392505 has been marked as a duplicate of this bug. ***
Comment 127 cdobrich 2018-05-28 21:33:08 UTC
(In reply to Rik Mills from comment #123)
@Rik Mills @Nate Graham 

Would you please alert this thread when the fixes have made their way into the Kubuntu 18.04 repositories. I'm waiting on updating until this fix has trickled down, because it is crippling bad for my system.
Comment 128 alexander 2018-08-13 18:55:50 UTC
(In reply to cdobrich from comment #127)
> (In reply to Rik Mills from comment #123)
> @Rik Mills @Nate Graham 
> 
> Would you please alert this thread when the fixes have made their way into
> the Kubuntu 18.04 repositories. I'm waiting on updating until this fix has
> trickled down, because it is crippling bad for my system.

I have the same problem. It took me ages to find that plasmashell and the slideshow is the problem .... almost ditched Kubuntu 18.04 completely.
Comment 129 Matthew Woehlke 2018-12-31 16:35:09 UTC
What's the story with this? Because it is apparently still happening with qt5-5.11.1-9.fc29.x86_64... (I just killed a plasma that had ballooned to almost 27G VIRT.)
Comment 130 David Edmundson 2018-12-31 17:16:28 UTC
Virtual memory is not a relevant measure of anything useful. Only Res + Shared matter.

If you do think you have an issue with memory and confirm you have https://codereview.qt-project.org/#/c/224684/ applied please open a new bug with as much relevant new information as possible.
Comment 131 Matthew Woehlke 2018-12-31 17:35:01 UTC
> Virtual memory is not a relevant measure of anything useful.

That is neither necessarily true, nor necessarily untrue. While I have seen instances of processes having huge VIRT that didn't correspond to real memory usage (e.g. 256G VIRT with reasonable RES), most of the time there is a strong correlation between VIRT and actual memory use. (The other problem with RES is it doesn't account for "real" memory that has been swapped out, AFAIK. IMO both are useful measurements.)

At any rate, RES was insane also. Here's the full htop line:

  1893 matthew    20   0 26.9G 17.1G 20308 S  0.3 54.5  3h10:59 /usr/bin/plasmashell

I doubt I can provide any *new* information. I'm using slideshow wallpaper. After a while (not sure if it is "always" or somewhat at random; I only just upgraded to F29 about a week or so ago), memory usage grows to unreasonable size. The only thing I know that's "new" is that this is still happening with qt5-5.11.1-9.fc29.x86_64. (Based on inspecting the debuginfo sources, I *do* have 224684 applied.)