Bug 440574 - Memory Leak / Memory Usage For Large Feeds
Summary: Memory Leak / Memory Usage For Large Feeds
Status: REPORTED
Alias: None
Product: akregator
Classification: Applications
Component: general (show other bugs)
Version: 5.16.3
Platform: Ubuntu Linux
: NOR crash
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-08-03 22:40 UTC by barray
Modified: 2021-08-09 15:29 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
RAM usage over 24 hours (48.39 KB, image/png)
2021-08-04 23:24 UTC, barray
Details
Valgrind log after running akgregator for a few hours (1.64 MB, text/plain)
2021-08-05 01:25 UTC, barray
Details
RAM usage over time for built-from-source version (42.66 KB, image/png)
2021-08-07 01:23 UTC, barray
Details
Replicated RAM usage issue in a debug build (43.61 KB, image/png)
2021-08-09 01:06 UTC, barray
Details
Valgrind log after running for quite a few hours on built source (548.09 KB, application/x-xz)
2021-08-09 15:29 UTC, barray
Details

Note You need to log in before you can comment on or make changes to this bug.
Description barray 2021-08-03 22:40:26 UTC
SUMMARY

I am currently observing ~250MB of RAM usage increase per day. This lead to a system crash.

Currently Akregator is consuming 1.7GB of RAM, 4.9GB of virtual memory. Compare this to Firefox with hundreds of active tabs, consuming 1.4GB of RAM and 8.9GB of virtual memory.

STEPS TO REPRODUCE
1. Large number of RSS feeds (OPML export is ~300kB and ~700 lines)
2. Set checking time to 30 minute intervals
3. Wait for 24 hours

OBSERVED RESULT

A ~250MB RAM increase over 24 hours. Over several weeks this will lead to a system lock-up/crash (in this case).

EXPECTED RESULT

A little to no RAM increase over 24 hours.

SOFTWARE/OS VERSIONS
Windows: -
macOS: -
Linux/KDE Plasma: Ubuntu 21.04
(available in About System)
KDE Plasma Version: -
KDE Frameworks Version: 5.80.0
Qt Version: 5.15.2

ADDITIONAL INFORMATION
Comment 1 barray 2021-08-03 22:56:15 UTC
I'm logging the RAM usage every 300 seconds using the SO answer here: https://unix.stackexchange.com/a/508799

The code is:

```
delay=300
pid="$1"
top -b -d $delay -p $pid | awk -v OFS="," '$1=="top"{ time=$3 } $1+0>0 { print time,$1,$NF,$9,$10; fflush() }'
```

This gives RAM as a percentage of total memory, but should allow the observation of memory increase to be obvious.

The purpose of doing this is to see whether it is linear (memory leak in small loop), periodic (when feeds are pulled) or sporadic (specific feed content is leaking memory). It also will show the extent of this problem.
Comment 2 barray 2021-08-04 23:24:15 UTC
Created attachment 140527 [details]
RAM usage over 24 hours

The attached image shows increased RAM usage of Akregator over a 24+ hour window - something weird is definitely going on here. I'll try to attach valgrind later and see what is specifically growing.
Comment 3 barray 2021-08-05 01:25:38 UTC
Created attachment 140528 [details]
Valgrind log after running akgregator for a few hours
Comment 4 Laurent Montel 2021-08-05 04:55:37 UTC
Thanks for valgrind report.
I will look at it.
Thanks
Comment 5 barray 2021-08-06 07:26:07 UTC
I have built from source (v20.12.3) and am now making sure it still replicates. If so, I plan to get a better valgrind output - as well as confirm the issue in later versions.
Comment 6 barray 2021-08-07 01:23:21 UTC
Created attachment 140567 [details]
RAM usage over time for built-from-source version

This is the built-from-source version - the issue doesn't appear to replicate (at least not to the same level).

It also crashes when selecting the menu item: `Settings > Configure Akregator`, giving:

    ./bin/akregator: symbol lookup error: /usr/lib/x86_64-linux-gnu/qt5/plugins/akregator_config_userfeedback.so: undefined symbol: _ZN9Akregator19UserFeedBackManager4selfEv

I suspect it could be some issue with the archiving [1].

I spoke with @pinaraf in #akregator on Libera IRC and they seemed to believe there are some large flaws in the archiving process in any case.

[1] https://bbs.archlinux.org/viewtopic.php?id=226889
Comment 7 Laurent Montel 2021-08-07 07:55:31 UTC
(In reply to barray from comment #6)
> Created attachment 140567 [details]
> RAM usage over time for built-from-source version
> 
> This is the built-from-source version - the issue doesn't appear to
> replicate (at least not to the same level).
> 
> It also crashes when selecting the menu item: `Settings > Configure
> Akregator`, giving:
> 
>     ./bin/akregator: symbol lookup error:
> /usr/lib/x86_64-linux-gnu/qt5/plugins/akregator_config_userfeedback.so:
> undefined symbol: _ZN9Akregator19UserFeedBackManager4selfEv
> 
> I suspect it could be some issue with the archiving [1].
> 
> I spoke with @pinaraf in #akregator on Libera IRC and they seemed to believe
> there are some large flaws in the archiving process in any case.
> 
> [1] https://bbs.archlinux.org/viewtopic.php?id=226889

You rebuild akregator but it seems that it uses akregator_config_userfeedback.so from your system not you install directory. If you rebuild remove your akregator from your system first.
Comment 8 barray 2021-08-08 07:58:23 UTC
(In reply to Laurent Montel from comment #7)
> You rebuild akregator but it seems that it uses
> akregator_config_userfeedback.so from your system not you install directory.
> If you rebuild remove your akregator from your system first.

I have done this and will try to replicate the issue again. I can now access the menus and metakit appears to be working.
Comment 9 barray 2021-08-09 01:06:01 UTC
Created attachment 140591 [details]
Replicated RAM usage issue in a debug build

As you can see, I am now able to replicate the RAM usage issue in a debug build. The next step for me is to run this under valgrind and try to generate more useful output.

After that I want to run on the latest release tag and try to replicate the issue there.
Comment 10 barray 2021-08-09 15:29:22 UTC
Created attachment 140612 [details]
Valgrind log after running for quite a few hours on built source

It was horrifically slow, but I tried to use it as normal over a decent time window. Hopefully this valgrind log is more useful for debugging memory use.

My experience of reading valgrind output is quite limited, but I will take a look in a few days.

In the meantime, if you would like me to check anything else, please let me know.