Bug 494632

Summary:	systemd: prevent broken apps from spamming the journal
Product:	[Plasma] plasmashell	Reporter:	Peter Eszlari <peter.eszlari>
Component:	Startup process	Assignee:	Plasma Bugs List <plasma-bugs-null>
Status:	RESOLVED UPSTREAM
Severity:	normal	CC:	kde, pallaswept
Priority:	NOR
Version First Reported In:	6.1.5
Target Milestone:	1.0
Platform:	Other
OS:	Linux
Latest Commit:		Version Fixed/Implemented In:
Sentry Crash Report:

Description Peter Eszlari 2024-10-13 05:19:52 UTC

SUMMARY
A broken app can spam systemd's journal with too many messages.

STEPS TO REPRODUCE
1. install Google Chrome (129.0.6668.100)
2. start Chrome and open "chrome://flags" and set "Preferred Ozone platform" to "Auto"
3. open konsole with: $ journalctl -f
4. watch a web video in Chrome

OBSERVED RESULT
systemd's journal gets spammed 100 times per second with:

Okt 12 01:43:54 xxx google-chrome-stable[48687]: [48908:3:1012/014354.205523:ERROR:client_native_pixmap_dmabuf.cc(50)] Failed to mmap dmabuf: Invalid argument (22)
Okt 12 01:43:54 xxx google-chrome-stable[48687]: [48908:3:1012/014354.205660:ERROR:client_shared_image.cc(146)] Failed to map the buffer.

When I noticed this, I already had ~1,000,000 entries in the journal.

EXPECTED RESULT
Above a certain number of messages per second, an app should be blocked from spamming systemd's journal.

SOFTWARE/OS VERSIONS
Operating System: Kubuntu 24.10
KDE Plasma Version: 6.1.5
KDE Frameworks Version: 6.6.0
Qt Version: 6.6.2
Kernel Version: 6.11.0-8-generic (64-bit)
Graphics Platform: Wayland
Processors: 8 × Intel® Core™ i7-4790K CPU @ 4.00GHz
Memory: 15.6 GiB of RAM
Graphics Processor: NVIDIA GeForce GTX 960/PCIe/SSE2

ADDITIONAL INFORMATION
systemd per unit journal rate limiting:
https://github.com/systemd/systemd/pull/10308

Comment 1 Peter Eszlari 2024-10-13 17:08:58 UTC

https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html

From the documentation, the global defaults are:

RateLimitIntervalSec=30s
RateLimitBurst=10000 (+ disk space multiplier)

10000+ messages in 30s seems quite high to me, but I guess this isn't strictly a bug then.

Maybe systemd should add a warning limit setting and send a notification.

Comment 2 pallaswept 2024-10-24 09:18:51 UTC

(In reply to Peter Eszlari from comment #1)
> 10000+ messages in 30s seems quite high to me, but I guess this isn't strictly a bug then.

Hi Peter! 

I've been bitten by this issue quite a few times. A single errant app can render my entire journal un-usable, but of course, I won't know of it until much later when I actually need to read the journal, and effectively can't, because it's so big that it can take hours to open it. I hadn't had the time to chase that bug yet, so I'm glad you did, and I just stumbled across it. Until now, I've been deleting my logs and wondering what was in them.

Knowing that there is actually a rate limiter, I'm of the very strong opinion that it's set far, far too loosely by default, and I doubt very much that this is intentional behaviour, that normal usage can effectively break the journal entirely... so maybe it really is a bug.

I think this should be reported upstream, so they can decide if that's how they wanted it to act. I don't want to hijack your issue, so let me know if you'd rather do it, or if you'd rather not, and I'll do it :)

In the meantime, I'll modify the config and give you my thanks for looking into this!

Comment 3 Peter Eszlari 2024-10-25 00:55:23 UTC

(In reply to pallaswept from comment #2)
> I think this should be reported upstream, so they can decide if that's how
> they wanted it to act. I don't want to hijack your issue, so let me know if
> you'd rather do it, or if you'd rather not, and I'll do it :)

I already did: https://github.com/systemd/systemd/issues/34753

Comment 4 pallaswept 2024-10-25 01:31:46 UTC

(In reply to Peter Eszlari from comment #3)
> I already did: https://github.com/systemd/systemd/issues/34753

Nice one Peter. I'll follow up over there.

Thanks heaps for your help and for looking into this. Now I can knock one more item off my "I'll deal with that one later" list :)