Bug 513368

Summary: krdpserver memory leak during H.264 encoding without hardware acceleration
Product: [Plasma] KRdp Reporter: Jay Brame <bramej>
Component: generalAssignee: Unassigned bugs <unassigned-bugs-null>
Status: REPORTED ---    
Severity: major CC: ahiemstra, bramej, nate
Priority: NOR    
Version First Reported In: 6.5.4   
Target Milestone: ---   
Platform: Fedora RPMs   
OS: Linux   
Latest Commit: Version Fixed/Implemented In:
Sentry Crash Report:

Description Jay Brame 2025-12-14 17:07:24 UTC 
Description:
  krdpserver leaks memory rapidly during active RDP sessions when using software H.264 encoding (no VAAPI/GPU
  available).

  Environment:
  - Fedora 42 KDE Spin (VM without GPU passthrough)
  - krdp version 6.5.4-1.fc42
  - Client: KRDC (native KDE client) and FreeRDP-based clients

  Steps to reproduce:
  1. Run KDE Plasma in a VM without VAAPI/GPU
  2. Enable KRdp remote desktop
  3. Connect with any RDP client
  4. Monitor krdpserver memory usage

  Expected: Memory stays stable
  Actual: RSS grows ~4GB/minute during active streaming

  Observed memory growth:
  - Start: ~600MB
  - After 2 min: ~3.8GB
  - After 5 min: ~8GB+ (VM becomes unresponsive)

  The leak appears to be in the software H.264 encoding path. This was reproduced with multiple RDP clients
  (KRDC, xfreerdp3), confirming it's server-side.