SUMMARY RDP sessions that are not closed client side but stop receiving data cause OOM condition STEPS TO REPRODUCE 1. Connect to the kRDP server as normal 2. Allow the client's operating system to sleep without closing the RDP client session OBSERVED RESULT RAM utilization on the system will slowly increase until the system becomes unresponsive, requiring either manual intervention or being killed by oomkiller and the like. EXPECTED RESULT A timeout or buffer size monitor threshold is triggered when the server can not send data to the client after an appropriate length, closing the connection server-side. SOFTWARE/OS VERSIONS Client Side: Thincast Remote Desktop Client 1.1.634 stable (tcc-stable-1.1.634) OpenSSL OpenSSL 1.1.1w 11 Sep 2023 OрenH264 2.6.0 CJSON: 1.7.16 FreeRDP: 3.16.1-devO (3.16.1) WinPR: 3.16.1-devO (3.16.1) Qt: 5.15.15 macOS: 15.6 Server Side: Operating System: CachyOS Linux KDE Plasma Version: 6.4.3 KDE Frameworks Version: 6.16.0 Qt Version: 6.9.1 Kernel Version: 6.16.0-5-cachyos (64-bit) Graphics Platform: Wayland Processors: 16 × AMD Ryzen 7 9800X3D 8-Core Processor Memory: 64 GiB of RAM (60.4 GiB usable) Graphics Processor 1: AMD Radeon RX 6800 XT Graphics Processor 2: AMD Radeon Graphics ADDITIONAL INFORMATION I believe this to be caused by either krdpserver buffering video, or the network socket buffering the data to be sent to the client. I'm in the process of collecting additional data about this should it be valuable. I have not been able to test this with other clients as this particular client is the only one I currently have that supports h.264 enabled clients appropriately. I have however been able to reproduce the issue several times successfully. It is easier to reproduce this when the server side is producing complex video data to send over the wire. In my case, this is usually triggered when I forget to close an RDP client session before switching to the RDP server and playing a game. I imagine watching a video would also produce sufficiently complex data to be encoded and trigger the condition.
Created attachment 184002 [details] pmap of krdpserver process while experiencing the high memory event Output of other memory diagnostic commands against krdpserver during a high memory event. Repro steps are identical to what was reported in the original bug submission. ❯ ps aux | grep krdpserver | grep -v grep belthes+ 51830 0.0 2.7 57349244 1754824 ? SLsl Aug07 6:17 /usr/bin/krdpserver ❯ grep -i vmrss /proc/51830/status VmRSS: 1525944 kB ❯ grep -e '^Rss' /proc/51830/smaps | awk '{sum += $2} END {print sum, "kB"}' 1355268 kB total used free shared buff/cache available Mem: 61885 56668 543 154 5520 5216 Swap: 61884 56238 5646 After stopping the session via the tray icon: total used free shared buff/cache available Mem: 61885 52626 4579 179 5552 9259 Swap: 61884 53285 8599 After restarting krdpserver: ❯ ps aux | grep krdpserver | grep -v grep belthes+ 51830 0.0 0.4 57180280 274420 ? SLsl Aug07 6:22 /usr/bin/krdpserver ❯ grep -i vmrss /proc/51830/status VmRSS: 271936 kB ❯ grep -e '^Rss' /proc/51830/smaps | awk '{sum += $2} END {print sum, "kB"}' 271936 kB Log output: ❯ journalctl --user -f -u app-org.kde.krdpserver Aug 12 17:43:50 fate krdpserver[51830]: QObject::killTimer: Timers cannot be stopped from another thread Aug 12 17:43:50 fate krdpserver[51830]: QObject::startTimer: Timers cannot be started from another thread Aug 12 18:29:32 fate krdpserver[51830]: org.kde.krdp: Portal session was closed! Aug 12 18:29:33 fate krdpserver[51830]: [18:29:33:610] [51830:000497f9] [ERROR][com.freerdp.core.transport] - [transport_default_write]: BIO_should_retry returned an error: error:80000020:system library::Broken pipe Aug 12 18:29:33 fate krdpserver[51830]: [18:29:33:610] [51830:000497f9] [ERROR][com.freerdp.core.peer] - [transport_default_write]: ERRCONNECT_CONNECT_TRANSPORT_FAILED [0x0002000D] Aug 12 18:29:33 fate krdpserver[51830]: org.kde.krdp: Unable to check Virtual Channel Manager file descriptor, closing connection Aug 12 18:29:33 fate krdpserver[51830]: org.kde.krdp: Closing session Aug 12 18:29:33 fate krdpserver[51830]: [18:29:33:634] [51830:0000ca76] [ERROR][com.freerdp.core.transport] - [transport_default_write]: BIO_should_retry returned an error: error:0A00010F:SSL routines::bad length Aug 12 18:29:33 fate krdpserver[51830]: org.kde.krdp: Closing Freedesktop Portal Session Aug 12 18:29:33 fate krdpserver[51830]: QObject::~QObject: Timers cannot be stopped from another thread After restarting krdpserver: ❯ free -m total used free shared buff/cache available Mem: 61885 12837 44389 179 5531 49047 Swap: 61884 12696 49188