SUMMARY *** Tested on Windows 10 / 11 (a similar issue was reported on Linux AppImage, but I was not able to personally reproduce on linux) Creating a KDirOperator, and setting its url to a folder with more than 178 files (that number might depend on your system), KDirOperator stops working, the files are not listed. This only happens on KF6. I tried to investigate a bit, this seems related to the connectionbackend.cpp file: https://invent.kde.org/frameworks/kio/-/blob/master/src/core/connectionbackend.cpp?ref_type=heads#L268 When we hit this condition (jumbo packet), commandReceived(task) is never emitted. On my test system (Windows 11), this condition is hit when I have 178 or more files in a folder. This code didn't change in 10 years, so not sure why I only ht this bug now. On linux, even when parsing a folder with > 600 files, I never hit that condition for jumbo packets. *** STEPS TO REPRODUCE 1. Open a Windows KF6 KDE App using KDirOperator (in my case only tested with Kdenlive): https://files.kde.org/kdenlive/unstable/kdenlive-24.02.0-RC_A_-x86_64.exe.mirrorlist 2. Open the "Project" menu, select "Add clip or folder...", this opens a file dialog using a KDirOperator 3. Browse to a folder with like 200 files OBSERVED RESULT Dialog is freezing EXPECTED RESULT Dialog listing the files SOFTWARE/OS VERSIONS Windows: 11, package built with Craft KDE Frameworks Version: 5.248.0 Qt Version: 6.6.1 ADDITIONAL INFORMATION I am available if more infos are needed
For the record, I can reproduce the bug on Windows with a very basic Qt Application embedding a KFileWidget: https://invent.kde.org/multimedia/kdenlive/-/snippets/3016 Also, the following patch - although I am not really sure if it is a proper fix - fixes the problem and allows browsing a folder with > 800 files without problem. _________________________________________________ diff --git a/src/core/connectionbackend_p.h b/src/core/connectionbackend_p.h index 39c3504c4..ae86eaaa5 100644 --- a/src/core/connectionbackend_p.h +++ b/src/core/connectionbackend_p.h @@ -40,7 +40,7 @@ private: bool signalEmitted; static const int HeaderSize = 10; - static const int StandardBufferSize = 32 * 1024; + static const int StandardBufferSize = 64 * 1024; Q_SIGNALS: void disconnected(); _________________________________________________
Also I wonder if it could be related to the Qt6 container type size change as mentioned in: https://invent.kde.org/frameworks/kio/-/merge_requests/1141#note_606633
The bug can also easily be reproduced using the KIO test kdirlister_gui: https://invent.kde.org/frameworks/kio/-/blob/master/tests/kdirlistertest_gui.cpp?ref_type=heads This test opens a window and proposes to browse a few folders by clicking a button, between "Home, Root, ...". Just put like 300 files in your home folder, click the "home" button and it will freeze. The number of files required to reproduce the freeze seems to vary between 140 and 300 Debug output: [9712] ::: CREATING LISTJOB WITH PREFIX: "" [9712] :::KIO ::: COONNECTING JOB... [9712] !!!!!! KIO CONNECTING ENTRIES.... [9712] :::: CONNECTION SEND COMMD: 85 , "\x00\x00\x00\x00" [9712] :::: CONNECTION SEND COMMD: 48 , "\xFF\xFF\xFF\xFF\x00\x00\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" [9712] ::: KIO :::: STARTING LIST JOB [9712] :::: CONNECTION SEND COMMD: 71 , "\x00\x00\x00\x14""file:///C:/Users/XXX" [9712] :::: SENT COMMAND: 85 , SUCC: true [9712] XXXXXXXXXXXXXXX READ SOCKET START XXXXXXXXXXXXXXXXXXXXXXXXX [9712] ::: KIO SOCKET Got 14 bytes [9712] :::: SENT COMMAND: 48 , SUCC: true [9712] QLocalSocket(0x151b46d8ca0) Want to read 4 bytes , AVAIL: 4 [9712] :::: SENT COMMAND: 71 , SUCC: true [9712] :::: KIO COMMAND EMITED [9712] AAAAAAAAAAA WORKER ACCEPTED AAAAAAAAAAAAAAAAa [9712] ::::COMMAND RCVD A [9712] 0x151b90ab120 Command 85 added to the queue [9712] KIO COMPARING TO HEADER: 0 >= 10 [9712] :::: SHOULD READ ANOTHER: false [9712] XXXXXXXXXXXXXXX READ SOCKET DONE XXXXXXXXXXXXXXXXXXXXXXXXX [9712] XXXXXXXXXXXXXXX READ SOCKET START XXXXXXXXXXXXXXXXXXXXXXXXX [9712] ::: KIO SOCKET Got 58 bytes [9712] QLocalSocket(0x151b46d8ca0) Want to read 14 bytes , AVAIL: 48 [9712] XXXXXXXXXXXXXXX READ SOCKET START XXXXXXXXXXXXXXXXXXXXXXXXX [9712] :::: KIO COMMAND EMITED [9712] ::: KIO SOCKET Got 0 bytes [9712] ::::COMMAND RCVD A [9712] ::: KIO ::: WAIT FOR MORE DATA... [9712] 0x151b90ab120 Command 48 added to the queue [9712] KIO COMPARING TO HEADER: 34 >= 10 [9712] :::: SHOULD READ ANOTHER: true [9712] ::: KIO SOCKET Got 34 bytes [9712] QLocalSocket(0x151b46d8ca0) Want to read 24 bytes , AVAIL: 24 [9712] :::: KIO COMMAND EMITED [9712] ::::COMMAND RCVD A [9712] 0x151b90ab120 Command 71 added to the queue [9712] KIO COMPARING TO HEADER: 0 >= 10 [9712] :::: SHOULD READ ANOTHER: false [9712] XXXXXXXXXXXXXXX READ SOCKET DONE XXXXXXXXXXXXXXXXXXXXXXXXX [9712] :::: CONNECTION SEND COMMD: 106 , "\x00\x00\x00\x0B\x01\x00\x00\x06\x00\x00\x00\x02\x00.\x02\x00\x00\r\x00\x00\x00\x00\x00\x00@\x00\x02\x00\x00\t\x00\x00\x00\x00\x00\x00\x01\xC0\x02\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\n\x00\x00\x00\x00""e\xDA\xE4r\x01\x00\x00\x03\x00\x00\x00\x00\x01\x00\x00\x05\x00\x00\x00\x00\x06\x00\x00\x0B\x00\x00\x00\x00""e\xDA\xE4r\x06\x00\x00\f\x00\x00\x00\x00""d|r\xA4\x02\x00\x00\x1B\x00\x00\x00\x00\x00\x00\x00\x02\x02\x00\x00\x1C\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0B\x01\x00\x0 (...removed like 60 similar lines...) 6\x02\x00\x00\x1B\x00\x00\x00\x00\x00\x00\x00\x02\x02\x00\x00\x1C\x00\x00\x00\x00\x00\x00\x00\x00" [9712] XXXXXXXXXXXXXXX READ SOCKET START XXXXXXXXXXXXXXXXXXXXXXXXX [9712] ::: KIO SOCKET Got 32768 bytes [9712] QLocalSocket(0x151b90d27e0) Want to read 33194 bytes , AVAIL: 32758 [9712] :::: KIO Jumbo packet of 33194 bytes [9712] :::: KIO ADJUSTED READ BUFF SIZE: 33195 == 32758 [9712] KIO COMPARING: 32758 >= 33194 [9712] :::: SHOULD READ ANOTHER: false [9712] XXXXXXXXXXXXXXX READ SOCKET DONE XXXXXXXXXXXXXXXXXXXXXXXXX [9712] XXXXXXXXXXXXXXX READ SOCKET START XXXXXXXXXXXXXXXXXXXXXXXXX [9712] ::: KIO SOCKET Got 32768 bytes [9712] QLocalSocket(0x151b90d27e0) Want to read 33194 bytes , AVAIL: 32768 [9712] :::: KIO Jumbo packet of 33194 bytes [9712] :::: KIO ADJUSTED READ BUFF SIZE: 33195 == 32768 [9712] KIO COMPARING: 32768 >= 33194 [9712] :::: SHOULD READ ANOTHER: false [9712] XXXXXXXXXXXXXXX READ SOCKET DONE XXXXXXXXXXXXXXXXXXXXXXXXX
Sorry, never really worked on or reasoned in detail about this part of code, only touched it for simple changes, also no time currently. This needs someone active here, with good KIO overview (cmp. https://invent.kde.org/frameworks/kio/-/commits/master for candidates), so have to unsubscribe me again.
For me craft wants to patch kiconthemes and fails so I can't really build things to look at. I am rather guessing that the problem is with the file worker sending too large packets though. Jumbo packet handling is very unreliable in my experience (there may well be a bug in it, but I haven't had the motivation to debug this). I'd argue the packet should be broken up into smaller pieces anyway what with reducing memory pressure. First task I guess is finding where the jumbo packet comes from exactly.
A possibly relevant merge request was started @ https://invent.kde.org/frameworks/kio/-/merge_requests/1563
As immediate hotfix you could probably patch KIO_MAX_ENTRIES_PER_BATCH to a much lower value - creating more smaller packets instead of fewer large ones.
Thanks a lot Harald, fantastic !! I can confirm that your MR fixes the issue in my tests on Windows. Lowering KIO_MAX_ENTRIES_PER_BATCH to 50 as a hotfix also allows browsing larger directories.
Git commit 4ded5dd12f5e752674b685833b77b9b546c0b6c8 by Méven Car, on behalf of Harald Sitter. Committed on 15/03/2024 at 18:12. Pushed by meven into branch 'master'. connectionbackend: improve jumbo packet handling (also should fix things on windows) jumbo handling was a bit weird in that we kept bumping the buffer size sky high. and then eventually (hopefully) would be able to read everything in one go and emit. In practice that seems to have not worked out on windows. do partial reads instead. to facilitate this change the existing Task struct is now serving as container we read into so it becomes a bit easier to manage the state. this should also make the code a bit easier to reason about. when we have no task we now start a new task. if we have a task we always read as much data as possible into the task. when we read all data then the task gets emitted. M +9 -0 autotests/CMakeLists.txt A +53 -0 autotests/connectionbackendtest.cpp [License: LGPL(v2.0+)] M +9 -0 src/core/CMakeLists.txt M +22 -25 src/core/connectionbackend.cpp M +7 -6 src/core/connectionbackend_p.h https://invent.kde.org/frameworks/kio/-/commit/4ded5dd12f5e752674b685833b77b9b546c0b6c8