| Summary: | Folder access times out on folders containing a high amount of files (> 10000) | ||
|---|---|---|---|
| Product: | [Frameworks and Libraries] kio-extras | Reporter: | Christoph Thielecke <crissi99> |
| Component: | MTP | Assignee: | Unassigned bugs mailing-list <unassigned-bugs> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | elvis.angelaccio, sitter |
| Priority: | NOR | ||
| Version: | 21.12.3 | ||
| Target Milestone: | --- | ||
| Platform: | Ubuntu | ||
| OS: | Linux | ||
| Latest Commit: | https://invent.kde.org/network/kio-extras/commit/ec022fb86947765b18cdccaae0853bb1c163d2f5 | Version Fixed In: | |
| Sentry Crash Report: | |||
|
Description
Christoph Thielecke
2022-11-20 12:57:20 UTC
Note the folder I tried to access contain ~26800 files. Can you still reproduce this with 22.12? 21.12 is no longer supported. I've tried listing some 14k files and that seems to succeed. (In reply to Harald Sitter from comment #2) > Can you still reproduce this with 22.12? 21.12 is no longer supported. This is strange, as 21.12 is used on Ubuntu Jammy (22.04LTS) and 22.08 is used on Ubuntu Kinetic (22.10). Both are not old. I'll try with Kubuntu Kinetic (22.10, uses 22.08) and KDE neon (neon-user-20221222-0714.iso) and report the feedback. > I've tried listing some 14k files and that seems to succeed. Have you tried with an amount of 50k? I have ~26800 files. Interesting is, that on Windows machine I saw that the file list will be dynamically updated. Maybe it would be interesting what commands are done there. 22.08 is also out of support. I haven't tried with more than 14k since I don't actually know how to synthesize that many files and transfering them over mtp takes ages because of how the protocol works. (In reply to Harald Sitter from comment #4) > 22.08 is also out of support. > > I haven't tried with more than 14k since I don't actually know how to > synthesize that many files and transfering them over mtp takes ages because > of how the protocol works. Use an external sd card and copy the file structure with an adapter on a local computer or use kde-connect. Found it!
> m_dbusInterface->setTimeout(std::chrono::milliseconds(5min).count()); // TODO: listing folders with a huge amount of files may take a while
What really needs to happen is rewriting the listing portion of the dbus interface. It currently runs a blocking list operation that is subject to timeouts. What should happen instead is that listing gives out a dbusobjectpath under which it will issue listing status signals. This would then also allow more atomic listing where entries are emitted as they arrive rather than having to load the entire file list into memory and only once that is done emitting the result all at once.
ListingObject {
Q_INVOKABLE void start();
Q_SIGNAL void file(file);
Q_SIGNAL void done();
Q_SIGNAL void error(msg);
}
(In reply to Harald Sitter from comment #6) > Found it! > > > m_dbusInterface->setTimeout(std::chrono::milliseconds(5min).count()); // TODO: listing folders with a huge amount of files may take a while > > What really needs to happen is rewriting the listing portion of the dbus > interface. It currently runs a blocking list operation that is subject to > timeouts. What should happen instead is that listing gives out a > dbusobjectpath under which it will issue listing status signals. This would > then also allow more atomic listing where entries are emitted as they arrive > rather than having to load the entire file list into memory and only once > that is done emitting the result all at once. > > ListingObject { > Q_INVOKABLE void start(); > Q_SIGNAL void file(file); > Q_SIGNAL void done(); > Q_SIGNAL void error(msg); > } Nice! Do you have a patch that I can test it? It would be nice to have a patch for 21.12.3 to, that would make things easy. (In reply to Christoph Thielecke from comment #3) > (In reply to Harald Sitter from comment #2) > > Can you still reproduce this with 22.12? 21.12 is no longer supported. > This is strange, as 21.12 is used on Ubuntu Jammy (22.04LTS) and 22.08 is > used on Ubuntu Kinetic (22.10). > > Both are not old. > > I'll try with Kubuntu Kinetic (22.10, uses 22.08) and KDE neon > (neon-user-20221222-0714.iso) and report the feedback. I tested both and they have same behavior. (In reply to Christoph Thielecke from comment #7) > Nice! > Do you have a patch that I can test it? Nope, that will be a fair amount of work. > It would be nice to have a patch for 21.12.3 to, that would make things easy. You can just patch that line I've referenced to use a larger value. That way it won't time out after 5 minutes. As it turns out we don't actually have the necessary facilities in libmtp. We'll need better handling there. I'll bump the timeout as a stop gap measure, but that will also not be enough given a large enough number of files :( Git commit 5821c30d1f6be18e0993f03db08ec41602e5dc82 by Harald Sitter. Committed on 24/02/2023 at 13:41. Pushed by sitter into branch 'release/22.12'. mtp: bump dbus timeout to 60 minutes to prevent timeouts when listing tens of thousands of files. (cherry picked from commit ec022fb86947765b18cdccaae0853bb1c163d2f5) M +4 -1 mtp/shared/kmtpstorageinterface.cpp https://invent.kde.org/network/kio-extras/commit/5821c30d1f6be18e0993f03db08ec41602e5dc82 Git commit ec022fb86947765b18cdccaae0853bb1c163d2f5 by Harald Sitter. Committed on 24/02/2023 at 13:41. Pushed by sitter into branch 'master'. mtp: bump dbus timeout to 60 minutes to prevent timeouts when listing tens of thousands of files. M +4 -1 mtp/shared/kmtpstorageinterface.cpp https://invent.kde.org/network/kio-extras/commit/ec022fb86947765b18cdccaae0853bb1c163d2f5 A possibly relevant merge request was started @ https://invent.kde.org/network/kio-extras/-/merge_requests/230 Git commit 87aeb0d9a61cd8a7e7041dc430395ed2cbf01c66 by Harald Sitter. Committed on 21/03/2023 at 15:03. Pushed by sitter into branch 'master'. mtp: use LIBMTP_Get_Children to implement async listing LIBMTP_Get_Children returns a list of object ids within a folder, which returns loads faster than the existing files listing API. once we have the list of ids we can asynchronously resolve the actual file entries for them and emit them as they appear. this allows us to enumerate entries as they appear rather than all at once. to facilitate this behavior a new function getFilesAndFolders2 was introduced that returns a qdbusobjectpath. under this path a new MTPLister object is registered that emits the listing. M +3 -0 CMakeLists.txt M +6 -0 mtp/CMakeLists.txt M +30 -2 mtp/kio_mtp.cpp M +2 -0 mtp/kiod_module/CMakeLists.txt A +20 -0 mtp/kiod_module/memory.h [License: LGPL(3+eV) LGPL(v3.0) LGPL(v2.1)] A +110 -0 mtp/kiod_module/mtpfile.h [License: GPL(v2.0+)] A +42 -0 mtp/kiod_module/mtplister.cpp [License: LGPL(3+eV) LGPL(v3.0) LGPL(v2.1)] A +36 -0 mtp/kiod_module/mtplister.h [License: LGPL(3+eV) LGPL(v3.0) LGPL(v2.1)] M +103 -139 mtp/kiod_module/mtpstorage.cpp M +17 -2 mtp/kiod_module/mtpstorage.h M +8 -0 mtp/shared/CMakeLists.txt A +4 -0 mtp/shared/config-mtp.h.cmake M +10 -0 mtp/shared/kmtpstorageinterface.cpp M +1 -0 mtp/shared/kmtpstorageinterface.h A +22 -0 mtp/shared/org.kde.kmtp.Lister.xml M +5 -0 mtp/shared/org.kde.kmtp.storage.xml https://invent.kde.org/network/kio-extras/commit/87aeb0d9a61cd8a7e7041dc430395ed2cbf01c66 |