| Summary: | "Too many open files" when trying to update a large number of refs | ||
|---|---|---|---|
| Product: | [Applications] Discover | Reporter: | Alberto Garcia <berto> |
| Component: | Flatpak Backend | Assignee: | Plasma Bugs List <plasma-bugs> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | aleixpol, jgrulich, nate, travier |
| Priority: | NOR | ||
| Version: | 5.27.5 | ||
| Target Milestone: | --- | ||
| Platform: | Arch Linux | ||
| OS: | Linux | ||
| See Also: |
https://bugs.kde.org/show_bug.cgi?id=474135 https://bugs.kde.org/show_bug.cgi?id=476726 |
||
| Latest Commit: | https://invent.kde.org/plasma/discover/-/commit/46d14515c3105e4e318d28db41057d9f1df3ce4d | Version Fixed In: | 5.27.9 |
|
Description
Alberto Garcia
2023-09-06 16:16:41 UTC
A possibly relevant merge request was started @ https://invent.kde.org/plasma/discover/-/merge_requests/630 Git commit db0ebc855517f189f64c1602a5d27e185cf02833 by Harald Sitter. Committed on 13/09/2023 at 09:13. Pushed by sitter into branch 'master'. flatpak: make FlatpakTransactionThread a qrunnable instead we can't just have an unlimited number of threads for flatpak transactions. it'd eventually cause excessive load on both CPU and network to the point where things will start misbehaving. we also run risk of exhausting other software limited resources such as file descriptors. to resolve this problem we now treat the transactionthread as runnable and put it in a limited threadpool for concurrent execution. the new runnable has a finished signal that is emitted on every return from run() to match the QThread API. concurrency is limited to no more than 4 runnables at a time. that should still be plenty concurrent while generally unexpected to exhaust the default 1024 file descriptor limit - an install transaction appears to weigh between 60 and 100 fds other backends don't necessarily have this problem since they have daemons that do the work for us so we have way fewer open fds for them. M +30 -5 libdiscover/backends/FlatpakBackend/FlatpakJobTransaction.cpp M +6 -2 libdiscover/backends/FlatpakBackend/FlatpakTransactionThread.cpp M +4 -1 libdiscover/backends/FlatpakBackend/FlatpakTransactionThread.h https://invent.kde.org/plasma/discover/-/commit/db0ebc855517f189f64c1602a5d27e185cf02833 Git commit 46d14515c3105e4e318d28db41057d9f1df3ce4d by Harald Sitter. Committed on 13/09/2023 at 09:19. Pushed by sitter into branch 'Plasma/5.27'. flatpak: make FlatpakTransactionThread a qrunnable instead we can't just have an unlimited number of threads for flatpak transactions. it'd eventually cause excessive load on both CPU and network to the point where things will start misbehaving. we also run risk of exhausting other software limited resources such as file descriptors. to resolve this problem we now treat the transactionthread as runnable and put it in a limited threadpool for concurrent execution. the new runnable has a finished signal that is emitted on every return from run() to match the QThread API. concurrency is limited to no more than 4 runnables at a time. that should still be plenty concurrent while generally unexpected to exhaust the default 1024 file descriptor limit - an install transaction appears to weigh between 60 and 100 fds other backends don't necessarily have this problem since they have daemons that do the work for us so we have way fewer open fds for them. (cherry picked from commit db0ebc855517f189f64c1602a5d27e185cf02833) M +29 -5 libdiscover/backends/FlatpakBackend/FlatpakJobTransaction.cpp M +7 -2 libdiscover/backends/FlatpakBackend/FlatpakTransactionThread.cpp M +4 -1 libdiscover/backends/FlatpakBackend/FlatpakTransactionThread.h https://invent.kde.org/plasma/discover/-/commit/46d14515c3105e4e318d28db41057d9f1df3ce4d |