Bug 474231 - "Too many open files" when trying to update a large number of refs
Summary: "Too many open files" when trying to update a large number of refs
Status: RESOLVED FIXED
Alias: None
Product: Discover
Classification: Applications
Component: Flatpak Backend (show other bugs)
Version: 5.27.5
Platform: Arch Linux Linux
: NOR normal
Target Milestone: ---
Assignee: Plasma Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-09-06 16:16 UTC by Alberto Garcia
Modified: 2023-11-09 20:18 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.27.9


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alberto Garcia 2023-09-06 16:16:41 UTC
STEPS TO REPRODUCE
1. Have a Flatpak installation with ~50 pending updates
2. Open Discover, select "Update" and click "Update all"

OBSERVED RESULT
The application crashes. The exact stack trace and journal messages are different each time, but they are all variations of "Too many open files":

plasma-discover[3235]: Creating pipes for GWakeup: Too many open files
plasma-discover[4090]: Error spawning revokefs-fuse: Too many open files
plasma-discover[3518]: Cannot create repo on revokefs mountpoint /var/tmp/flatpak-cache-T1LSA2/org.gnome.Platform.Locale-SQUKA2: opening repo: While checking parent repository '/var/lib/flatpak/repo': opening repo: opendir(tmp): Too many open files

There is also a large number of /usr/lib/revokefs-fuse processes running.

EXPECTED RESULT
All refs are updated correctly.

SOFTWARE/OS VERSIONS
KDE Plasma Version: 5.27.5
KDE Frameworks Version: 5.107.0
Qt Version: 5.19.5

ADDITIONAL INFORMATION
Having a look at the discover Flatpak backend I see that FlatpakBackend::installApplication() is called in a loop for every application that needs to be updated:

https://invent.kde.org/plasma/discover/-/blob/v5.27.5/libdiscover/resources/StandardBackendUpdater.cpp?ref_type=tags#L65

Each one of these creates a new FlatpakJobTransaction, each with its own FlatpakTransactionThread which in turn has its own FlatpakTransaction object:

https://invent.kde.org/plasma/discover/-/blob/v5.27.5/libdiscover/backends/FlatpakBackend/FlatpakTransactionThread.cpp?ref_type=tags#L101

In other words, instead of creating one FlatpakTransaction and putting all refs in it it creates one transaction per ref and (as far as I can see) runs them all in parallel. I haven't confirmed it yet but I suspect that the problem is related to this.
Comment 1 Bug Janitor Service 2023-09-11 07:47:42 UTC
A possibly relevant merge request was started @ https://invent.kde.org/plasma/discover/-/merge_requests/630
Comment 2 Harald Sitter 2023-09-13 07:14:12 UTC
Git commit db0ebc855517f189f64c1602a5d27e185cf02833 by Harald Sitter.
Committed on 13/09/2023 at 09:13.
Pushed by sitter into branch 'master'.

flatpak: make FlatpakTransactionThread a qrunnable instead

we can't just have an unlimited number of threads for flatpak
transactions. it'd eventually cause excessive load on both CPU and
network to the point where things will start misbehaving. we also run
risk of exhausting other software limited resources such as file
descriptors.

to resolve this problem we now treat the transactionthread as runnable
and put it in a limited threadpool for concurrent execution. the new
runnable has a finished signal that is emitted on every return from
run() to match the QThread API.

concurrency is limited to no more than 4 runnables at a time. that
should still be plenty concurrent while generally unexpected to exhaust
the default 1024 file descriptor limit - an install transaction appears
to weigh between 60 and 100 fds

other backends don't necessarily have this problem since they have
daemons that do the work for us so we have way fewer open fds for them.

M  +30   -5    libdiscover/backends/FlatpakBackend/FlatpakJobTransaction.cpp
M  +6    -2    libdiscover/backends/FlatpakBackend/FlatpakTransactionThread.cpp
M  +4    -1    libdiscover/backends/FlatpakBackend/FlatpakTransactionThread.h

https://invent.kde.org/plasma/discover/-/commit/db0ebc855517f189f64c1602a5d27e185cf02833
Comment 3 Harald Sitter 2023-09-13 07:20:40 UTC
Git commit 46d14515c3105e4e318d28db41057d9f1df3ce4d by Harald Sitter.
Committed on 13/09/2023 at 09:19.
Pushed by sitter into branch 'Plasma/5.27'.

flatpak: make FlatpakTransactionThread a qrunnable instead

we can't just have an unlimited number of threads for flatpak
transactions. it'd eventually cause excessive load on both CPU and
network to the point where things will start misbehaving. we also run
risk of exhausting other software limited resources such as file
descriptors.

to resolve this problem we now treat the transactionthread as runnable
and put it in a limited threadpool for concurrent execution. the new
runnable has a finished signal that is emitted on every return from
run() to match the QThread API.

concurrency is limited to no more than 4 runnables at a time. that
should still be plenty concurrent while generally unexpected to exhaust
the default 1024 file descriptor limit - an install transaction appears
to weigh between 60 and 100 fds

other backends don't necessarily have this problem since they have
daemons that do the work for us so we have way fewer open fds for them.

(cherry picked from commit db0ebc855517f189f64c1602a5d27e185cf02833)

M  +29   -5    libdiscover/backends/FlatpakBackend/FlatpakJobTransaction.cpp
M  +7    -2    libdiscover/backends/FlatpakBackend/FlatpakTransactionThread.cpp
M  +4    -1    libdiscover/backends/FlatpakBackend/FlatpakTransactionThread.h

https://invent.kde.org/plasma/discover/-/commit/46d14515c3105e4e318d28db41057d9f1df3ce4d