Summary: | KDirWatch's useFam loops indefinitely (actual fam, not gamin) | ||
---|---|---|---|
Product: | [Frameworks and Libraries] frameworks-kcoreaddons | Reporter: | boospy <linux> |
Component: | general | Assignee: | Michael Pyne <mpyne> |
Status: | RESOLVED FIXED | ||
Severity: | major | CC: | germano.massullo, kdelibs-bugs, kfm-devel, ricardo.jpg, sitter, spam |
Priority: | NOR | ||
Version: | unspecified | ||
Target Milestone: | --- | ||
Platform: | Neon | ||
OS: | Linux | ||
See Also: |
https://bugs.kde.org/show_bug.cgi?id=423928 https://bugs.kde.org/show_bug.cgi?id=423931 |
||
Latest Commit: | https://invent.kde.org/frameworks/kcoreaddons/commit/8599c4d58b82ad0096d59a4a23e54d75b5c7f558 | Version Fixed In: | 5.74 |
Description
boospy
2020-07-03 01:11:53 UTC
Ok, it look likes not depending really on open files. So this day dolphin screamed already on over 300.000 files. Really bad Bug. Only a Reboot solved the issue. I found another thing. This entry in sysct.conf is necessary. I have tested it many times too. Look likes stable now. fs.inotify.max_user_instances = 1024 For future reference we only can deal with bug reports in english other languages are lost on the majority of developers. It'd be grand if you could run this script and post its output: https://raw.githubusercontent.com/fatso83/dotfiles/master/utils/scripts/inotify-consumers Oh yes sorry for "no english". That happens to me sometimes when I switch between languages. Often you don't even notice what language is currently being written. The brain only understands the meaning, not the language :) :) Here the Output. INOTIFY WATCHER COUNT PID CMD ---------------------------------------- 645 19889 /usr/bin/baloo_file 438 19938 /usr/bin/nextcloud 200 19961 /usr/bin/strawberry 154 19938 /usr/bin/nextcloud 72 1 /sbin/init splash 61 19838 /usr/bin/kded5 49 25115 /usr/libexec/xdg-desktop-portal 19 580 /lib/systemd/systemd-udevd 15 3120 /usr/bin/krunner 14 1307 /usr/lib/policykit-1/polkitd --no-debug 13 22665 dolphin [kdeinit5] 13 19898 /usr/bin/plasmashell 9 31384 /opt/google/chrome/chrome 9 19691 /lib/systemd/systemd --user 8 20217 /usr/bin/akonadi_maildir_resource --identifier akonadi_maildir_resource_8 5 1 /sbin/init splash 5 1 /sbin/init splash 5 1 /sbin/init splash 5 19999 kopano-deskapp --eml= 5 1241 /usr/sbin/NetworkManager --no-daemon 4 8801 ./steamwebhelper -lang=de_DE -cachedir=/home/loma/.steam/config/htmlcache -steampid=8790 -buildid=1591251555 -steamid=0 4 5016 /opt/Hamsket/hamsket 4 1 /sbin/init splash 4 1 /sbin/init splash 4 1 /sbin/init splash 4 19867 /usr/lib/x86_64-linux-gnu/libexec/kactivitymanagerd 4 1403 /usr/lib/colord/colord 4 1307 /usr/lib/policykit-1/polkitd --no-debug 4 1241 /usr/sbin/NetworkManager --no-daemon 3 8846 ./steamwebhelper --type=utility --field-trial-handle=4282113078734543613,18238172750589149565,131072 --disable-features= 3 5057 /opt/Hamsket/hamsket --type=utility --field-trial-handle=5119176024368400076,7026167398216092141,131072 --enable-feature 3 31422 /opt/google/chrome/chrome --type=utility --field-trial-handle=2263038877973320156,6923300084768939570,131072 --lang=de - 3 24345 /usr/bin/kate -b --tempfile 3 20015 /usr/lib/packagekit/packagekitd 3 19931 /usr/bin/openfortigui 3 19842 /usr/bin/kwin_x11 3 1202 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only 2 7504 avahi-daemon: running [darkbox.osit.cc] 2 3120 /usr/bin/krunner 2 24345 /usr/bin/kate -b --tempfile 2 22665 dolphin [kdeinit5] 2 19979 /usr/bin/krb5-auth-dialog 2 19965 /usr/lib/x86_64-linux-gnu/libexec/org_kde_powerdevil 2 19958 /usr/bin/smb4k 2 19922 /usr/lib/x86_64-linux-gnu/libexec/kdeconnectd 2 19898 /usr/bin/plasmashell 2 19889 /usr/bin/baloo_file 2 19764 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only 2 1388 /usr/lib/accountsservice/accounts-daemon 2 1179 /usr/lib/udisks2/udisksd 1 9482 /usr/lib/firefox/firefox 1 5016 /opt/Hamsket/hamsket 1 31384 /opt/google/chrome/chrome 1 27714 /opt/zoom/zoom 1 27714 /opt/zoom/zoom 1 25091 /usr/lib/gvfs/gvfs-udisks2-volume-monitor 1 20216 /usr/bin/akonadi_maildir_resource --identifier akonadi_maildir_resource_7 1 20215 /usr/bin/akonadi_maildir_resource --identifier akonadi_maildir_resource_6 1 20214 /usr/bin/akonadi_maildir_resource --identifier akonadi_maildir_resource_5 1 20213 /usr/bin/akonadi_maildir_resource --identifier akonadi_maildir_resource_4 1 20210 /usr/bin/akonadi_maildir_resource --identifier akonadi_maildir_resource_3 1 20209 /usr/bin/akonadi_maildir_resource --identifier akonadi_maildir_resource_2 1 20208 /usr/bin/akonadi_maildir_resource --identifier akonadi_maildir_resource_1 1 20207 /usr/bin/akonadi_maildir_resource --identifier akonadi_maildir_resource_0 1 20196 /usr/bin/akonadi_akonotes_resource --identifier akonadi_akonotes_resource_0 1 20186 /usr/bin/ksysguardd 1 1 /sbin/init splash 1 19999 kopano-deskapp --eml= 1 19961 /usr/bin/strawberry 1 19958 /usr/bin/smb4k 1 19953 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3 1 19941 /usr/bin/keepassxc 1 19924 /usr/bin/gmenudbusmenuproxy 1 19922 /usr/lib/x86_64-linux-gnu/libexec/kdeconnectd 1 19917 /usr/lib/x86_64-linux-gnu/libexec/DiscoverNotifier 1 19912 /usr/bin/pulseaudio --start --log-target=syslog 1 19912 /usr/bin/pulseaudio --start --log-target=syslog 1 19912 /usr/bin/pulseaudio --start --log-target=syslog 1 19842 /usr/bin/kwin_x11 1 19726 /usr/bin/kwalletd5 --pam-login 9 3 1 19706 /usr/bin/appimagelauncherd 1 19691 /lib/systemd/systemd --user 1 1257 /usr/sbin/acpid 1 1191 /usr/sbin/sssd -i --logger=files 1 1179 /usr/lib/udisks2/udisksd 1 1137 /lib/systemd/systemd-timesyncd 1 1135 /lib/systemd/systemd-resolved 0 9049 /usr/bin/choqok -qwindowtitle Choqok 0 8801 ./steamwebhelper -lang=de_DE -cachedir=/home/loma/.steam/config/htmlcache -steampid=8790 -buildid=1591251555 -steamid=0 0 8801 ./steamwebhelper -lang=de_DE -cachedir=/home/loma/.steam/config/htmlcache -steampid=8790 -buildid=1591251555 -steamid=0 0 5016 /opt/Hamsket/hamsket 0 5016 /opt/Hamsket/hamsket 0 5016 /opt/Hamsket/hamsket 0 5016 /opt/Hamsket/hamsket 0 5016 /opt/Hamsket/hamsket 0 5016 /opt/Hamsket/hamsket 0 5016 /opt/Hamsket/hamsket 0 5016 /opt/Hamsket/hamsket 0 31384 /opt/google/chrome/chrome 0 31384 /opt/google/chrome/chrome 0 31384 /opt/google/chrome/chrome 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 3120 /usr/bin/krunner 0 24345 /usr/bin/kate -b --tempfile 0 22749 /usr/bin/ktorrent 0 20202 /usr/bin/akonadi_ical_resource --identifier akonadi_ical_resource_0 0 20201 /usr/bin/akonadi_davgroupware_resource --identifier akonadi_davgroupware_resource_3 0 20200 /usr/bin/akonadi_davgroupware_resource --identifier akonadi_davgroupware_resource_2 0 20199 /usr/bin/akonadi_davgroupware_resource --identifier akonadi_davgroupware_resource_0 0 19999 kopano-deskapp --eml= 0 19999 kopano-deskapp --eml= 0 19965 /usr/lib/x86_64-linux-gnu/libexec/org_kde_powerdevil 0 19941 /usr/bin/keepassxc 0 19917 /usr/lib/x86_64-linux-gnu/libexec/DiscoverNotifier 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19898 /usr/bin/plasmashell 0 19838 /usr/bin/kded5 0 19838 /usr/bin/kded5 0 19838 /usr/bin/kded5 0 19838 /usr/bin/kded5 0 19838 /usr/bin/kded5 0 19838 /usr/bin/kded5 0 19838 /usr/bin/kded5 0 19838 /usr/bin/kded5 0 19823 /usr/lib/x86_64-linux-gnu/libexec/kf5/klauncher --fd=9 sudo ./inotify-consumers 1,97s user 3,07s system 58% cpu 8,623 total Please lower your limit again to trigger the applications not opening then run - gdb --args dolphin --new-window that will give you a gdb prompt. In there type - run wait 10 or so seconds so it's stuck. Then hit ctrl+c and type - backtrace paste the output of that here and use - quit to quit gdb again. I do it that way, but it could take a time, it doesn't always react immediately. Should occur several times within this week. Well, you can just set a reasonably low limit and it'll trigger faster ;) Here is the output: Starting program: /usr/bin/dolphin --new-window [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7fffdf50c700 (LWP 26207)] [New Thread 0x7fffd4b3e700 (LWP 26208)] ^C Thread 1 "dolphin" received signal SIGINT, Interrupt. 0x00007ffff77d69d0 in __GI___nanosleep (requested_time=0x7fffffffc870, remaining=0x7fffffffc870) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 28 ../sysdeps/unix/sysv/linux/nanosleep.c: Datei oder Verzeichnis nicht gefunden. (gdb) backtrace #0 0x00007ffff77d69d0 in __GI___nanosleep (requested_time=0x7fffffffc870, remaining=0x7fffffffc870) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 #1 0x00007ffff13b012d in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 #2 0x00007ffff30c99f8 in ?? () from /usr/lib/x86_64-linux-gnu/libKF5CoreAddons.so.5 #3 0x00007ffff30c9e6b in ?? () from /usr/lib/x86_64-linux-gnu/libKF5CoreAddons.so.5 #4 0x00007ffff30ca555 in ?? () from /usr/lib/x86_64-linux-gnu/libKF5CoreAddons.so.5 #5 0x00007ffff506f6f5 in KSycocaPrivate::findDatabase() () from /usr/lib/x86_64-linux-gnu/libKF5Service.so.5 #6 0x00007ffff506f7f3 in KSycoca::connectNotify(QMetaMethod const&) () from /usr/lib/x86_64-linux-gnu/libKF5Service.so.5 #7 0x00007ffff1382d8a in QObjectPrivate::connectImpl(QObject const*, int, QObject const*, void**, QtPrivate::QSlotObjectBase*, Qt::ConnectionType, int const*, QMetaObject const*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 #8 0x00007ffff13830ed in QObject::connectImpl(QObject const*, void**, QObject const*, void**, QtPrivate::QSlotObjectBase*, Qt::ConnectionType, int const*, QMetaObject const*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 #9 0x00007ffff7b3a0b9 in ?? () from /usr/lib/x86_64-linux-gnu/libkdeinit5_dolphin.so #10 0x00007ffff7b2aa2f in kdemain () from /usr/lib/x86_64-linux-gnu/libkdeinit5_dolphin.so #11 0x00007ffff7713b97 in __libc_start_main (main=0x555555554720, argc=2, argv=0x7fffffffd128, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd118) at ../csu/libc-start.c:310 #12 0x000055555555475a in _start () Please install libkf5coreaddons5-dbgsym and libqt5core5a-dbgsym, then try again. Here is the new output after installing these packages. But look like similar. Starting program: /usr/bin/dolphin --new-window [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7fffdf50c700 (LWP 32235)] [New Thread 0x7fffd4b3e700 (LWP 32237)] ^C Thread 1 "dolphin" received signal SIGINT, Interrupt. 0x00007ffff77d69d0 in __GI___nanosleep (requested_time=requested_time@entry=0x7fffffffc870, remaining=remaining@entry=0x7fffffffc870) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 28 ../sysdeps/unix/sysv/linux/nanosleep.c: Datei oder Verzeichnis nicht gefunden. (gdb) backtrace #0 0x00007ffff77d69d0 in __GI___nanosleep (requested_time=requested_time@entry=0x7fffffffc870, remaining=remaining@entry=0x7fffffffc870) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 #1 0x00007ffff13b012d in qt_nanosleep (amount=...) at kernel/qelapsedtimer_unix.cpp:195 #2 0x00007ffff30c99f8 in KDirWatchPrivate::useFAM (this=this@entry=0x555555c2ab50, e=e@entry=0x555555ab1210) at ./src/lib/io/kdirwatch.cpp:676 #3 0x00007ffff30c9e6b in KDirWatchPrivate::addWatch (this=this@entry=0x555555c2ab50, e=e@entry=0x555555ab1210) at ./src/lib/io/kdirwatch.cpp:979 #4 0x00007ffff30ca555 in KDirWatchPrivate::addEntry (this=0x555555c2ab50, instance=0x555555c25988, _path=..., sub_entry=<optimized out>, isDir=<optimized out>, watchModes=...) at ./src/lib/io/kdirwatch.cpp:930 #5 0x00007ffff506f6f5 in KSycocaPrivate::findDatabase() () from /usr/lib/x86_64-linux-gnu/libKF5Service.so.5 #6 0x00007ffff506f7f3 in KSycoca::connectNotify(QMetaMethod const&) () from /usr/lib/x86_64-linux-gnu/libKF5Service.so.5 #7 0x00007ffff1382d8a in QObjectPrivate::connectImpl (sender=sender@entry=0x555555cf8510, signal_index=3, receiver=receiver@entry=0x555555a4f6e0, slot=slot@entry=0x7fffffffcea0, slotObj=slotObj@entry=0x555555d259b0, type=<optimized out>, types=<optimized out>, senderMetaObject=<optimized out>) at kernel/qobject.cpp:5040 #8 0x00007ffff13830ed in QObject::connectImpl (sender=0x555555cf8510, signal=<optimized out>, receiver=0x555555a4f6e0, slot=0x7fffffffcea0, slotObj=0x555555d259b0, type=Qt::AutoConnection, types=0x0, senderMetaObject=<optimized out>) at kernel/qobject.cpp:4970 #9 0x00007ffff7b3a0b9 in ?? () from /usr/lib/x86_64-linux-gnu/libkdeinit5_dolphin.so #10 0x00007ffff7b2aa2f in kdemain () from /usr/lib/x86_64-linux-gnu/libkdeinit5_dolphin.so #11 0x00007ffff7713b97 in __libc_start_main (main=0x555555554720, argc=2, argv=0x7fffffffd128, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd118) at ../csu/libc-start.c:310 #12 0x000055555555475a in _start () Thanks. Unfortunately I can't reproduce this.
Given the backtrace what appears to happen is that since inotify is not functional, because we've exhausted the inotify instance cap, the file monitoring will fall back to a different system called FAM and that has a loop that has potential to never exit it looks like. A user needs to manually install the FAM daemon to actually cause this behavior though, at least on neon.
> do {
> famEventReceived();
> if (startedFAMMonitor && !e->m_famReportedSeen) {
> // 50 is ~half the time it takes to setup a watch. If gamin's latency
> // gets better, this can be reduced.
> QThread::msleep(50);
> }
> } while (startedFAMMonitor &&!e->m_famReportedSeen);
^ if m_famReportedSeen is never true, that loop never terminates. I'm not quite sure why that would happen but in any event there ought to be a cap on how long we are willing to wait which should then solve the problem that apps deadlock.
Moving bug to correct product.
A possibly relevant merge request was started @ https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/8 Strange because FAM nor libfam0 are installed on this system. I found this one lib: /usr/lib/libfam.so.0.0.0 This LIB is in the package libgamin0. This is an Coresytempart. If i would remove the package, whole NEON will be deinstalled. Git commit 8599c4d58b82ad0096d59a4a23e54d75b5c7f558 by Harald Sitter. Committed on 20/08/2020 at 13:59. Pushed by sitter into branch 'master'. do not wait for fam events indefinitely when famEventReceived encountered trouble with getting events it'll disable FAM entirely and instead register another fallback system. this means we'll have to check if FAM has been disabled every iteration and if it has break out of the loop. otherwise we'd loop indefinitely. we'll now also cap the amount of time (or rather iterations) we'll wait for the event to arrive. this acts as a safety net against further infinity loops. what's more, since this will get called from the main thread the GUI is blocked by us, so, 4s is already a lot all things considered. FIXED-IN: 5.74 M +33 -19 src/lib/io/kdirwatch.cpp M +1 -0 src/lib/io/kdirwatch_p.h https://invent.kde.org/frameworks/kcoreaddons/commit/8599c4d58b82ad0096d59a4a23e54d75b5c7f558 *** Bug 411747 has been marked as a duplicate of this bug. *** *** Bug 389781 has been marked as a duplicate of this bug. *** *** Bug 401450 has been marked as a duplicate of this bug. *** *** Bug 400583 has been marked as a duplicate of this bug. *** Git commit 5a648c56ebbbb9b8ba87fedf8423b5830edbd367 by Harald Sitter. Committed on 16/09/2020 at 11:30. Pushed by sitter into branch 'master'. disabe ksycoca rebuilding on runner threads this frees the internal per-thread inotify instance used to watch database changes meaning runner applications no longer claim N inotify instances (N=size of threadweaver pool). the database is still reloaded by ksyscoca internally if it finds the backing file has changed. M +5 -1 runners/services/autotests/CMakeLists.txt M +40 -1 runners/services/autotests/servicerunnertest.cpp M +5 -2 runners/services/servicerunner.cpp https://invent.kde.org/plasma/plasma-workspace/commit/5a648c56ebbbb9b8ba87fedf8423b5830edbd367 Git commit 7223eaa5c5a662e6da6bf8190b41acb9bd8a50d8 by Harald Sitter. Committed on 22/09/2020 at 11:03. Pushed by sitter into branch 'master'. fix(kded): move to kdirwatch to reduce inotify footprint Each QFileSystemWatch claims its own inotify instance. This is problematic because there is a default limit of 127 instances per user, fivolously claiming instances is therefore really poor behavior. KDirWatch meanwhile shares the same underlying instance behind the scene so each thread would only have a single instance with N different watches on it. Moving to KDW effectively reduces the inotify instance footprint of the kded module from N per Control to 0 since kded always has a kdirwatch for ksyscoca/kservice monitoring and our KDW is backed by the same instance. M +5 -4 common/control.cpp M +3 -3 common/control.h M +1 -1 tests/kded/CMakeLists.txt https://invent.kde.org/plasma/kscreen/commit/7223eaa5c5a662e6da6bf8190b41acb9bd8a50d8 Git commit 44a5f089b69d349f27d53e9dc5daa7e60ac50445 by Harald Sitter. Committed on 22/09/2020 at 13:06. Pushed by sitter into branch 'Plasma/5.20'. fix(kded): move to kdirwatch to reduce inotify footprint Each QFileSystemWatch claims its own inotify instance. This is problematic because there is a default limit of 127 instances per user, fivolously claiming instances is therefore really poor behavior. KDirWatch meanwhile shares the same underlying instance behind the scene so each thread would only have a single instance with N different watches on it. Moving to KDW effectively reduces the inotify instance footprint of the kded module from N per Control to 0 since kded always has a kdirwatch for ksyscoca/kservice monitoring and our KDW is backed by the same instance. (cherry picked from commit 7223eaa5c5a662e6da6bf8190b41acb9bd8a50d8) M +5 -4 common/control.cpp M +3 -3 common/control.h M +1 -1 tests/kded/CMakeLists.txt https://invent.kde.org/plasma/kscreen/commit/44a5f089b69d349f27d53e9dc5daa7e60ac50445 Git commit 783e856ea0b970fb864115aabe9df530235eb3c2 by Harald Sitter. Committed on 23/09/2020 at 13:31. Pushed by sitter into branch 'master'. disable ksyscoca rebuilding in appstream runner same as 5a648c56ebbbb9b8ba87fedf8423b5830edbd367 for the service runner. Kservice internally will hold a KDirWatch instance through KSycoca and that instance has an inotify instance. Since inotify instances have a per-user limit we mustn't be wasteful with them. Since our runner threads do not actually need automatic ksyscoca reloading (reloading on-demand as we use ksyscoca is perfectly reasonable) we need to explicitly disable this feature so the internal inotify instance can be freed. M +5 -0 runners/appstream/appstreamrunner.cpp https://invent.kde.org/plasma/plasma-workspace/commit/783e856ea0b970fb864115aabe9df530235eb3c2 Git commit d684f83254fa56aeba32c9d638b24466e909c8ed by Harald Sitter. Committed on 23/09/2020 at 13:34. Pushed by sitter into branch 'Plasma/5.20'. disable ksyscoca rebuilding in appstream runner same as 5a648c56ebbbb9b8ba87fedf8423b5830edbd367 for the service runner. Kservice internally will hold a KDirWatch instance through KSycoca and that instance has an inotify instance. Since inotify instances have a per-user limit we mustn't be wasteful with them. Since our runner threads do not actually need automatic ksyscoca reloading (reloading on-demand as we use ksyscoca is perfectly reasonable) we need to explicitly disable this feature so the internal inotify instance can be freed. (cherry picked from commit 783e856ea0b970fb864115aabe9df530235eb3c2) M +5 -0 runners/appstream/appstreamrunner.cpp https://invent.kde.org/plasma/plasma-workspace/commit/d684f83254fa56aeba32c9d638b24466e909c8ed |