Bug 406868 - baloo_file_extractor fails MTimeDB::del with MDB_BAD_TXN, constant high CPU use
Summary: baloo_file_extractor fails MTimeDB::del with MDB_BAD_TXN, constant high CPU use
Status: RESOLVED FIXED
Alias: None
Product: frameworks-baloo
Classification: Frameworks and Libraries
Component: Baloo File Daemon (show other bugs)
Version: 5.57.0
Platform: Arch Linux Linux
: NOR normal
Target Milestone: ---
Assignee: baloo-bugs-null
URL:
Keywords:
: 392037 415132 462866 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-04-25 02:48 UTC by Ash
Modified: 2023-05-25 07:29 UTC (History)
12 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
screenshot from htop showing the large CPU and vmem use, as well as which processes are responsible (550.41 KB, image/png)
2019-04-25 02:48 UTC, Ash
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ash 2019-04-25 02:48:38 UTC
Created attachment 119620 [details]
screenshot from htop showing the large CPU and vmem use, as well as which processes are responsible

SUMMARY
The baloo_file_extractor and tags.so [kdeinit5] processes on my machine seem to be always running and using large (though not 100%) amounts of CPU, and a full 256G of vmem (htop screencap attached). The journald  logs are filled with the following:

Apr 25 12:38:54 sylveon kdeinit5[1097]: (QUrl("tags:/"), QUrl("tags:/"))
Apr 25 12:38:54 sylveon kdeinit5[1097]: (QUrl("tags:/"), QUrl("tags:/"))
Apr 25 12:38:54 sylveon kdeinit5[1097]: ()

which repeats many, many times. It always follows the same pattern: 2 QUrls and then an empty. The following messages are mixed in, then repeated many times after the kdeinit5 messages stop:

Apr 25 12:38:55 sylveon baloo_file_extractor[1458]: org.kde.baloo.engine: MTimeDB::del 0 3391499450514944 MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
Apr 25 12:38:55 sylveon baloo_file_extractor[1458]: org.kde.baloo.engine: Transaction::commit MDB_BAD_TXN: Transaction must abort, has a child, or is invalid

The transaction ID (I assume that's what the number starting in 33 is?) seems to remain constant. After a few hundred of each message, they'll stop for roughly 30 seconds, and then another batch come through.

STEPS TO REPRODUCE
I honestly don't know. I ran a full system update after the recent batch of KDE Applications updates and now this happens every time I boot my machine - it was working fine before. I'm happy to dump config files and databases and whatever else is needed.

OBSERVED RESULT
baloo_file_extractor constantly uses large amounts of vmem and cpu, filling the system logs with error messages

EXPECTED RESULT
baloo_file_extractor doesn't do that.

SOFTWARE/OS VERSIONS
Linux/KDE Plasma: Antegros, kernel 5.0.9-arch1-1-ARCH
KDE Plasma Version: 5.15.4
KDE Frameworks Version: 5.57.0
Qt Version: 5.12.3

ADDITIONAL INFORMATION
This is my first time filing a bug with a large project like KDE, hopefully this is okay! As I said before, more than happy to provide other information; I really had no idea where to start with this.
Comment 1 Dan 2019-04-25 09:37:03 UTC
Hi Ash, I think I may have experienced the same problem as you, and I wonder if you may need to increase your inotify watch limit. See here?

https://askubuntu.com/questions/1137733/baloo-running-amok-kubuntu-19-04/1137997#1137997
Comment 2 Dan 2019-04-25 10:29:26 UTC
(In reply to Dan from comment #1)
> Hi Ash, I think I may have experienced the same problem as you, and I wonder
> if you may need to increase your inotify watch limit. See here?
> 
> https://askubuntu.com/questions/1137733/baloo-running-amok-kubuntu-19-04/
> 1137997#1137997

Oh dear, I spoke to soon. Baloo is back. I've managed to disable it thank to the comments by DK Bose in that same thread. But I don't know what is causing it.

Errors include:

KDE Baloo File Indexer has reached the inotify folder watch limit. File changes will be ignored

baloo/index: No such file or directory
Comment 3 Ash 2019-04-25 10:30:51 UTC
(In reply to Dan from comment #1)
> Hi Ash, I think I may have experienced the same problem as you, and I wonder
> if you may need to increase your inotify watch limit. See here?
> 
> https://askubuntu.com/questions/1137733/baloo-running-amok-kubuntu-19-04/
> 1137997#1137997

Hey Dan!
Thanks for the suggestion, but sadly that didn't change anything for me. I grepped through my journalctl logs and didn't see any mention of inotify, so perhaps we're experiencing different bugs. Thanks to the comments on that question though, I was able to disable baloo, which has at least given me a temporary fix to rescue my battery life ;)
Comment 4 Nate Graham 2020-10-26 16:13:03 UTC
No duplicate bug reports seen in almost a year. After all the bugfixing work done during that time, I think we can call this fixed!
Comment 5 Nate Graham 2020-10-26 16:45:06 UTC
*** Bug 415132 has been marked as a duplicate of this bug. ***
Comment 6 Nate Graham 2020-10-26 16:47:51 UTC
*** Bug 392037 has been marked as a duplicate of this bug. ***
Comment 7 Bart Ribbers 2022-06-09 07:51:46 UTC
I can sadly reproduce this with Baloo 5.94.0. The SDDM Wayland log is getting filled up to gigabytes at a time if I let Baloo run and the CPU usage of the process is 100% constantly.

The following in the log just repeats itself every half a second or so:

kf.baloo.engine: MTimeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdTreeDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: IdFilenameDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: DocumentDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: DocumentDB::put MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
kf.baloo.engine: DocumentTimeDB::put 3979527717912610 MDB_BAD_TXN: Transaction must abort, has a child, or is invalid

First time I noticed this was when I was checking for something else at which point the log file was 1.5GB big. Removing just these messages from it got it back down to a few KB, so just Baloo was responsible for the huge file size.
Comment 8 tagwerk19 2022-06-13 07:22:41 UTC
(In reply to Bart Ribbers from comment #7)
> ... if I let Baloo run and the CPU usage of the process is 100% constantly ...
In this instance, apologies, I think the best option is to do a "balooctl purge". That is, delete the index and reindex from scratch.

The MDB_BAD_TXN is a report from LMDB library and suggests a corruption of the index.

It might be worth checking to see how big the index is (with "balooctl indexSize") and see whether it is "too large" (coming up to the size of your RAM or the "Out of Memory" trigger threshold). If the index has "ballooned" in size, baloo uses a lot of CPU when trying to add/remove entries from it.

I think though the only practical option is to purge and reindex.
Comment 9 nyanpasu64 2022-06-27 10:52:31 UTC
Bart Ribbers (or anyone else encountering this corrupt index), have you already deleted your index file? If not, can you share your own corrupted database files for me to analyze, here or to Bug 434926, so I have more samples of how a database gets corrupted? Note that it will contain possibly-sensitive file paths (and perhaps even contents).

If you're not willing to share private file paths, can you build my custom lmdb at https://codeberg.org/nyanpasu64/lmdb-debug, copy `index` to `index.orig` as a backup, mark both files as read-only to avoid accidentally writing into them, run my custom `.../lmdb/libraries/liblmdb/mdb_dump index -na -f /dev/null`, and report the errors detected (this does not leak personal information)?
Comment 10 tagwerk19 2022-12-27 10:04:15 UTC
*** Bug 462866 has been marked as a duplicate of this bug. ***
Comment 11 David Kredba 2023-05-25 07:29:59 UTC
I had to rebuild the index file after the "[TermGenerator] Skip all unprintable characters" change were made, because it broke it.
It worked for a few weeks even with Baloo KF 5.106.0 but last two days KDE reports crashes like below.
Your repository of lmdb does not support my system's lmdb version so I used mdb_dump that is present on my FS:

mdb_dump index -na -f /dev/null
mdb.c:5882: Assertion 'IS_BRANCH(mc->mc_pg[mc->mc_top])' failed in mdb_cursor_sibling()

mdb_copy -n -c index index.new
*** stack smashing detected ***: terminated
Neúspěšně ukončen (SIGABRT)

Index file size is 42360246272 Bytes.

The crash BT:
Application: Baloo File Extractor (baloo_file_extractor), signal: Segmentation fault
Content of s_kcrashErrorMessage: std::unique_ptr<char []> = {get() = 0x0}
[KCrash Handler]
#6  0x00007fe40b6dcab3 in mdb_midl_xmerge (idl=idl@entry=0x7fa3472c3018, merge=merge@entry=0x7fa87eee3010) at midl.c:207
#7  0x00007fe40b6d3e26 in mdb_page_alloc (num=num@entry=23531, mp=mp@entry=0x7ffc09b946f0, mc=0x7ffc09b94870) at mdb.c:2306
#8  0x00007fe40b6d6123 in mdb_page_new (mp=<synthetic pointer>, num=23531, flags=4, mc=0x7ffc09b94870) at mdb.c:7220
#9  mdb_node_add (mc=mc@entry=0x7ffc09b94870, indx=<optimized out>, key=key@entry=0x7ffc09b94c60, data=0x7ffc09b94c70, pgno=pgno@entry=0, flags=0) at mdb.c:7362
#10 0x00007fe40b6d97a9 in mdb_cursor_put (mc=mc@entry=0x7ffc09b94870, key=key@entry=0x7ffc09b94c60, data=data@entry=0x7ffc09b94c70, flags=flags@entry=0) at mdb.c:6987
#11 0x00007fe40b6dbf61 in mdb_put (txn=0x558dd2818a30, dbi=<optimized out>, key=0x7ffc09b94c60, data=0x7ffc09b94c70, flags=0) at mdb.c:9076
#12 0x00007fe40c9d3d09 in Baloo::PositionDB::put (this=this@entry=0x7ffc09b94d60, term=..., list=...) at /var/tmp/portage/kde-frameworks/baloo-5.106.0/work/baloo-5.106.0/src/engine/positiondb.cpp:69
#13 0x00007fe40c9e6d25 in Baloo::WriteTransaction::commit (this=<optimized out>) at /var/tmp/portage/kde-frameworks/baloo-5.106.0/work/baloo-5.106.0/src/engine/writetransaction.cpp:319
#14 0x00007fe40c9de1bf in Baloo::Transaction::commit (this=0x558dd2819c20) at /var/tmp/portage/kde-frameworks/baloo-5.106.0/work/baloo-5.106.0/src/engine/transaction.cpp:272
#15 0x0000558dd165a90c in Baloo::App::processNextFile (this=0x7ffc09b952b0) at /var/tmp/portage/kde-frameworks/baloo-5.106.0/work/baloo-5.106.0/src/file/extractor/app.cpp:109
#16 0x00007fe40bfe2ac2 in QtPrivate::QSlotObjectBase::call (a=0x7ffc09b94ea0, r=<optimized out>, this=<optimized out>) at /var/tmp/portage/dev-qt/qtcore-5.15.9/work/qtbase-everywhere-src-5.15.9/include/QtCore/../../src/corelib/kernel/qobjectdefs_impl.h:398
#17 QSingleShotTimer::timerEvent (this=0x558dd2894c50) at /var/tmp/portage/dev-qt/qtcore-5.15.9/work/qtbase-everywhere-src-5.15.9/src/corelib/kernel/qtimer.cpp:320
#18 0x00007fe40bfd6385 in QObject::event (this=0x558dd2894c50, e=0x7ffc09b94fd0) at /var/tmp/portage/dev-qt/qtcore-5.15.9/work/qtbase-everywhere-src-5.15.9/src/corelib/kernel/qobject.cpp:1369
#19 0x00007fe40bfaaf28 in QCoreApplication::notifyInternal2 (receiver=0x558dd2894c50, event=0x7ffc09b94fd0) at /var/tmp/portage/dev-qt/qtcore-5.15.9/work/qtbase-everywhere-src-5.15.9/src/corelib/kernel/qcoreapplication.cpp:1064
#20 0x00007fe40bfab11e in QCoreApplication::sendEvent (receiver=<optimized out>, event=<optimized out>) at /var/tmp/portage/dev-qt/qtcore-5.15.9/work/qtbase-everywhere-src-5.15.9/src/corelib/kernel/qcoreapplication.cpp:1462
#21 0x00007fe40bffc059 in QTimerInfoList::activateTimers (this=0x558dd27f2220) at /var/tmp/portage/dev-qt/qtcore-5.15.9/work/qtbase-everywhere-src-5.15.9/src/corelib/kernel/qtimerinfo_unix.cpp:643
#22 0x00007fe40bffc921 in timerSourceDispatch (source=<optimized out>) at /var/tmp/portage/dev-qt/qtcore-5.15.9/work/qtbase-everywhere-src-5.15.9/src/corelib/kernel/qeventdispatcher_glib.cpp:183
#23 0x00007fe40ac8a73c in g_main_dispatch (context=0x7fe400000ee0) at ../glib-2.76.2/glib/gmain.c:3460
#24 g_main_context_dispatch (context=context@entry=0x7fe400000ee0) at ../glib-2.76.2/glib/gmain.c:4200
#25 0x00007fe40ac8a9e8 in g_main_context_iterate (context=context@entry=0x7fe400000ee0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib-2.76.2/glib/gmain.c:4276
#26 0x00007fe40ac8aa7c in g_main_context_iteration (context=0x7fe400000ee0, may_block=1) at ../glib-2.76.2/glib/gmain.c:4343
#27 0x00007fe40bffccb6 in QEventDispatcherGlib::processEvents (this=0x558dd27cc7f0, flags=...) at /var/tmp/portage/dev-qt/qtcore-5.15.9/work/qtbase-everywhere-src-5.15.9/src/corelib/kernel/qeventdispatcher_glib.cpp:423
#28 0x00007fe40bfa994b in QEventLoop::exec (this=this@entry=0x7ffc09b95210, flags=..., flags@entry=...) at /var/tmp/portage/dev-qt/qtcore-5.15.9/work/qtbase-everywhere-src-5.15.9/include/QtCore/../../src/corelib/global/qflags.h:69
#29 0x00007fe40bfb1c26 in QCoreApplication::exec () at /var/tmp/portage/dev-qt/qtcore-5.15.9/work/qtbase-everywhere-src-5.15.9/include/QtCore/../../src/corelib/global/qflags.h:121
#30 0x00007fe40c3a2bec in QGuiApplication::exec () at /var/tmp/portage/dev-qt/qtgui-5.15.9/work/qtbase-everywhere-src-5.15.9/src/gui/kernel/qguiapplication.cpp:1870
#31 0x0000558dd1657430 in main (argc=<optimized out>, argv=<optimized out>) at /var/tmp/portage/kde-frameworks/baloo-5.106.0/work/baloo-5.106.0/src/file/extractor/main.cpp:43
[Inferior 1 (process 15736) detached]

I have one backup that I will try return to and will see if it will brake too.