Bug 341581

Summary: baloo_file - using 100% cpu
Product: [Unmaintained] Baloo Reporter: Vishesh Handa <me>
Component: Baloo File DaemonAssignee: Vishesh Handa <me>
Status: RESOLVED FIXED    
Severity: critical CC: asturm, cfcohen77, dimitrisfx, fabriziomarcotulli, hotmusicfan, hrvoje.senjan, kamiox, luca.p.fama, lukas, mail, matt, olivier, PVince81, wengxt
Priority: NOR    
Version First Reported In: unspecified   
Target Milestone: ---   
Platform: Other   
OS: Linux   
Latest Commit: Version Fixed In: 5.4
Sentry Crash Report:
Attachments: Crash when re-enabling baloo_file

Description Vishesh Handa 2014-12-05 15:47:36 UTC
A number of bug reports indicate that Xapian gets stuck occasionally and starts using 100% of the CPU. This has sadly also happened to me.

Thread 1 (Thread 0x7f9fea9b47c0 (LWP 20964)):
#0  0x00007f9fe937b79c in Key::operator<(Key) const () from /usr/lib64/libxapian.so.22
#1  0x00007f9fe937b888 in ChertTable::find_in_block(unsigned char const*, Key, bool, int) () from /usr/lib64/libxapian.so.22
#2  0x00007f9fe937c9b3 in ChertTable::find(Cursor*) const () from /usr/lib64/libxapian.so.22
#3  0x00007f9fe9358089 in ChertCursor::find_entry(std::string const&) () from /usr/lib64/libxapian.so.22
#4  0x00007f9fe9383073 in ChertValueManager::get_chunk_containing_did(unsigned int, unsigned int, std::string&) const () from /usr/lib64/libxapian.so.22
#5  0x00007f9fe938745b in ChertValueManager::get_value(unsigned int, unsigned int) const () from /usr/lib64/libxapian.so.22
#6  0x00007f9fe9367c6a in ChertDocument::do_get_value(unsigned int) const () from /usr/lib64/libxapian.so.22
#7  0x00007f9fe92ddcf6 in Xapian::Document::Internal::get_value(unsigned int) const () from /usr/lib64/libxapian.so.22
#8  0x00007f9fe92ddd5c in Xapian::Document::get_value(unsigned int) const () from /usr/lib64/libxapian.so.22
#9  0x00007f9fe969843a in Baloo::XapianDocument::value(int) const () from /usr/lib64/libKF5BalooXapian.so.1
#10 0x000000000041f423 in Baloo::BasicIndexingQueue::shouldIndex(Baloo::FileMapping&, QString const&) const ()
#11 0x0000000000420832 in Baloo::BasicIndexingQueue::process(Baloo::FileMapping&, QFlags<Baloo::UpdateDirFlag>) ()
#12 0x0000000000420db4 in Baloo::BasicIndexingQueue::processNextIteration() ()
#13 0x00007f9fe726ab36 in QObject::event(QEvent*) () from /usr/lib64/libQt5Core.so.5
#14 0x00007f9fe7eb59cc in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib64/libQt5Widgets.so.5
#15 0x00007f9fe7eba880 in QApplication::notify(QObject*, QEvent*) () from /usr/lib64/libQt5Widgets.so.5
#16 0x00007f9fe7239cfd in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /usr/lib64/libQt5Core.so.5
#17 0x00007f9fe723bbf7 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib64/libQt5Core.so.5
#18 0x00007f9fe7293a33 in postEventSourceDispatch(_GSource*, int (*)(void*), void*) () from /usr/lib64/libQt5Core.so.5
#19 0x00007f9fdf8b59e4 in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#20 0x00007f9fdf8b5c28 in g_main_context_iterate.isra () from /usr/lib64/libglib-2.0.so.0
#21 0x00007f9fdf8b5ccc in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#22 0x00007f9fe7292e7c in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/libQt5Core.so.5
#23 0x00007f9fe7237b3b in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/libQt5Core.so.5
#24 0x00007f9fe723f54f in QCoreApplication::exec() () from /usr/lib64/libQt5Core.so.5
#25 0x0000000000417faf in main ()
Comment 1 Vishesh Handa 2015-01-20 16:21:34 UTC
*** Bug 343011 has been marked as a duplicate of this bug. ***
Comment 2 Vishesh Handa 2015-02-10 12:07:27 UTC
*** Bug 343798 has been marked as a duplicate of this bug. ***
Comment 3 Vishesh Handa 2015-02-12 16:12:40 UTC
*** Bug 344037 has been marked as a duplicate of this bug. ***
Comment 4 Luca 2015-03-22 18:36:45 UTC
I can confirm that baloo_file process uses 100% CPU and when i restart, baloo_file process gets stuck at some random indexed file:

balooctl status
Baloo File Indexer is running
Indexed 3085 / 34791 files
Comment 5 Vishesh Handa 2015-03-23 14:10:38 UTC
@Luca: Thanks for the info.

This happens because the underlying database is getting corrupted. The only way to avoid it is to recreate the entire baloo index. You can do this via -

$ balooctl disable
$ balooctl enable

We're working on making the database more resistant to corruptions.
Comment 6 Lukáš Tinkl 2015-04-09 15:33:52 UTC
Created attachment 91966 [details]
Crash when re-enabling baloo_file

... and it crashes when I disable and reenable baloo_file with this backtrace
Comment 7 Vishesh Handa 2015-04-09 16:12:06 UTC
This is (In reply to Lukáš Tinkl from comment #6)
> Created attachment 91966 [details]
> Crash when re-enabling baloo_file
> 
> ... and it crashes when I disable and reenable baloo_file with this backtrace

This bug is due to database corruption. The only way around it to reset your database -

$ balooctl disable
$ balooclt enable
Comment 8 Lukáš Tinkl 2015-04-09 16:44:25 UTC
That's exactly what I did to make it crash :)
Comment 9 Vishesh Handa 2015-04-09 17:11:29 UTC
(In reply to Lukáš Tinkl from comment #8)
> That's exactly what I did to make it crash :)

Sorry. I should read it properly. Is this reproducible, if you disable/enable it again?
Comment 10 Lukáš Tinkl 2015-04-09 17:37:20 UTC
Actually not, seems to have started fine this time. I'll report back if I see it crash or consume CPU again
Comment 11 Dimitris Menounos 2015-04-26 07:59:53 UTC
Clean installation of Kubuntu 15.04. I am downloading the Android SDK when I notice the CPU fan spinning loudly. KSysGuard shows one CPU core at 100%. I assumed it was the download process. I leave the PC open and go to sleep. The other day the CPU is still working constantly at 100%. This time I check the process list and it is baloo!

I don't know what is going on. Thankfully we now have an option to disable the file search!

du -h .local/share/baloo/
20K     .local/share/baloo/notes
20K     .local/share/baloo/contacts
20K     .local/share/baloo/calendars
20K     .local/share/baloo/emailContacts
4,0K    .local/share/baloo/file
20K     .local/share/baloo/email
108K    .local/share/baloo/
Comment 12 Vincent Petry 2015-04-29 09:53:40 UTC
Not sure if it's the same thing I'm seeing on my env, here is what I see:
- before the ugprade I had Plasma 5.2.99 (Suse RPM) and baloo_file wasn't noticeable
- after the upgrade to Plasma 5.3 (and KDE 5.9 possibly, it was a system update), I now see baloo_file using 100% CPU (one core) after logging in

Possibly a regression.
I killed it now. Next time it happens I'll try to provide more info using the commands suggested above.
Comment 13 Cory F Cohen 2015-05-01 00:01:21 UTC
Top reported "10035 cory      39  19  471780  38144   5904 R 100.0  0.2   6689:43 baloo_file".
That's more than 4 days at 100% CPU.  So reading the thread I tried: "balooctl disable".  Which produced:

...
SQL Committed
Removing: 16 docs
ERROR DocNotFoundError: Can't delete non-existent document #28
ERROR DocNotFoundError: Can't delete non-existent document #29
ERROR DocNotFoundError: Can't delete non-existent document #30
ERROR DocNotFoundError: Can't delete non-existent document #31
ERROR DocNotFoundError: Can't delete non-existent document #32
ERROR DocNotFoundError: Can't delete non-existent document #33
ERROR DocNotFoundError: Can't delete non-existent document #38
ERROR DocNotFoundError: Can't delete non-existent document #69
ERROR DocNotFoundError: Can't delete non-existent document #73
ERROR DocNotFoundError: Can't delete non-existent document #246
ERROR DocNotFoundError: Can't delete non-existent document #309
ERROR DocNotFoundError: Can't delete non-existent document #311
ERROR DocNotFoundError: Can't delete non-existent document #357
ERROR DocNotFoundError: Can't delete non-existent document #358
ERROR DocNotFoundError: Can't delete non-existent document #362
ERROR DocNotFoundError: Can't delete non-existent document #363
Adding: 0 docs
Xapian Committed
SQL Committed

But the process did not exit.  Several more attempts to run "balooctl disable" would report only "Disabling the File Indexer", but not actually kill the process.   Running "kill 10035" successfully killed the process, and "balooctl status" now reported that the indexer was NOT running, and that it had indexed 0/0 files.  Running "balooctl enable" appears to have created an index after a few minutes, because I can now run baloosearch and balooshow.

I doubt that this will help much in troubleshooting the problem, but I at least wanted to report that it occurred.
Comment 14 Vincent Petry 2015-05-02 18:11:54 UTC
Tried this:
balooctl disable

But that doesn't kill it, have to do it manually with `killall baloo_file`

After killing it, I am able to do:
balooctl disable
balooctl enable

and the process restarts and seems to rescans a few files then stops (no more 100% cpu error).
However if I reboot the computer, the next time I login it will get stuck at 100%.
Comment 15 Fabrizio Marcotulli 2015-05-09 18:10:59 UTC
(In reply to Dimitris Menounos from comment #11)
> Clean installation of Kubuntu 15.04. I am downloading the Android SDK when I
> notice the CPU fan spinning loudly. KSysGuard shows one CPU core at 100%. I
> assumed it was the download process. I leave the PC open and go to sleep.
> The other day the CPU is still working constantly at 100%. This time I check
> the process list and it is baloo!
> 
> I don't know what is going on. Thankfully we now have an option to disable
> the file search!
> 
> du -h .local/share/baloo/
> 20K     .local/share/baloo/notes
> 20K     .local/share/baloo/contacts
> 20K     .local/share/baloo/calendars
> 20K     .local/share/baloo/emailContacts
> 4,0K    .local/share/baloo/file
> 20K     .local/share/baloo/email
> 108K    .local/share/baloo/

I just wantto piggybank on this comment. I just installed Kubuntu15.04 and Baloo takes 49% CPU (one of my two cores on an Acer Aspire 5553-4220)
Comment 16 Vishesh Handa 2015-05-11 11:06:17 UTC
Fixed with Plasma 5.4
Comment 17 hotmusicfan 2017-08-04 15:09:21 UTC
Hello!

Baloo_file or baloo_file_extractor have very high CPU usage when multiple files has being downloaded to the computer. 
openSUSE Leap 42.2/42.3 with Plasma 5.8 LTS.