Bug 476752 - Option to un-throttle Baloo File Indexing Process
Summary: Option to un-throttle Baloo File Indexing Process
Status: REPORTED
Alias: None
Product: frameworks-baloo
Classification: Frameworks and Libraries
Component: general (show other bugs)
Version: unspecified
Platform: unspecified Linux
: NOR wishlist
Target Milestone: ---
Assignee: baloo-bugs-null
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-11-09 12:22 UTC by marcus
Modified: 2023-11-10 09:46 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments
System load during IDLE indexing (377.50 KB, image/png)
2023-11-10 09:46 UTC, marcus
Details

Note You need to log in before you can comment on or make changes to this bug.
Description marcus 2023-11-09 12:22:05 UTC
SUMMARY
***
NOTE: If you are reporting a crash, please try to attach a backtrace with debug symbols.
See https://community.kde.org/Guidelines_and_HOWTOs/Debugging/How_to_create_useful_crash_reports
***

I am asking for a button (or command) to speed up indexing process.

https://forum.manjaro.org/t/priorize-baloo-file-indexing-process/151223
"Baloo_file_extractor is busy indexing files at 5% cpu usage. Because of poor thermal design pc fans are going up and down all the time. Is there a way to get through this process quicker? I would rather have it done fast than being gentle."


STEPS TO REPRODUCE
1. 
2. 
3. 

OBSERVED RESULT


EXPECTED RESULT


SOFTWARE/OS VERSIONS
Windows: 
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

ADDITIONAL INFORMATION
Comment 1 Nate Graham 2023-11-09 20:11:41 UTC
Funny, this is usually the opposite of what people ask for. :)

I'm not sure it makes that much sense to add a user-facing knob for some kind of "turbo mode", but I wouldn't argue strongly against it either. What do you think, Stefan?
Comment 2 Stefan Brüns 2023-11-09 20:34:45 UTC
There is some information missing:

1. Does "5%" refer to one CPU core, or the whole system (i.e 100% on 1 of 20 cores)
2. Is the system currently idle, or are you actively using it?
3. Is this during the initial indexing, or during content indexing?

Most indexers are limited to single-threading by the underlying libraries. Although it would be possible to spawn multiple processes, one for each file, this would require some significant changes.

If the system has many cores it may be OK to use more than one, preferably efficiency cores if available. But this is definitely something which should be implemented outside of Baloo, so it becomes reusable.

There is also the IO throttling happening when the computer is in use (according to KIdleTime). Currently the throttling is fairly aggressive and could probably be relaxed. But pushing this up too much will result complains, flame wars etc. And "too much" is highly system dependent.
Comment 3 Stefan Brüns 2023-11-09 20:35:43 UTC
information required
Comment 4 tagwerk19 2023-11-09 20:58:14 UTC
(In reply to Stefan Brüns from comment #2)
> There is also the IO throttling happening when the computer is in use (according to KIdleTime).
My experience is that when baloo thinks the system is not busy and says
    kf.baloo: Not busy, fast indexing
you can use a full CPU core. As soon as you ask the system to do "anything", baloo switches to
    kf.baloo: Busy, paced indexing
and you get a noticeable pause after indexing each batch of 40 files. (This is with content indexing...)

> ... pushing this up too much will result complains, flame wars etc. And "too much" is highly system dependent ...
If you have a "turbo" button, you are not changing the defaults, it's then a user choice and you can ignore the flame wars :-)

Maybe increase the number of files indexed per batch. Maybe switch to "fast indexing". Maybe increase the "MemoryHigh" in the systemd unit file.

I think it's a useful option...
Comment 5 marcus 2023-11-09 22:18:32 UTC
I use stock hardware
This is copied from system settings. Sorry for language mix.
> Betriebssystem: Manjaro Linux 
> KDE-Plasma-Version: 5.27.9
> KDE-Frameworks-Version: 5.111.0
> Qt-Version: 5.15.11
> Kernel-Version: 6.1.60-1-MANJARO (64-bit)
> Grafik-Plattform: X11
> Prozessoren: 16 × 11th Gen Intel® Core™ i7-11700F @ 2.50GHz
> Speicher: 15,5 GiB Arbeitsspeicher
> Grafikprozessor: NVIDIA GeForce RTX 3060/PCIe/SSE2
> Hersteller: Acer
> Produktname: Predator PO3-630
> Systemversion: 1.0

Yesterday I startet a Manjaro system update which might have triggered reindexing. I just don't know.
The system was in IDLE state when I noticed the fan going up and down. In KDE Systemmonitor I could see one CPU thread being active  while others were inactive which led to ~5% CPU usage.

Please let me know if more information is missing
Comment 6 marcus 2023-11-10 00:45:42 UTC
Status changed back to reported. I hope this is correct
Comment 7 marcus 2023-11-10 09:46:24 UTC
Created attachment 163009 [details]
System load during IDLE indexing

So this is the ~5% cpu load visualized. It is core 5 and 6 that do stuff the rest is idle