Bug 442453 - Significant amount of disk write in short time after deleting many files in monitored folder.
Summary: Significant amount of disk write in short time after deleting many files in m...
Status: CONFIRMED
Alias: None
Product: frameworks-baloo
Classification: Frameworks and Libraries
Component: Baloo File Daemon (show other bugs)
Version: 5.85.0
Platform: Manjaro Linux
: NOR major
Target Milestone: ---
Assignee: baloo-bugs-null
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-09-14 21:17 UTC by cantfind
Modified: 2024-07-09 18:59 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description cantfind 2021-09-14 21:17:21 UTC
SUMMARY

I was deleting quite a bit of files from my home folder (not adding anything, not moving anything - deleting with ctrl+shift+del so not even moving to recycle bin). 

baloo_file process consumed about 50% of a single core, and wrote 20-100 Mega Bytes per second for at least several minutes, reaching a size of index of more than 6GB.

Not sure I can reproduce it again, though.


STEPS TO REPRODUCE
1. delete files from monitored folder, or move files to a different (unmonitored disk)?

OBSERVED RESULT

baloo_file process wrote a lot of data, and used up a lot of CPU for a while.
Even though balooctl status reported it was idle.

The balooctl status command took about half a minute to print the data, and balooctl monitor didn't show any activity while running from a different tab the status command.


EXPECTED RESULT

To not increase the index size while indexed files decrease...


SOFTWARE/OS VERSIONS
Linux/KDE Plasma: Manjaro
(available in About System)
KDE Plasma Version: 5.22.5
KDE Frameworks Version: 5.85.0
Qt Version: 5.15.2

ADDITIONAL INFORMATION


here are the results of 2 subsequent runs of balooctl status:

Baloo File Indexer is running
Indexer state: Idle
Total files indexed: 199,551
Files waiting for content indexing: 0
Files failed to index: 0
Current size of index is 5.95 GiB


Baloo File Indexer is running
Indexer state: Idle
Total files indexed: 199,109
Files waiting for content indexing: 0
Files failed to index: 0
Current size of index is 6.42 GiB

Even though, the time between running these two commands was about a minute, the size of index grew by  about 500MB, while the Total files indexed number went down...

Here's the output of balooctl monitor:

Press ctrl+c to stop monitoring
File indexer is running
Idle
Comment 1 cantfind 2021-09-14 21:20:50 UTC
Killing the baloo_file process, and than starting indexing again "fixed" it - baloo_file no longer used up cpu and no longer increased the index size.

But now my index size is quite gigantic.
Comment 2 cantfind 2021-09-14 21:37:49 UTC
I was able to reproduce it again. All it takes is deleting a lot of files in Dolphin from a monitored folder. (The ones I deleted now were archived books).
Comment 3 cantfind 2021-09-14 21:39:05 UTC
now my balooctl status reports:

Baloo File Indexer is running
Indexer state: Idle
Total files indexed: 198,471
Files waiting for content indexing: 0
Files failed to index: 0
Current size of index is 7.20 GiB
Comment 4 tagwerk19 2021-09-15 08:15:10 UTC
Removing entries seems to be hard...

I think that baloo_file does not "batch up" deletes in the same way as it batches up its content indexing. I've tried watching with iotop. 

There seems to be a "gotcha" if you try to check progress with "balooctl status" - ref Bug 437754. Could be where you get your 6GB index...
Comment 5 cantfind 2021-09-15 13:41:15 UTC
(In reply to tagwerk19 from comment #4)
> Removing entries seems to be hard...
> 
> I think that baloo_file does not "batch up" deletes in the same way as it
> batches up its content indexing. I've tried watching with iotop. 
> 
> There seems to be a "gotcha" if you try to check progress with "balooctl
> status" - ref Bug 437754. Could be where you get your 6GB index...

It's not just wrong reporting on balooctl... ksysguard shows a lot of writing going on in baloo_file process, and quite a bit of cpu usage too.
Comment 6 cantfind 2021-09-15 13:42:18 UTC
RAM usage was also at ~5GB...
Comment 7 tagwerk19 2021-09-20 18:13:37 UTC
(In reply to cantfind from comment #0)
> OBSERVED RESULT
> 
> baloo_file process wrote a lot of data, and used up a lot of CPU for a while.
> Even though balooctl status reported it was idle.
> 
> The balooctl status command took about half a minute to print the data, and
> balooctl monitor didn't show any activity while running from a different tab
> the status command.
Flagging "Confirmed"...