Summary: | "balooctl status" can trigger high memory use | ||
---|---|---|---|
Product: | [Frameworks and Libraries] frameworks-baloo | Reporter: | tagwerk19 |
Component: | balooctl | Assignee: | Stefan BrĂ¼ns <stefan.bruens> |
Status: | CONFIRMED --- | ||
Severity: | normal | CC: | adam.m.fontenot+kde, baloo-bugs-null, katyaberezyaka, Martin, med.medin.2014, nate |
Priority: | NOR | ||
Version: | 5.82.0 | ||
Target Milestone: | --- | ||
Platform: | Neon | ||
OS: | Linux | ||
See Also: |
https://bugs.kde.org/show_bug.cgi?id=442453 https://bugs.kde.org/show_bug.cgi?id=460460 |
||
Latest Commit: | Version Fixed In: |
Description
tagwerk19
2021-05-27 21:26:43 UTC
(In reply to tagwerk19 from comment #0) > It's quite likely that creating so many files so quickly hits the > inotify "event limit" and baloo doesn't get told of all the new files. Creating the testfiles via a script can give you a kf.baloo: Inotify - too many event - Overflowed and you need to run "balooctl check" when the script has finished to find the rest of the newly created files. It is also possible to get an "Overflowed" message when deleting files and when this happens, baloo stops removing deleted entries and a "balooctl check" does not to resolve the situation. EXPECTED RESULTS: Ideally, if a baloo receives an "inotify overflow", it should queue up a "balooctl check" "balooctl check" should recognise up files that no longer exist in the filesystem and remove the index entries tagwerk can you show the output of "balooctl indexSize". For me it currently is: % balooctl indexSize File Size: 8,12 GiB Used: 77,17 MiB PostingDB: 1,36 GiB 1801.974 % PositionDB: 1,68 GiB 2228.958 % DocTerms: 784,75 MiB 1016.891 % DocFilenameTerms: 68,79 MiB 89.137 % DocXattrTerms: 4,00 KiB 0.005 % IdTree: 17,55 MiB 22.742 % IdFileName: 77,23 MiB 100.071 % DocTime: 51,05 MiB 66.157 % DocData: 38,66 MiB 50.101 % ContentIndexingDB: 9,32 MiB 12.077 % FailedIdsDB: 0 B 0.000 % MTimeDB: 15,06 MiB 19.518 % I do not claim I understand the output though. 2228% of what? Why 77 MiB used? (In reply to Martin Steigerwald from comment #2) > % balooctl indexSize > File Size: 8,12 GiB > Used: 77,17 MiB There's some analysis/discussion/confusion about the percentages here: https://bugs.kde.org/show_bug.cgi?id=354636#c10 I think the "used" sizes are believable. I have copied/compressed a test index with mdb_copy -n -c index index.new (from lmdb-utils) and this changes "indexSize" details from: File Size: 2,28 GiB Used: 18,99 MiB PostingDB: 4,89 MiB 25.735 % PositionDB: 4,92 MiB 25.921 % DocTerms: 2,47 MiB 13.001 % DocFilenameTerms: 1,70 MiB 8.969 % DocXattrTerms: 4,00 KiB 0.021 % IdTree: 240,00 KiB 1.234 % IdFileName: 1,94 MiB 10.204 % DocTime: 1,29 MiB 6.809 % DocData: 1,53 MiB 8.044 % ContentIndexingDB: 0 B 0.000 % FailedIdsDB: 0 B 0.000 % MTimeDB: 12,00 KiB 0.062 % to: File Size: 19,37 MiB Used: 18,99 MiB PostingDB: 4,89 MiB 25.735 % PositionDB: 4,92 MiB 25.921 % DocTerms: 2,47 MiB 13.001 % DocFilenameTerms: 1,70 MiB 8.969 % DocXattrTerms: 4,00 KiB 0.021 % IdTree: 240,00 KiB 1.234 % IdFileName: 1,94 MiB 10.204 % DocTime: 1,29 MiB 6.809 % DocData: 1,53 MiB 8.044 % ContentIndexingDB: 0 B 0.000 % FailedIdsDB: 0 B 0.000 % MTimeDB: 12,00 KiB 0.062 % Which points at loads of "empty space" created during the deletions/status. This is on ext4, after having created 50000 files and deleted 20000. I will try the same on BTRFS Whether this helps any...? I helped, but not as much as with your setup: % ~/.local/share/baloo> balooctl indexSize File Size: 8,12 GiB Used: 79,78 MiB PostingDB: 1,36 GiB 1743.970 % PositionDB: 1,68 GiB 2157.734 % DocTerms: 785,45 MiB 984.552 % DocFilenameTerms: 68,79 MiB 86.226 % DocXattrTerms: 4,00 KiB 0.005 % IdTree: 17,55 MiB 22.000 % IdFileName: 77,23 MiB 96.803 % DocTime: 51,05 MiB 63.996 % DocData: 38,67 MiB 48.475 % ContentIndexingDB: 9,29 MiB 11.649 % FailedIdsDB: 0 B 0.000 % MTimeDB: 15,06 MiB 18.881 % % ~/.local/share/baloo> mdb_copy -n -c index index.new % ~/.local/share/baloo> LANG=en ls -lh total 13G -rw-r--r-- 1 martin martin 8.2G Jun 26 15:21 index -rw-r--r-- 1 martin martin 8.0K Jun 26 21:02 index-lock -rw-r--r-- 1 martin martin 4.1G Jun 26 21:02 index.new % ~/.local/share/baloo> mv index.new index % ~/.local/share/baloo> balooctl indexSize File Size: 4,08 GiB Used: 79,78 MiB PostingDB: 1,36 GiB 1743.970 % PositionDB: 1,68 GiB 2157.734 % DocTerms: 785,45 MiB 984.552 % DocFilenameTerms: 68,79 MiB 86.226 % DocXattrTerms: 4,00 KiB 0.005 % IdTree: 17,55 MiB 22.000 % IdFileName: 77,23 MiB 96.803 % DocTime: 51,05 MiB 63.996 % DocData: 38,67 MiB 48.475 % ContentIndexingDB: 9,29 MiB 11.649 % FailedIdsDB: 0 B 0.000 % MTimeDB: 15,06 MiB 18.881 % *** Bug 449713 has been marked as a duplicate of this bug. *** Flagging as Confirmed on the basis of Bug 460460 |