Here is an example script to reproduce (has to be runned inside baloo indexed directory): ``` NAME=$(pwgen -N1); CONTENT=$(pwgen -N1) echo "$CONTENT" > "${NAME}"_1.txt echo "$CONTENT" > "${NAME}"_2.txt sleep 5 baloosearch "${CONTENT}" echo mv -v "${NAME}"_1.txt "${NAME}"_2.txt sleep 5 baloosearch "${CONTENT}" ``` The first baloosearch as expected finds files "${NAME}"_1.txt and "${NAME}"_2.txt But the second invocation (after "${NAME}"_1.txt is renamed to "${NAME}"_2.txt) finds "${NAME}"_2.txt twice!!! I believe this is the reason why a get tons of duplicates of zim notes in search results. Zim desktop wiki for robustness saves its notes in file $NOTE.txt.zim-new~ and then moves this file over $NOTE.txt. Also this obviously leads to unreasonably large index file etc etc
This appears to be working now with Baloo 5.51.
Whoops, it appears that this is actually fixed with https://phabricator.kde.org/D15944, which I forgot I was testing.
Git commit 7e5c005e6a3b563013e2ba8cb9c8f1b282e6f7b2 by Igor Poboiko. Committed on 08/10/2018 at 22:10. Pushed by poboiko into branch 'master'. [balooctl] Fix "index" command with already indexed, but moved file Summary: If I move a file (while baloo was not running) and perform `balooctl index` on new file (so that document Id is not changed), it won't update the path of the file, keeping the invalid entry in index. Explicitly tell Baloo that we want to update everything concerning this file. FIXED-IN: 5.52 Test Plan: 1) `echo "hello world" >~/file1` 2) `balooctl stop` 3) `mv ~/file1 ~/file2` 4) `balooctl start && balooctl index ~/file2` 5) `balooshow file2` and `balooshow -x <DOCUMENT_ID_OF_FILE>` The first command should show the right path, and the second command shouldn't complain about bug in Baloo (invalid index entry) Reviewers: #baloo, #frameworks, ngraham Reviewed By: #baloo, ngraham Subscribers: ngraham, kde-frameworks-devel Tags: #frameworks, #baloo Differential Revision: https://phabricator.kde.org/D15944 M +1 -1 src/tools/balooctl/indexer.cpp https://commits.kde.org/baloo/7e5c005e6a3b563013e2ba8cb9c8f1b282e6f7b2
Just tested sequence that I've proposed in bug description with baloo 5.52. It still yields the same result, baloosearch finds test file twice. The issue that Igor fixed must have been something else. His test plan is quite different and involves stopping baloo. The bug that I describe here is triggered when baloo is running nonstop. Please retest this with test plan that I provided in description https://bugs.kde.org/show_bug.cgi?id=397242#c0 exactly as it is.
That was supposed to be caused by the same issue. Sorry, I cannot reproduce it anymore. Your script (with NAME="testfile" and CONTENT="testtesttest", since I don't have pwgen) yields: ``` /home/eol/testfile_1.txt /home/eol/testfile_2.txt /home/eol/test.sh Elapsed: 0.299802 msecs renamed 'testfile_1.txt' -> 'testfile_2.txt' /home/eol/testfile_2.txt /home/eol/test.sh Elapsed: 0.277733 msecs ``` while monitor yields: ``` Indexing new files Indexing file content Indexing: /home/eol/testfile_1.txt: Ok Indexing: /home/eol/testfile_2.txt: Ok Idle Indexing modified files Indexing file content Indexing: /home/eol/testfile_2.txt: Ok ``` so everythings seems to be running smooth...
I'm running recent Arch linux and still able to reproduce this even with clean and fresh user. I have even recorded screencast: https://asciinema.org/a/CTun0j5byw1FsYwJvcfRw89mx I don't know why you can't reproduce. I suggest to test with random string like this: random_string() { head -c 100 /dev/urandom | md5sum | head -c 10 } NAME=$(random_string); CONTENT=$(random_string) echo "$CONTENT" > "${NAME}"_1.txt echo "$CONTENT" > "${NAME}"_2.txt sleep 5 baloosearch "${CONTENT}" echo mv -v "${NAME}"_1.txt "${NAME}"_2.txt sleep 5 baloosearch "${CONTENT}" I'm going to retest this in virtual machine on some public live CD to make sure I have perfectly reproducible test rig.
I runned http://cdimage.ubuntu.com/kubuntu/daily-live/current/disco-desktop-amd64.iso in VirtualBox and in this image baloo handles such renames as expected, bug is NOT reproducible inside VM. But it is STILL reproducible on my PC with Arch Linux. Both systems have $ balooctl --version baloo 5.56.0 I'm confused. Can't figure out what is it so special on my PC that keeps this issue.
It is indeed weird. What filesystem do you use? How does "balooctl monitor" react to those files?
I use ext4 on LUKS encrypted volume on LVM2. This issue gets weirder and weirder. When it gets successfully reproduced it looks in balooctl monitor like this: ``` Idle Indexing new files Indexing file content Indexing: /home/self/5fe45b3d45_1.txt: Ok Indexing: /home/self/5fe45b3d45_2.txt: Ok Idle ^C ``` There is no reaction to renaming. But several times attempts to reproduce lead to this: ``` Indexing new files Indexing file content Indexing: /home/self/e0f789d783_2.txt: Ok Idle ``` And only second file was present in baloosearch "$CONTENT" output. First one is ignored? Why? So I tried echo "$CONTENT" | tee "${NAME}"_{01..10}.txt And got once in balooctl monitor: ``` Press ctrl+c to stop monitoring File indexer is running Idle Indexing new files Indexing modified files Indexing file content Indexing: /home/self/e0f789d783_02.txt: Ok Indexing: /home/self/e0f789d783_03.txt: Ok Indexing: /home/self/e0f789d783_04.txt: Ok Indexing: /home/self/e0f789d783_05.txt: Ok Indexing: /home/self/e0f789d783_06.txt: Ok Indexing: /home/self/e0f789d783_07.txt: Ok Indexing: /home/self/e0f789d783_08.txt: Ok Indexing: /home/self/e0f789d783_09.txt: Ok Indexing: /home/self/e0f789d783_10.txt: Ok Idle ``` First file is not indexed! Again!
It might be intentional: IIRC, Baloo doesn't want to reindex the same file too often (so it won't kill your machine when it finds i.e. logs being continuously written)
When baloo does not notice renames, thats likely a bad inotify limit on Arch.