When accessing a SMB share (mounted via a VPN across the Internet), nepomuk servicestub filewatcher digs the entire share (a few hundred TB) consuming all my upload bandwidth. Nepomuk file indexer is disabled and Desktop Query is limited to a single empty folder in my home directory (just to make sure it does not index anything) and removable media should be ignored. To me it looks like the filewatch part is ignoring my setting. blub@larry $ lsof /mnt/SMB-Share/share/ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME nepomukse 4155 blub 17r DIR 0,35 0 320970531 /mnt/SMB-Share/share/CRM nepomukse 4155 blub 18r DIR 0,35 0 69227841 /mnt/SMB-Share/share nepomukse 4155 blub 19r DIR 0,35 0 69228137 /mnt/SMB-Share/share/CRM/expenses nepomukse 4155 blub 20r DIR 0,35 0 69228154 /mnt/SMB-Share/share/CRM/expenses/adrian nepomukse 4155 blub 25r DIR 0,35 0 69228155 /mnt/SMB-Share/share/CRM/expenses/adrian/CASH 09:20:25 [~] blub@larry $ ps -ef | grep nepomuk blub 3908 3876 0 07:24 ? 00:00:10 /usr/bin/akonadi_nepomuk_feeder --identifier akonadi_nepomuk_feeder blub 3954 1 0 07:24 ? 00:00:00 /usr/bin/nepomukserver blub 3957 3954 5 07:24 ? 00:06:17 /usr/bin/nepomukservicestub nepomukstorage blub 4152 3954 0 07:24 ? 00:00:00 /usr/bin/nepomukservicestub nepomukfileindexer blub 4153 3954 0 07:24 ? 00:00:00 /usr/bin/nepomukservicestub nepomukqueryservice blub 4154 3954 0 07:24 ? 00:00:00 /usr/bin/nepomukservicestub nepomukbackupsync blub 4155 3954 0 07:24 ? 00:00:02 /usr/bin/nepomukservicestub nepomukfilewatch blub 4244 1 0 07:24 ? 00:00:00 /usr/bin/nepomukcontroller -session 10e5617272000133423828300000072100027_1350295774_133445 blub 22097 7518 0 09:20 pts/1 00:00:00 grep --colour=auto nepomuk Even when I select "suspend file indexing" nepomukfileindexer continues to dig the share. I recently delete all related configs and started from scratch, but it didn't help so I consider this as an bug. I even created a new user, started KDE, run lsof and saw the same - nepomuk indexing the share... all with default settings. Cheers, Bjoern Reproducible: Always Steps to Reproduce: 1. Mount SMB share 2. Access SMB share with Dolphin 3. lsof /path/to/share/somefolder 4. Watch nepomuk indexing the share Actual Results: Nepomuk indexes the share recursively eating all my bandwidth Expected Results: According to my configuration, nepomuk should not index anything since fileindexing is disabled. In Fact I think filindexing should stay away from anything mounted via network per default!
Well, I don't even have to access the share with Dolphin. I just have mount it (via automount) and the nightmare will start. Cheers, Bjoern
Same here. I have posted this bug here some time ago: http://sourceforge.net/apps/trac/oscaf/ticket/141 Unfortunately it was not a correct place. This bug, by the way, is not limited only to NAS mounts. It sometimes happens when I connect my iPod (indexing removable devices is off). As a result of this there is no way to unmount the device. I have to kill the nepomukserver. The bug is terrible and tortures me on Ubuntu 12.04 and Fedora Core F17. So I have to keep nepomuk completely disabled.
Okay, now this gets really annoying... Even after hours when the VPN is disconnected, nepomukservicestub still keeps the share busy although it should stall at some point an realize that the share is dead. Saving an attachment from an e-mail within Kontact took me >10 minutes since I had to wait for an timeout for each action I did in the folder selection dialog. Please fix it! Make nepomukservicestub aware of network mounted shares, make it smarter, or at least make it stick to the configuration. Or if this is even to complicated, let it die once it cant proceed indexing because of a stalled share, or whatever, but please KILL it somehow! Let it die! Kind regards, Bjoern
I recognized it too. System: openSUSE 12.2 x86_64 KDE 4.9.3 NFS shares After login nepomuk start indexing the shares, in the config only my local Documents Folder is activated for indexing.
So since nobody seems to care I started to investigate... (also I do not have the time, this kills my nerves and has to be fixed somehow, even if I start to hack around in the code) The problem seems to be "nepomukfilewatch" service. Once disabled, the described problem is gone. To disable nepomukfilewatch execute: qdbus org.kde.NepomukServer /servicemanager stopService nepomukfilewatch Just add it to the KDE autostart and you are good to go. I didn't find any drawback for my work-flow by disabling nepomukfilewatch. If I find some more time I'll see if I can add a simple hack to just ignore "network folders". A second thing that bugs me: Why the hack has Dolphin to count the items inside a folder and there is no way to disable this! This eats also a lot of time when working on remote folders *sigh* Looks like the Devs are all sitting on a machine with a high speed SSD-RAID and nobody ever thinks about performance anymore... Cheers, Bjoern
*** This bug has been confirmed by popular vote. ***
I just had the same bug with a simple usb stick in KDE 4.9.90 - I couldn't unmount it because a nepomuk process was blocking it, and re-spawned a new one as soon as I killed it. File Indexing is set to "Ignore all removable media" and generally only in use for one particular folder inside ~. Suspending file indexing didn't solve it.
It seems the problem has worsened with 4.9.90 - I can't remember that happening in 4.9. But I haven't yet found a sure way to reproduce it. Right now it happened after copying one file to the usb stick. nepomuk file indexing is suspended because I'm running on battery, nevertheless it blocks me from unmounting the stick.
(In reply to comment #8) > It seems the problem has worsened with 4.9.90 - I can't remember that > happening in 4.9. But I haven't yet found a sure way to reproduce it. Right > now it happened after copying one file to the usb stick. nepomuk file > indexing is suspended because I'm running on battery, nevertheless it blocks > me from unmounting the stick. Could you define what you mean by 4.9.90? Do you mean the current master or Beta1? Cause I have fixed the umounting, but the fix will only be in RC1 - https://bugs.kde.org/show_bug.cgi?id=304943 @Everyone else: Please don't mix the two bugs up. One bug is about removable media being rendered unmountable due to the file watch service - That is bug 304943. This bug is about how the file watch service installs watches in all the directories of network shares/removable media, even though it was told not to index, and thereby consumes a lot of bandwidth. This second bug is still there. I'm not sure how to fix it. People could tag/rate any of the files in the mounted directory. We need those watches to make sure the tags are not lost if those files are ever moved. Another case could be the moving of files to the network share. One would expect the metadata to still be there. Without the watches, the meta-data would disappear.
(In reply to comment #9) > Could you define what you mean by 4.9.90? Do you mean the current master or > Beta1? Cause I have fixed the umounting, but the fix will only be in RC1 - > https://bugs.kde.org/show_bug.cgi?id=304943 That is beta2. Thx for the fix!
(In reply to comment #9) > This second bug is still there. I'm not sure how to fix it. People could > tag/rate any of the files in the mounted directory. We need those watches to > make sure the tags are not lost if those files are ever moved. Another case > could be the moving of files to the network share. One would expect the > metadata to still be there. Without the watches, the meta-data would > disappear. Wouldn't it be an option to let the user decide? Split the functionality in indexing local files and remote files. Together with a small hint that the meta-data would be lost, if the file is on a not indexed remote share (or moved to it) would clarify the behavior. Sitting on a LAN with GBit isn't such a big deal, but having shares from a few web servers mounted over a 6MBit WAN is a problem. Or Even easier - with the current selection options it should work. If I activate only a single directory for indexing (lets say "Documents"), then I would expect that only files in this directory are indexed. And if I move files from this directory to another location I expect that the meta-data disappear. If I move the file back to this directory I would expect that the meta-data appears again. This could be based on another option where I can decide how long meta-data is kept after moving/deleting a file, until it is finally cleared from the meta-data storage. At the moment the behavior is irritating. Also your description implies that the whole local hard disk(s) must be indexed to keep track of moving local files. And I wouldn't expect that a task (under Linux) do something that I never told him to do.
It's not just about indexed files, if the indexed information is lost, we can re-index the file. Not a problem. It's the tags and ratings that should not be lost. Tags and Rating have nothing to do with the indexed directories. Do you see the problem?
I understand the problem - sorry for mixing-up indexing and meta-data. And I really appreciate the effort and work that you are taking to keep the meta-data. I can imaging that this isn't easy. But... ;-) Why not giving the user a more fine grained control over the handling of meta-data like tags, ratings, keywords and so on. At least an exclusion option of files and/or folders. Together with an expiration option of the meta-data storage. I don't say it's a perfect one - but it would be a configurable one. Cheers Regi
I second this. Add a option to generally "Ignore network shares" and an option to exclude folders. Additionally to this respect the indexing options and limit all the metadata stuff to files in this directory. I don't think users expect metadata to be available outside the specified folder. Finally add a warning that metadata is lost if a file is moved somewhere where but not within the specified path to index. The current behavior is a bummer for KDE in a corporate environment where you share data cross networks... Cheers, Bjoern
> This second bug is still there. I'm not sure how to fix it. People could > tag/rate any of the files in the mounted directory. We need those watches to > make sure the tags are not lost if those files are ever moved. Another case > could be the moving of files to the network share. One would expect the > metadata to still be there. Without the watches, the meta-data would > disappear. I do not see how it can possibly work even in an ideal situation. Suppose that someone has some metadata on a file in a local directory and then moved the file to a NAS. Is the watch necessary to change association of this metadata from local file to file on a removable media? How would it help if the media is actually removed afterwards? If you mount it on another computer the metadata will not be available there. Even if you mount it on the same computer but using a different path (to the NAS) or after someone changed some sort of ID of the removable disk then you won't see that metadata again, will you? Isn't it more sensible to store the metadata itself or some sort of "metadata ID" directly in the files using extended attributes? Then the metadata or its marker will travel with the file without any special work. And I realise that some file systems do not support extended attributes. But noone grieves that elephants do not fly. So following this approach the metadata would be safely processed within a proper Linux ecosystem (and maybe further) and won't work only in some weird cases where it does not matter much anyway. And frankly, while this bug is still there many people do not care how KDE's semantic desktop works. In fact, the first thing they do after installing KDE is disabling it completely.
(In reply to comment #15) > > This second bug is still there. I'm not sure how to fix it. People could > > tag/rate any of the files in the mounted directory. We need those watches to > > make sure the tags are not lost if those files are ever moved. Another case > > could be the moving of files to the network share. One would expect the > > metadata to still be there. Without the watches, the meta-data would > > disappear. > > I do not see how it can possibly work even in an ideal situation. > > Suppose that someone has some metadata on a file in a local directory and > then moved the file to a NAS. Is the watch necessary to change association > of this metadata from local file to file on a removable media? How would it > help if the media is actually removed afterwards? If you mount it on another > computer the metadata will not be available there. Even if you mount it on > the same computer but using a different path (to the NAS) or after someone > changed some sort of ID of the removable disk then you won't see that > metadata again, will you? Different mounted path: yes. Though I have never tested it out. If you change the id of the removable disk. Then no. How about if the watches are not added for network shares? But are still added for other removable media? I could also provide options to enable a config file to disable it completely. I cannot add new strings, but I could document all these hidden options on the userbase? I hope you understand why I'm slightly reluctant on removing this entirely. > > Isn't it more sensible to store the metadata itself or some sort of > "metadata ID" directly in the files using extended attributes? Then the > metadata or its marker will travel with the file without any special work. > And I realise that some file systems do not support extended attributes. But > noone grieves that elephants do not fly. So following this approach the > metadata would be safely processed within a proper Linux ecosystem (and > maybe further) and won't work only in some weird cases where it does not > matter much anyway. That I agree should be done, but it would take considerable effort. It would amount to a new feature. That it not something I can do for 4.10. Lets focus on what I can do? > > And frankly, while this bug is still there many people do not care how KDE's > semantic desktop works. In fact, the first thing they do after installing > KDE is disabling it completely. Well, I have we can improve on that.
Hi, to give you an impression of how intrusive this feature is at the moment on my system, I attached an image of the CPU/Network-Graph right after login into KDE. System: Phenom 2 Quad core, with 8GB Ram and 1GB LAN speed. With a small NFS-Share of my Dev-Server mounted (approx 2.6 GB Size / 20,000 files). BTW: Production-Server Shares I can't mount - If I do so, monitoring jumps in ;-) Marker 1 - System start-up Marker 2 - Tried to start dolphin, giving up because disk activity makes it too slow Marker 3 - I stop all activities to speed up nepomuk (25% CPU usage means 1 Core with 100%) Marker 4 - Finished, I can start to work Between Marker 1 and Marker 4 it took nearly 10 Minutes. Please consider a solution, as Denys said most people I know disable it completely - even it's a great feature. Cheers Regi
Created attachment 75833 [details] CPU/Network Graph
> Different mounted path: yes. Though I have never tested it out. > If you change the id of the removable disk. Then no. > > How about if the watches are not added for network shares? But are still > added for other removable media? I could also provide options to enable a > config file to disable it completely. I cannot add new strings, but I could > document all these hidden options on the userbase? This would certainly be a good thing. > I hope you understand why I'm slightly reluctant on removing this entirely. Of course. > That I agree should be done, but it would take considerable effort. It would > amount to a new feature. That it not something I can do for 4.10. Lets focus > on what I can do? Sure. I thought a bit more on this and it seems that while using extended attributes would be more reliable, it also has its limitations. While metadata would follow the files, users will probably need to have write permissions to update it.
Would it be possible for someone to test the patch out? https://git.reviewboard.kde.org/r/108047/ I want to make sure it works, and I don't have any network shares. (Nor do I want to set one up)
I am more than happy to test it once I am at home. (3-4 days) Thanks a lot, Bjoern
Git commit 19f9f3fcc094ad65d4b11eb2fd1d5fc20c3a255e by Vishesh Handa. Committed on 31/12/2012 at 10:26. Pushed by vhanda into branch 'KDE/4.10'. FileWatch: Do not always add inotify watches for removable media Not everyone wants watches to be added to the removable media. They are okay with loosing tags/ratings and in removable media. Also, certain users do not watch watches ever added for Network Shares - It results in a large amount of network load which is not required. The following options can disable watches in nepomukstrigirc - [RemovableMedia] add watches=true add watches network share=false By default watches are never added to network shares, and they are always added for removable media. REVIEW: 108047 M +20 -2 services/filewatch/nepomukfilewatch.cpp http://commits.kde.org/nepomuk-core/19f9f3fcc094ad65d4b11eb2fd1d5fc20c3a255e
(In reply to comment #20) > Would it be possible for someone to test the patch out? > > https://git.reviewboard.kde.org/r/108047/ > > I want to make sure it works, and I don't have any network shares. (Nor do I > want to set one up) Tested and it works for me. It does no longer index the network share (extensive tests done) or USB-Stick (Quick one shot test) with the following config: [RemovableMedia] add watches=false add watches network share=false I would really like to see these new to options to go into the "Nepomuk/Strigi Server Configuration" panel under the KDE System-Settings. Thanks a lot! Cheers, Bjoern
(In reply to comment #20) Yep - did a quick test in a VirtualBox with an USB-Drive and a NFS-Share. Looks good. Great job.
*** Bug 281450 has been marked as a duplicate of this bug. ***
It seems that this bug is back. Tested on Archlinux: Linux 3.8.3-2-ARCH #1 SMP PREEMPT Sun Mar 17 13:04:22 CET 2013 x86_64 GNU/Linux My version KDE 4.10.1. Fresh install of KDE (deleted all .kde4, .local and .config files before loging in). Configuration: Configuration Settings -> Nepomuk server -> Indexing: "ignore all removable devices" ... -> Custom folders: I've left only folders inside /home checked How to reproduce: 1) Place an USB drive copy, copy some video files to it. 2) Try to "safely remove" from Dolphing or from "available devices" on the system tray. Message says impossible because there is a program accessing it. 3) From the command line, issuing lsof /dev/sdd1 shows nepomuk. Not even root can unmount the device. Killing all nepomuk related processes does'nt help, because they get respawned. Question: do I open another bug or would someone reopen this same?
(In reply to comment #26) > It seems that this bug is back. > > Tested on Archlinux: > Linux 3.8.3-2-ARCH #1 SMP PREEMPT Sun Mar 17 13:04:22 CET 2013 x86_64 > GNU/Linux > > My version KDE 4.10.1. Fresh install of KDE (deleted all .kde4, .local and > .config files before loging in). > > Configuration: > Configuration Settings -> Nepomuk server -> Indexing: "ignore all removable > devices" > ... -> Custom folders: I've left only folders inside /home checked > > How to reproduce: > 1) Place an USB drive copy, copy some video files to it. > 2) Try to "safely remove" from Dolphing or from "available devices" on the > system tray. Message says impossible because there is a program accessing it. > 3) From the command line, issuing lsof /dev/sdd1 shows nepomuk. > > Not even root can unmount the device. Killing all nepomuk related processes > does'nt help, because they get respawned. > > Question: do I open another bug or would someone reopen this same? What you're describing is bug 304943. Also, you might want to read this - http://userbase.kde.org/Nepomuk/FileIndexer
Thanks for the pointer. Did anyone create the other bug report (about nupomuk ignoring the indexing settings)?