Hi, I was debugging an issue where the "Loading folder..." text (in the lower right corner of dolphin) was showing for a few seconds after pressing one of the columns to resort the folder. Yes, this happened to be my famous folder with 500.000 test entries (0 bytes each). The issue that i observed is being caused by dolphin itself. If you press the column name the .directory file in that specific folder gets updated with the new settings. That causes KDirWatch to emit a dirty signal which then causes KDirLister to completely re-list the directory because it's dirty. And that specific action causes the "Loading folder..." to pop up because it actually is doing that. Now i noticed this quite heavily because the test folder is so large, but this happens in every folder where you change a sort order and are not using the global view properties. In fact, anything that triggers a change in the .directory fill will cause the above to happen. So the .directory is the one to blame here. I can see why it's existing and i find it's features very useful so i don't think the feature should go. I do think the .directory file should be placed somewhere else. perhaps the .directory file should be stored in the local users .config/dolphinDirecoryProperties/... or .kde or .<whatever> with a filename that is the md5 of the full uri. so the name would become: md5(file:///home/user/bla) Perhaps even append .directory so you get: md5(file:///home/user/bla).directory And yes, a temporary solution would be not to use this and go for the global view properties. The above proposal is to keep the same functionality but stay not inside the folder that is currently being browsed. Note: i think it's not a good idea to store hidden config files in a folder where they are needed. I know that windows does it (.desktop, .folder and thumbsdb) and mac is doing it as well. But i really think we should try and avoid that and keep the folder as the user intended it. Note2: all of the above took me ~4 hours of deep debugging! Kind regards, Mark
Thanks for the report. IMHO, this is wishlist material. It wasn't my idea to create those .directory files. But AFAIK, it has been like that since the KDE 3 times, and changing it is not as trivial as you might think. I don't think that users will like it if the custom settings that are stored in their .directory files are lost, so we would have to check both the potential new storage location *and* the .directory file inside the directory for user settings. Implementing this is not straightforward, but probably quite error-prone. And if the only justification is that it causes problems in folders with half a million files, then I don't think that this should have a high priority (maybe something to consider for KDE 5/KF 5). Moreover, I have serious doubts that changing the storage location for the .directory files is the best way to fix the problem that you are seeing (see below). (In reply to comment #0) > Note2: all of the above took me ~4 hours of deep debugging! I hope that you have spent at least a small part of this time figuring out what KDirLister is doing, and if it does it in a smart way. Could you please share your findings with us? If you haven't done it yet, please run Dolphin in Callgrind and repeat the steps (possibly in a directory with N < 500000 files to make sure that you don't have to wait a few hours). I wouldn't be surprised if you find that KUrl::operator==() is called insanely often. At least that's what I found out some time ago when I tried to figure out why adding many files in a folder which already has many files takes long, even though I've eliminated many sources of O(N^2) behaviour in Dolphin recently. I have some ideas how this problem can be fixed, and I hope I'll manage to summarize them and send my findings to k-c-d and David in the next one or two weeks.
(In reply to comment #1) > Thanks for the report. IMHO, this is wishlist material. > > It wasn't my idea to create those .directory files. But AFAIK, it has been > like that since the KDE 3 times, and changing it is not as trivial as you > might think. I don't think that users will like it if the custom settings > that are stored in their .directory files are lost, so we would have to > check both the potential new storage location *and* the .directory file > inside the directory for user settings. Implementing this is not > straightforward, but probably quite error-prone. And if the only > justification is that it causes problems in folders with half a million > files, then I don't think that this should have a high priority (maybe > something to consider for KDE 5/KF 5). Well, we can obviously still support reading the .directory file if it's there. Just not write it there. But i agree, this should probably be KF5 material. > > Moreover, I have serious doubts that changing the storage location for the > .directory files is the best way to fix the problem that you are seeing (see > below). > > (In reply to comment #0) > > Note2: all of the above took me ~4 hours of deep debugging! > > I hope that you have spent at least a small part of this time figuring out > what KDirLister is doing, and if it does it in a smart way. Could you please > share your findings with us? If you haven't done it yet, please run Dolphin > in Callgrind and repeat the steps (possibly in a directory with N < 500000 > files to make sure that you don't have to wait a few hours). I sure can share my findings. I know exactly why the observed actions happen. KDirLister is not to blame here. I't actually a lot more low level then that in the linux kernel, specifically inotify. The KDirWatch class is watching a folder and gets messages (from inotify or whatever backend is selected, i believe that inotify is the default). It only gets a message like "this directory is dirty" and that's it. It doesn't get a message about which file is dirty or what changed at all. Thus KDirLister has to simply re-list all entries in a folder to update it even though no files might be added at all. It really is that simple. You can take a look at this link for some detailed inotify information: http://www.ibm.com/developerworks/library/l-inotify/ > > I wouldn't be surprised if you find that KUrl::operator==() is called > insanely often. At least that's what I found out some time ago when I tried > to figure out why adding many files in a folder which already has many files > takes long, even though I've eliminated many sources of O(N^2) behaviour in > Dolphin recently. I have some ideas how this problem can be fixed, and I > hope I'll manage to summarize them and send my findings to k-c-d and David > in the next one or two weeks. Well, KUrl might be part of it because it's used in KDirLister as well and if it re-lists a folder it creates N KFileItem object and each of them has a KUrl.. But that's not an issue. The actual issue is KDirLister not being smart enough to not see that no files have changed, but by the time it _can_ detect that it has already created all the new KFileItem objects. What we actually need is an "inotify" that gives us more details about what exactly is dirty. Note for all of the above. If i'm wrong on some (or all) of the above points, please do share :)
It looks like I wasn't clear enough. I do understand that KDirLister receives a signal from KDirWatch and has to take some sort of action because of that. (In reply to comment #2) > KDirLister is not to blame here. My question was: could it be that KDirLister does anything inefficient when it receives that signal and re-lists the directory? And that there would be no noticeable delay if that issue was fixed? If KDirLister's algorithm for reacting to changes in the directory is O(N^2), then KDirLister *is* to blame.
(In reply to comment #3) > If KDirLister's algorithm for reacting to changes in the directory is > O(N^2), then KDirLister *is* to blame. I'm not 100% sure. If memory serves me well then KDirLister is "just" filling a list. And i know from past debugging in that area that: - parsing the massive stream data that comes from the actual kio slave to construct the UDSEntry is taking up a lot of time. (this ends with a UDSEntryList) - filling a list with KFileItem objects is taking up another big chunk And there is not anything you can easily do about that other then rewriting the whole chain up to the kio SlaveBase class. Believe me, it's a very big task! Anyhow, the Big O complexity is not an issue here. It potentially could be and it likely could be improved but even that would only make the issue smaller and not fix the actual issue that is causing this in the first place. The actual issue is that KDirLister just gets a dirty signal without more information which causes a full directory re-list. That is the issue. Or rather, the limitation of inotify is the issue. However, it seems like we have been blessed by yet another file system monitor library in 2009: Fanotify http://lwn.net/Articles/339253/ I haven't done any fanotify testing yet, but from reading that page i get the impression that it provides the more detailed information that we need to fix this issue. Though that would be KF5 material :(
I tried it, and it seems that there is indeed no O(N^2) problem when the .directory file is modified.
This is very interesting: "When monitoring a directory, the events marked with an asterisk (*) above can occur for files in the directory, in which case the name field in the returned inotify_event structure identifies the name of the file within the directory." from http://linux.die.net/man/7/inotify Which leads me to think that you can in fact receive a dirty signal for only one file when monitoring an entire directory. I'm going to further investigate this issue to see if i'm right and in that case where KDirWatch might be going wrong. Or do you (Frank, David?) have any suggestion here?
This issue still is present in Dolphin 16.08 (Debian Stretch), by default the .directory files are created everywhere, littering the filesystem. There are many complaints elsewhere on the internet that repeat the suggested "common view properties for all folders" workaround: - https://forum.kde.org/viewtopic.php?f=223&t=119876 - https://forums.opensuse.org/showthread.php/454850-Prevent-Dolphin-From-Dropping-Files-Everywhere - https://superuser.com/questions/569549/is-it-possible-to-prevent-creation-of-directory-files Such a workaround does prevent the littering but only with a loss of functionality, so, it's not a good solution. Also I did notice the existence of ~/.local/share/dolphin/view_properties/local/**. It seems like there is a nested directory tree in here with some subset of the .directory files, used when Dolphin cannot write to the target directory. If Dolphin had an option to use this directory regardless of permission, that would resolve the issue for me,
Can anyone explain the actual negative impact for an average user (not someone who has 500,000 empty files in a folder) that comes from storing .directory files inside their folders? Windows and macOS both use per-directory settings files like these and it hasn't caused the end of the world there.
(In reply to Nate Graham from comment #8) > Can anyone explain the actual negative impact for an average user (not > someone who has 500,000 empty files in a folder) that comes from storing > .directory files inside their folders? Windows and macOS both use > per-directory settings files like these and it hasn't caused the end of the > world there. Guess i don't have to respond ;) (will do so anyhow) Yes, it's not the end of the world. Yes, it works just fine for the average user as is. 500.000 files is obviously a benchmark scenario, but it "points" as a possible issue. You can have two real fixes here. 1. Re-evaluate KDirLister (i don't know if it got smarter over the last 4 years, therefore look at it again) to not update the folder when the only change that caused the folder to be dirty is caused by updating the hidden config file. I'm not sure anymore why KDirWatch merely emitted a dirty signal.. inotify (the backend on linux) does show which files got changed so it should be possible to monitor for just the changes and ignore the said case. 2. Move the per directory config out of the directory and into either some "shadow" folder (same hiërarchie, just under a different path), or follow the same principle as is done right now when making thumbnails (md5 the full path + filename). Point 1 is probably the most desirable fix, but could be tricky to get right. Is anything needed at all? Nope. It's merely a "would be nice to have" request.
Thanks Mark. So if I'm understanding correctly, this is a possible performance issue with exceptionally large directories? I agree that if this is worth fixing, option 1 (optimizing KDirLister) is the right approach.
(In reply to Nate Graham from comment #8) > Can anyone explain the actual negative impact for an average user (not > someone who has 500,000 empty files in a folder) that comes from storing > .directory files inside their folders? Windows and macOS both use > per-directory settings files like these and it hasn't caused the end of the > world there. That's me - Other people's bad behaviour is no excuse for our own, - Windows mostly stopped doing this over 10 years ago, Vista doesn't create thumbs.db files any more (there's still desktop.ini, but everything else now goes in ~ or NTFS ADS forks). - Just today I had to help someone asking me what this DS_Store stuff is when they received a .zip file created on a mac. It's clutter and a nuisance. You have changed the bug to be about a specific KIO issue, that's great, but i subscribed to this bug under its original title and culprit (Dolphin) because it was about getting rid of .directory files, which i think clutter up the place (e.g. shared network drives and zipped directories), not every OS hides dotfiles. Dolphin demonstrably supports not having the .directory files in the directory itself (see #7 ), for me personally the issue would be resolved by making this a configurable preference.
(In reply to Nate Graham from comment #10) > Thanks Mark. So if I'm understanding correctly, this is a possible > performance issue with exceptionally large directories? I agree that if this > is worth fixing, option 1 (optimizing KDirLister) is the right approach. Well, kind of. It's a bit of a grey area. It's weird to let KDirLister exclude .directory file changes. Some would consider that a bug while that very same "fix" would stop it to update the folder when the .directory file got changed in dolphin. Patching KDirLister for ".directory" handling certainly is a no-go! That file is for Dolphin and KDirLister should not know a thing about that. A "perhaps acceptable solution" on that side is something like a general "KDirLister::excludeFromMonitor(QString expression)" where the string is then fed into a QRegularExpression and anything matching the expression is excluded from being monitored. That "could" work... Feel free to implement that and hope it gets accepted ;) Another issue that will arise is different file system monitor backends. I know inotify supports what we want, but what about the range of others on linux alone? Then you still have those other platforms (Windows, Mac) they need to follow suit. The change for this is really not trivial it seems. In conclusion for the above.Having that in KDirLister would be a bonus! A big plus, but not a requirement. I agree with mappu04 that the .directory file should not be in an individual directory. Why would i need that file when i happen to copy a folder from my local drive to a completely different machine where i might not even run Dolphin. And if i were, i would certainly _not_ want that copy to look different on that other machine just because i accidentally copied over a .directory file! So, could you please set the title back to what it was and keep this about getting rid of the .directory file from a folder. It's fine if you don't agree, but then this issue should be a "won't fix".
Fair enough! That's for the clarification, guys. Often it's really hard to tell what a bug is actually about when there it's been used to discuss multiple issues and there's a lot of chatter.
*** Bug 17289 has been marked as a duplicate of this bug. ***
*** Bug 404629 has been marked as a duplicate of this bug. ***
Closed (not a bug): - bug 393960 - Dolphin should not write a .directory file in any directory that is used by IndexedDB
@Nate: note that extended attributes are UNIX-specific. I think that storing the view properties in $XDG_DATA_HOME (which dolphin already does for remote or non-writable folders) is our best way forward. Another problem is that KFileItem in KIO assumes that folders could store .directory files, so that should be addressed as well.
Git commit d6c086ad04adf306aea7cfc6961af805d8c4350e by Elvis Angelaccio. Committed on 17/03/2019 at 15:43. Pushed by elvisangelaccio into branch 'Applications/19.04'. Make global view properties the default setting Summary: Automatic creation of .directory files is one of the most complained about things in dolphin. These days applications are supposed to store application-specific data within $XDG_DATA_HOME. Long-term that's the proper solution for this issue, but for now we can at least flip the default so that we no longer create .directory files out of the box. A tooltip is also added as a kind of warning for the users who want to change the setting. Ideally a proper warning should be shown using a KMessageWidget or KMessageBox, but neither play too well with the setting dialog workflow used by dolphin. Reviewers: #dolphin, #vdg, ngraham Reviewed By: #dolphin, #vdg, ngraham Subscribers: ngraham, kfm-devel Tags: #dolphin Differential Revision: https://phabricator.kde.org/D19661 M +1 -1 src/settings/dolphin_generalsettings.kcfg M +5 -4 src/settings/general/behaviorsettingspage.cpp https://commits.kde.org/dolphin/d6c086ad04adf306aea7cfc6961af805d8c4350e
(In reply to Elvis Angelaccio from comment #18) > Git commit d6c086ad04adf306aea7cfc6961af805d8c4350e by Elvis Angelaccio. > Committed on 17/03/2019 at 15:43. > Pushed by elvisangelaccio into branch 'Applications/19.04'. > > Make global view properties the default setting > > Summary: > Automatic creation of .directory files is one of the most > complained about things in dolphin. > > These days applications are supposed to store application-specific data > within $XDG_DATA_HOME. Long-term that's the proper solution for this > issue, but for now we can at least flip the default so that we no longer > create .directory files out of the box. > > A tooltip is also added as a kind of warning for the users who want to > change the setting. Ideally a proper warning should be shown using a > KMessageWidget or KMessageBox, but neither play too well with the > setting dialog workflow used by dolphin. > > Reviewers: #dolphin, #vdg, ngraham > > Reviewed By: #dolphin, #vdg, ngraham > > Subscribers: ngraham, kfm-devel > > Tags: #dolphin > > Differential Revision: https://phabricator.kde.org/D19661 > > M +1 -1 src/settings/dolphin_generalsettings.kcfg > M +5 -4 src/settings/general/behaviorsettingspage.cpp > > https://commits.kde.org/dolphin/d6c086ad04adf306aea7cfc6961af805d8c4350e So thats one of the reasons why my per-folder settings have been broken since updating.. It's all global now, no more local settings or option for it. All the View Properties related to it are gone too..
(In reply to Diana from comment #19) > (In reply to Elvis Angelaccio from comment #18) > > Git commit d6c086ad04adf306aea7cfc6961af805d8c4350e by Elvis Angelaccio. > > Committed on 17/03/2019 at 15:43. > > Pushed by elvisangelaccio into branch 'Applications/19.04'. > > > > Make global view properties the default setting > > > > Summary: > > Automatic creation of .directory files is one of the most > > complained about things in dolphin. > > > > These days applications are supposed to store application-specific data > > within $XDG_DATA_HOME. Long-term that's the proper solution for this > > issue, but for now we can at least flip the default so that we no longer > > create .directory files out of the box. > > > > A tooltip is also added as a kind of warning for the users who want to > > change the setting. Ideally a proper warning should be shown using a > > KMessageWidget or KMessageBox, but neither play too well with the > > setting dialog workflow used by dolphin. > > > > Reviewers: #dolphin, #vdg, ngraham > > > > Reviewed By: #dolphin, #vdg, ngraham > > > > Subscribers: ngraham, kfm-devel > > > > Tags: #dolphin > > > > Differential Revision: https://phabricator.kde.org/D19661 > > > > M +1 -1 src/settings/dolphin_generalsettings.kcfg > > M +5 -4 src/settings/general/behaviorsettingspage.cpp > > > > https://commits.kde.org/dolphin/d6c086ad04adf306aea7cfc6961af805d8c4350e > > So thats one of the reasons why my per-folder settings have been broken > since updating.. It's all global now, no more local settings or option for > it. > > All the View Properties related to it are gone too.. oops.. found the setting. Guess it's not broken.
(In reply to Elvis Angelaccio from comment #17) > @Nate: note that extended attributes are UNIX-specific. I think that storing > the view properties in $XDG_DATA_HOME (which dolphin already does for remote > or non-writable folders) is our best way forward. > What happens with this approach when I move a directory in another application (like for example Nautilus or with mv?)
A possibly relevant merge request was started @ https://invent.kde.org/system/dolphin/-/merge_requests/573