KDE Neon 5.10, KDE Frameworks 5.37.0, Qt 5.9.1, Dolphin 17.08.0 Source filesystem: ext4 Target filesystem: NTFS Prepare a copy test folder with 5 large files like this: # mkdir copytest # cd copytest/ # for i in `seq -w 1 5`; do dd if=/dev/urandom of=file-$i.bin bs=1M count=4196; done # cd .. First observation: if Dolphin is open on 'copytest' during the above operation, it hangs until the operation is finished. Copy test folder with the cp command like this: # date; cp -r copytest/ /media/user/ExternUSBDrive/; sync; date On my PC I got: sun 10 sep 2017, 13.40.14, CEST sun 10 sep 2017, 13.47.36, CEST Total time: 7 minutes, 22 seconds Do the same operation with Dolphin: - right click on 'copytest' -> Copy - go to external USB drive - right click -> Paste - minimize Dolphin - don't do anything else and watch the green bar slowly progressing Total time: 31 minutes, 12 seconds The CPU usage is ridicolous too: cp stays around 20%, while Dolphin stays around 40% This is an example with just 20GB of data, which is nothing nowadays. Sometimes it occurs to me to copy 150GB of data, which would mean something like 53 minutes (cp) against more than 4 hours (Dolphin) I suspect there is a severe issue with KDE filesystem watchers, which is the only thing that can make the difference in such a basic operation where you just need to do fread -> fwrite. However, I'd expect that minimizing Dolphin would disable every FS watcher.
I forgot to mention that the target filesystem is NTFS on an external USB 2 3.5" hard disk
2 weeks and no signs of life for such an epic bug. Wow.
Almost 3 months and nobody cares about this. No comments, no status change. No wonder why KDE is so buggy when developers don't actually listen to their software users. I use KDE since 15 years and instead of improving, KDE is degrading into something unusable.
I understand your frustration, but aiming verbal abuse at the people who you are expecting to perform work for free on an item of your choice isn't the best way to get it done. There are currently 27,062 open bugs across all of KDE[1]. This seems like an important one, to be sure, but it competes with 27,061 others. If you want to get this done faster, you're always welcome to submit a patch using http://phabricator.kde.org/. If you don't feel up to the task, perhaps you could locate someone else willing and able to have a look? Or polish your own C++ and Qt skills. 4 months ago I had never written a line or c++ in my life, yet by today I've had dozens of patches accepted: https://phabricator.kde.org/people/commits/11806/ But please understand that FOSS projects like this one (And GNOME too, to be fair) are always limited for resources. The absolute best way to improve this situation is to lend some resources of your own to the effort. Possible avenues include submitting patches, screening bugs, donating financially[2], etc. But posting anger and negativity in bug reports works against your desire because it makes people *less* likely to do the work. I am an unpaid volunteer. I am choosing to screen bugs for the benefit of others instead of playing with my kids right now on this lovely Saturday morning. Do you think it makes me want to screen bugs more or less when I wake up to an email notification that people like me "don't actually listen to their software users" and that the project I choose to work on for no money "is degrading into something unusable" ? Don't let cynicism and negativity drag you down. Nothing improves without effort. There's a huge amount to do here--climb aboard, and we'd be happy to have you! https://www.kde.org/get-involved https://www.kde.org/develop [1] https://bugs.kde.org/weekly-bug-summary.cgi [2] https://www.kde.org/donations
The main difference between cp/mv and file.so (copying/moving from dolphin) to a hdd ntfs partition, as strace tells, is that: cp/mv only does this loop: * read from one file and write to another. while file.so does this loop: * read from one file, write to another file and stat /etc/localtime 5 times.
With iotop or ksysguard and plots about I/O, there are huge differences. with cp/mv, no process is waiting for io, with dolphin plasma spends a lot of time waiting for io (column io> value 99%). with cp/mv, the i/o plots are more or less constant at some i/o speed , but with dolphin, at the beginning there are "mountains" and later on only peaks. In the code: * the cp/mv copy buffer is 128Kib allocated in the heap and is memory aligned * the current buffer used by dolphin is 32Kib in the stack and is not guaranteed to be memory aligned. I've tried changing the buffer to the heap, with 128Kib, and aligned and it improves a little bit the problem at the beginning, but after a while the performance drops and the peaks are shown again in ksysguard.
Git commit f7e00b40a6d35fbfe536cdb99d6f22b643676e5e by Jaime Torres. Committed on 14/01/2018 at 12:39. Pushed by jtamate into branch 'master'. Don't stat(/etc/localtime) between read() and write() copying files Summary: Unfortunately, QDateTime::currentDateTime() checks /etc/localtime each time it is called. Chaning to QElapsedTime, no check of /etc/localtime. Reproducing bug 384561, the strace of file.so was something like: read(), stat(/etc/localtime), stat(/etc/localtime), stat(/etc/localtime), stat(/etc/localtime), stat(/etc/localtime), write(), read() ...... Now it is: read(), write() It also reduces the cpu in io/wait around 10% in a debug build. Reviewers: #frameworks, dfaure Reviewed By: dfaure Subscribers: broulik, ngraham, #dolphin Tags: #frameworks Differential Revision: https://phabricator.kde.org/D9844 M +13 -12 src/core/slavebase.cpp https://commits.kde.org/kio/f7e00b40a6d35fbfe536cdb99d6f22b643676e5e
Git commit 240a9c1f01d0f11638be61e0869a494c7f3ab697 by Jaime Torres. Committed on 15/01/2018 at 18:39. Pushed by jtamate into branch 'master'. Code de-duplication in byteSize(double size) Summary: This code is used to pretty print the amount of data copied. Replace this duplicated code by calling formatByteSize from KFormat. Test Plan: Reviewers: #frameworks, mwolff Reviewed By: mwolff Subscribers: ngraham, cfeck, mwolff, broulik Tags: #frameworks Differential Revision: https://phabricator.kde.org/D9770 M +3 -25 src/kjobtrackerformatters.cpp https://commits.kde.org/kjobwidgets/240a9c1f01d0f11638be61e0869a494c7f3ab697
@Massimo: It would be nice as feedback, if the upcoming KDE Frameworks v5.43 fixes this for you - just to be sure your problem is truly fixed :-)
"3 months and nobody cares" you say? HAHAHAHAHA https://bugs.kde.org/show_bug.cgi?id=342056 3 years :)
(In reply to Alexander Nestorov from comment #10) > "3 months and nobody cares" you say? HAHAHAHAHA > > https://bugs.kde.org/show_bug.cgi?id=342056 > > 3 years :) Yes, I know it feels like a long time, but please see Nate's comment above. A simple thank you would go a long way... Try to imagine what it feels like to be a developer, who invests a nontrivial amount of time into fixing this bug only to get a snarky "it took 3 years" comment.
I just want you to notice that not every developer has a privative filesystem available, and it took me "only" two weeks to investigate and reduce the time spent copying files, and is not over yet. The other bug, that I noticed sometimes reproducing this one, is even harder to fix.
First and foremost, @Jaime Torres: thanks a lot for looking into this ! > @Massimo: It would be nice as feedback, if the upcoming KDE Frameworks v5.43 > fixes this for you - just to be sure your problem is truly fixed :-) Will surely do. I tried to avoid flaming this thread, but now I'd like to clarify my comment #3. I never asked for a fix in 3 months. I am a (Qt) developer too and I know how this works. However, as a developer I try to quickly identify the real issues from the fake ones. In a few days/weeks I try at least to reproduce the issue and give it a priority. This way I know where I have to concentrate my efforts. Moreover if I'm working on something, I tend to forget everything else, so "marking" an issue with a priority helps my memory months later. What I complained about is that in 3 months no one changed the status of this issue to 'confirmed', meaning (to me) no one tried to reproduce it. This is the best way to loose an issue report that can help KDE to get better among other 27000+. And because of this, I found Nate's comment quite ridiculous: > There are currently 27,062 open bugs across all of KDE[1]. This seems like an important one, to be sure, but it competes with 27,061 others. This means there is no priority assignment and all those 27062 bugs have the same weight. As for me fixing the issue...no way ! I already contributed to KDE many years ago (Kate's Symbol Viewer plugin is mine) and I know how difficult is even to prepare the KDE build environment. Plus, I am already swamped with my project issues, and I believe I already did my job -as user- by dedicating time to open this issue report as much accurately as possible. I am frustrated, yes, because I use KDE since 15 years and as a user it hurts me to see how Plasma is still not in shape, even after a dozen of minors. I see changelogs of new versions saying "new breeze theme, new panel design, etc" and it makes me angry when I know there are core issues to be fixed first. In conclusion, no harsh from my side. I know for sure there are excellent developers behind KDE that dedicate an awful amount of their personal time to this project. I only wish there could be a better priority system in place.
We get 30-40 new bugs every day. Since you are a developer, you know how important bug screening is. So why don't you help us screen the bugs? That's what I do (in addition to submitting patches and organizing a KDE-wide effort to address longstanding or high-impact bugs, like this one). I do it with a smile, too, because negativity is destructive. This isn't a commercial company. People donate their time because they love the project. Filing good bug reports is very important--and we appreciate that very much! But as you've observed, we have more than 27,000 bugs and we need help to triage them. With your development expertise, I'm sure you could submit some patches, too! And if you need a hand setting up a KDE development environment, I'd be happy to help; just shoot me an email.
Thank you for your report and for your help making KDE software great. Bugzilla, as far as I know, does have priorities: by project, by component, by user votes, by number of duplicates, by keywords, by tags, by date, ... The main problem is that there are lots of duplicated bugs that right now only humans mark as such, even the previous search as you type a new bug is insufficient (I know because I've reported duplicated bugs myself), and this manual task consumes a lot of time. With the improvements of the so called deep learning, is there any free software to search for duplicated reports that is as good as humans or better?
Yes, unifying duplicates is a big job. I've done a lot of bug screening in my time, in Bugzilla as well as others, and I doubt hunting for duplicates can be done by anyone other than a human being. In general, KDE suffers from a lack of what you might call corporate professionalism: we don't have many paid programmers or any paid bug screeners; we don't have project managers to keep projects on task; we don't categorize bugs based on priority; etc. There are certain projects like Plasmashell that are run closer to this model, but a lot of KDE is run like a volunteer organization because, well, that's what it is! My personal opinion is that a bit more discipline would really help us, if we can apply it without making the project un-fun to work on, which would defeat the purpose by repelling volunteers. That's mostly a cultural matter--getting people to screen bugs for projects they work on and use the Priority field--but I think having at least one overall release manager would be hugely beneficial, as that person would have a high level overview of the whole community and could help keep things on track, which ultimately increases the quality of our products and generates more excitement and volunteer contribution.
Git commit 8d73867b3d4339ec026f92c4c2e89c265249aeb2 by Jaime Torres. Committed on 07/02/2018 at 18:05. Pushed by jtamate into branch 'master'. Don't stat(/etc/localtime) between read() and write() copying files Summary: Unfortunately, QDateTime::currentDateTime() checks /etc/localtime each time it is called. Chaning to QElapsedTime, no check of /etc/localtime. Reproducing bug 384561, the strace of file.so was something like: read(), stat(/etc/localtime), stat(/etc/localtime), stat(/etc/localtime), stat(/etc/localtime), stat(/etc/localtime), write(), read() ...... Now it is: read(), write() It also reduces the cpu in io/wait around 10% in a debug build. Test Plan: kio tests work as before desktop: works in dolphin Reviewers: #frameworks, fvogt, dfaure Reviewed By: dfaure Subscribers: fvogt, ngraham Tags: #frameworks Differential Revision: https://phabricator.kde.org/D9983 M +14 -13 src/core/slavebase.cpp https://commits.kde.org/kio/8d73867b3d4339ec026f92c4c2e89c265249aeb2
Hey guys, I have repeated both the initial tests: cp vs Dolphin KDE Plasma 5.12, KDE Frameworks 5.43.0, Qt 5.9.3, Dolphin 17.12.2 cp took 7 minutes, 31 seconds Dolphin took 9 minutes, 57 seconds Excellent result ! @Jaime Torres, thanks a lot again for looking into this. I'm pretty sure many users will appreciate this performance gain as much as I do !
Hi, thanks so much for the fix ! I was affected too but always thought the issue was the USB stack or something else. Regarding Nate, Massimo & Jaime's exchange : I must say, as a user and occasional reporter, the bugtracker is a bit overwhelming. Despite my being quite careful, I filed dozens of duplicates, which made me feel very bad. It gave me the feeling of wasting the other volunteers' time. Some other times, I think I didn't file against the right component and the bugreport was kinda lost. So I'm wondering 1) is there something in the bugtracker that is intrinsically misleading and adds work to triagers & devs ? 2) or should we help "educate" the users such as me so they file fewer reports, fewer duplicates and target them properly ? 3) the current "usability & productivity" campaign makes me extremely enthusiastic. Could it set the groundwork for a long-term meta-organization of the triaging / fixing work ? Thanks again so much to everyone involved. Exciting times :-)
In fact, I just wrote a new page for how to file a good bug: https://community.kde.org/Get_Involved/Bug_Reporting We are *very* interested in making life easier for people who want to get involved in bug triaging.
Let me add that we rather prefer people reporting duplicates instead of not reporting issues just because they fear they could report a bug that is already known. The work involved to mark a duplicate is neglegible compared to work for any other bugs.
Another recently discovered solution to avoid thousands of stat(/etc/timezone) is to set the environment variable TZ, "export TZ=:/etc/localtime" seen in https://news.ycombinator.com/item?id=13697555