I'm assigning this to Dolphin, but I'm not really sure if this should be assigned to Dolphin or KIO or any other component. Feel free to changed it to whatever component you think is better. The problem is that copying a 15GB folder with small files (thumbnails, ~20-40 kb each file) will: a) be ridiculously slow. Where talking about a +20x difference in times, compared to rsync. rsync will take around 20 minutes to copy that amount of data, while KDE takes several hours (5-6, even 10). b) the entire desktop (or maybe just plasma and all KDE-related stuff?) gets completely laggy/unreasonably slow. Even things as simple as moving the mouse pointer feels slow (the mouse pointer jumps from one location to another). Note that this doesn't happen with rsync. The CPU usage in both cases is more or less the same, around 30% to 50%. The CPU is a i7-3820 @ 3,6Ghz I'm on EXT4, and the drive isn't damaged, I can completely asure this (no errors in SMART, ran a few bad-sectors detectors liveCDs, everything is fine). The HDD is a Western Digital WD Blue WD5000AAKS 500GB 16MB Cache SATA 3.0Gb/s 3.5". This is what 'lshw' says about it: configuration: created=2014-04-12 16:46:46 filesystem=ext4 lastmountpoint=/home modified=2014-12-13 12:58:29 mount.fstype=ext4 mount.options=rw,relatime,data=ordered mounted=2014-12-13 12:58:29 state=mounted The MB is an Intel DX79SI. Reproducible: Always Steps to Reproduce: 1. Create a 15GB folder with subfolders and small files (20-40 kb each file) 2. Copy the entire folder 3. Watch your entire desktop lag. Actual Results: The desktop is laggy and the copy process is slow. Expected Results: The desktop should stay responsive and the file progress should be hitting the HDD's limits instead of hitting who-knows-what-bug in KDE which is making the progress slow. I'm leaving this as "Grave" because this is something that shouldn't be happening in such a mature DE and because this bug is making my machine unusable (while copying files). I agree that "I'm copying a 15GB of thumbnails" is not a common case, but regardless whether it's a common case or not, there is clearly something broken in KDE (or Dolphin or KIO or XYZ component).
Can you please check the performance of "kioclient copy src dest" on your system? I doubt it is a Dolphin issue.
Yes, sorry for the delay. I'll test it tomorrow evening.
I tested everything again, and couldn't reproduce the second bug anymore. Now the system is quite responsive. Anyways, the first bug (slow copy) is still there, both with Dolphin copy and with kioclient copy. Should I open another bug?
Just to be sure, do you copy a folder which contains lots of files, or do you copy the files individually? I.e., do you select a single (but very large) folder and copy it, or do you enter the folder, select all files and then copy them? Moreover, could you tell us the exact number of files, and, if they are not in one directory, roughly how they are distributed in the directory tree? I can confirm that copying lots of individual files is *extremely* slow and tried to investigate that with poor-man's-profiling. I usually get backtraces like the one below. Looking at KDirListerCache::findByUrl shows that it scans the (unsorted) list of KFileItems until the URL is found, so the runtime has a quadratic dependence on the number of files that are copied, which is problematic if that number is very large. It's probably not trivial to fix this. One could use a QHash to provide O(1) lookup for URLs, but this would require a lot of memory and CPU cycles in large directories. An alternative would be to sort the KFileItems in the list by their URLs. This would provide O(log N) lookup without any additional memory requirements, but would require some overhead for sorting whenever items are inserted or modified. (gdb) bt #0 0x00007ffff297b57a in QUrl::QUrl(QUrl const&) (this=0x7fffffffc770, other=...) at io/qurl.cpp:4267 #1 0x00007ffff2ef6755 in KUrl::KUrl(KUrl const&) (this=0x7fffffffc770, _u=...) at /home/kde-devel/kde/src/kde/kdelibs/kdecore/io/kurl.cpp:490 #2 0x00007ffff59f8875 in KFileItem::url() const (this=0x497b260) at /home/kde-devel/kde/src/kde/kdelibs/kio/kio/kfileitem.cpp:1548 #3 0x00007ffff59ccf9a in KDirListerCache::findByUrl(KDirLister const*, KUrl const&) const (this=0xbeaca0, lister=0x0, _u=...) at /home/kde-devel/kde/src/kde/kdelibs/kio/kio/kdirlister.cpp:836 #4 0x00007ffff59ccba1 in KDirListerCache::itemForUrl(KUrl const&) const (this=0xbeaca0, url=...) at /home/kde-devel/kde/src/kde/kdelibs/kio/kio/kdirlister.cpp:782 #5 0x00007ffff59d93b0 in KDirLister::cachedItemForUrl(KUrl const&) (url=...) at /home/kde-devel/kde/src/kde/kdelibs/kio/kio/kdirlister.cpp:2785 #6 0x00007ffff597dbd1 in KIO::CopyJobPrivate::statCurrentSrc() (this=0xa5b4090) at /home/kde-devel/kde/src/kde/kdelibs/kio/kio/copyjob.cpp:738 #7 0x00007ffff597d5d8 in KIO::CopyJobPrivate::statNextSrc() (this=0xa5b4090) at /home/kde-devel/kde/src/kde/kdelibs/kio/kio/copyjob.cpp:696 #8 0x00007ffff597c6dc in KIO::CopyJobPrivate::sourceStated(KIO::UDSEntry const&, KUrl const&) (this=0xa5b4090, entry=..., sourceUrl=...) at /home/kde-devel/kde/src/kde/kdelibs/kio/kio/copyjob.cpp:482 #9 0x00007ffff5986d54 in KIO::CopyJob::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (_o=0xa5b4240, _c=QMetaObject::InvokeMetaMethod, _id=19, _a=0xa64be00) at /home/kde-devel/kde/build/kde/kdelibs/kio/copyjob.moc:108 #10 0x00007ffff29ef869 in QMetaCallEvent::placeMetaCall(QObject*) (this=0xa65d920, object=0xa5b4240) at kernel/qobject.cpp:524 #11 0x00007ffff29f0aed in QObject::event(QEvent*) (this=0xa5b4240, e=0xa65d920) at kernel/qobject.cpp:1222 #12 0x00007ffff3a2662e in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=0x642a30, receiver=0xa5b4240, e=0xa65d920) at kernel/qapplication.cpp:4565 #13 0x00007ffff3a23bca in QApplication::notify(QObject*, QEvent*) (this=0x7fffffffdb00, receiver=0xa5b4240, e=0xa65d920) at kernel/qapplication.cpp:3947
I copy the entire folder, without getting inside it. Exact size of the folder (it's bigger now, but the bug is exactly the same): 48054231169 == 44,8GiB Total number of files: 3050371 Total number of folders inside: 16285 Output of tree -d -L 3 media/ | xclip: media/ ├── avatar │ └── 0001 │ ├── 01 │ ├── 02 │ ├── 03 │ ├── 04 │ ├── 05 │ ├── 06 │ ├── 07 │ ├── 08 │ ├── 09 │ ├── 10 │ ├── 11 │ ├── 12 │ ├── 13 │ ├── 14 │ ├── 15 │ ├── 16 │ ├── 17 │ ├── 18 │ ├── 19 │ ├── 20 │ ├── 21 │ ├── 22 │ ├── 23 │ ├── 24 │ └── 25 ├── blog │ └── 0001 │ └── 01 └── default └── 0001 ├── 01 ├── 02 ├── 03 ├── 04 ├── 05 ├── 06 ├── 07 ├── 08 ├── 09 ├── 10 ├── 11 ├── 12 ├── 13 ├── 14 ├── 15 ├── 16 ├── 17 ├── 18 ├── 19 ├── 20 ├── 21 ├── 22 ├── 23 ├── 24 └── 25 57 directories Each one of the folders at level 3 has many folders, with names like this one: "ff6b907341d5e6ce81704fdb8ef29b2e98e1703d_zoom". I think you're over-engineering the entire thing. Have a look how "cp" or "rsync" are doing it. They both are small tools and they do it just fine.
Thanks for the quick reply and for the additional information. I do agree that there is some potential for optimization, but the "over-engineering" is not quite as pointless as it may seem at first sight. Note that KIO and the applications that use it are not just replacements for the command-line tools which mostly work on local directories. They can be used for accessing lots of different protocols (like ftp, smb, webdav, tar, zip, and many, many more), and the code that copies files works exactly the same for all of them. The actual access to the files and folders is done by kioslaves, i.e., external processes which abstract the characteristics of the different protocols away. Moreover, KIO tries to make sure that views in Dolphin or elsewhere which show the URL where files are modified are updated even if the protocol does not support directory watching. Therefore, it will never be possible to just do what cp or rsync do. A certain amount of overhead will always be there (but as I said, I do agree that there is too much overhead in some situations, and it makes sense to investigate ways to reduce it).
(In reply to Frank Reininghaus from comment #4) > Just to be sure, do you copy a folder which contains lots of files, or do > you copy the files individually? I.e., do you select a single (but very > large) folder and copy it, or do you enter the folder, select all files and > then copy them? Moreover, could you tell us the exact number of files, and, > if they are not in one directory, roughly how they are distributed in the > directory tree? > > I can confirm that copying lots of individual files is *extremely* slow and > tried to investigate that with poor-man's-profiling. I usually get > backtraces like the one below. Looking at KDirListerCache::findByUrl shows > that it scans the (unsorted) list of KFileItems until the URL is found, so > the runtime has a quadratic dependence on the number of files that are > copied, which is problematic if that number is very large. > > It's probably not trivial to fix this. One could use a QHash to provide O(1) > lookup for URLs, but this would require a lot of memory and CPU cycles in > large directories. An alternative would be to sort the KFileItems in the > list by their URLs. This would provide O(log N) lookup without any > additional memory requirements, but would require some overhead for sorting > whenever items are inserted or modified. And that sorting is going to be expensive as the list grows. I have a simple test program that tests lookup in a QList with 50 million QString entries and running that on my monster of a machine results in the following: ********* Start testing of TestBenchmark ********* Config: Using QtTest library 5.4.0, Qt 5.4.0 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 4.9.2) QDEBUG : TestBenchmark::initTestCase() LOOKUP: INDEX= 7898741 , ITEM AT INDEX= "Entry-48903726" , TOTAL ITEMS= 50000000 PASS : TestBenchmark::initTestCase() PASS : TestBenchmark::indexBasedLookup() RESULT : TestBenchmark::indexBasedLookup(): 379 msecs per iteration (total: 379, iterations: 1) PASS : TestBenchmark::forEachBasedLookup() RESULT : TestBenchmark::forEachBasedLookup(): 437 msecs per iteration (total: 437, iterations: 1) PASS : TestBenchmark::iteratorBasedLookup() RESULT : TestBenchmark::iteratorBasedLookup(): 378 msecs per iteration (total: 378, iterations: 1) PASS : TestBenchmark::constIteratorBasedLookup() RESULT : TestBenchmark::constIteratorBasedLookup(): 366 msecs per iteration (total: 366, iterations: 1) PASS : TestBenchmark::concurrencyBasedLookup() RESULT : TestBenchmark::concurrencyBasedLookup(): 382 msecs per iteration (total: 382, iterations: 1) PASS : TestBenchmark::sortList() RESULT : TestBenchmark::sortList(): 9,237 msecs per iteration (total: 9,237, iterations: 1) PASS : TestBenchmark::binaryFindBasedLookup() RESULT : TestBenchmark::binaryFindBasedLookup(): 0.00017 msecs per iteration (total: 94, iterations: 524288) PASS : TestBenchmark::cleanupTestCase() Totals: 9 passed, 0 failed, 0 skipped, 0 blacklisted ********* Finished testing of TestBenchmark ********* Notice that it takes a huge amount of time (9 secs) to sort such a large list and that would have to happen each time someone inserts a new record.
Can I get this bug in a "Confirmed" state?
(In reply to Dawit Alemayehu from comment #7) > Notice that it takes a huge amount of time (9 secs) to sort such a large > list But the sorting has to be done in the application that uses KDirLister anyway (at least if "Sort by Name" is used, which is true most of the time, I think). And if the user tries to open a directory with 50 million files, then there might be things that take even longer than 9 seconds of sorting in KDirListerCache ;-) > and that would have to happen each time someone inserts a new record. You don't have to sort the entire list every time a new record (or some more new records) are inserted. One would first sort the new records, and then merge the two sorted lists (old and new records). For a single new file, one could just do a binary search to find the correct place in the list, which should be quite fast. BTW, if many files are deleted in a folder with 50 million files, then I guess that looking these up in the unsorted list that we currently have would be a huge problem. With a sorted list, it could be done quite efficiently. Maybe I'll try to analyze this in more detail when I find some time. (In reply to Alexander Nestorov from comment #8) > Can I get this bug in a "Confirmed" state? I haven't analyzed the problem (which is different from the problem which I first thought you meant and which I discussed with Dawit) in detail yet, so I can't really say how bad it is and if it should be considered a bug or rather wishlist material (I'll let others decide that). It's definitely not "grave" though in my opinion. You've said yourself that "this is not a common use case"! And a factor of 20 in the performance between KIO and rsync does not really sound that bad to me. I have seen performance issues that were *much* worse. Moreover, you cannot really compare rsync to KIO because both are designed to do quite different things. I do agree though that any optimizations in KIO would be welcome.
i've reported bug #358231 with somewhat similar symptoms, but they seemed to be different enough to warrant a new report
I have a similar problem with KF 5.27.0, but only when moving files. Tested with 1000 empty files on local fs (ext4, normal hd): $time mv 1000_files/* 1000_files_temp/ real 0m0.028s user 0m0.017s sys 0m0.010s $time kioclient move 1000_files/* 1000_files_temp/ real 1m44.809s user 0m1.407s sys 0m0.263s Copy and delete works fine. Doing the same in Dolphin causes similar behaviour (Bug #358231, as already mentioned): Moving the files is fast but the plasma workspace freezes up after it for over 5 minutes. Dolphin itself and all applications are responsive.
*** Bug 389296 has been marked as a duplicate of this bug. ***
This old bug makes dolphin unusable to deal with high amount of small files.
For people experiencing this, how are you doing the copy? copy-paste? drag-and-drop? Does it happen for both methods, or just one?
I did some tests on Arch Linux running plasma 5.12 beta and dolphin 17.12.1. Always copying/moving a folder containing 104.607 small files and 1.678 subfolders (889 MiB) from a hard disk to another hard disk, both formatted with NTFS file system. copy using ctrl+c/ctrl+v: completed in 9 minutes copy using "copy" and "paste" from dolphin context menu: completed in 11 minutes move using ctrl+x/ctrl+v: completed in 72 minutes move using "cut" and "paste" from dolphin context menu: completed in 158 minutes (???)
Thanks, that's hugely helpful! We're actively investigating this, BTW.
Git commit 18e4d245d3d595cdc17ad40aa88495d6d2c30bf7 by Jaime Torres. Committed on 28/01/2018 at 11:01. Pushed by jtamate into branch 'master'. Use the much faster urls() method from QMimeData Summary: When requesting a list of text/uri-list, use the much faster QMimeData urls() method. The unittests pass (the desktop: protocol) and is half solved. The paste speed is as fast as drag&drop in local files and with fish: files. The lock in X11 plasma or kwin still needs another patch. Test Plan: Select 2000 files in one folder, cut and paste them in another disk. Reviewers: #frameworks, dfaure Reviewed By: dfaure Tags: #frameworks Differential Revision: https://phabricator.kde.org/D10155 M +6 -3 src/lib/io/kurlmimedata.cpp https://commits.kde.org/kcoreaddons/18e4d245d3d595cdc17ad40aa88495d6d2c30bf7
Git commit 9fbf7a0b624aee6b116efdf69462e73f0275fab6 by Jaime Torres. Committed on 05/02/2018 at 18:25. Pushed by jtamate into branch 'master'. Faster drag&drop in directories with thousands of files Summary: The check is called when the mouse is moved in a drag&drop operation. Dragging all files in a directory with 3000 files under callgrind, moving the mouse to the other panel and then canceling, doing it twice, callgrind shows that the method urlListMatchesUrl is called around 200 times, spending around 9,30% of the cpu in those calls. Applying the patch, callgrind tells it uses now 0.31% of the cpu in 1208 calls. Reviewers: #dolphin, elvisangelaccio, markg Reviewed By: #dolphin, elvisangelaccio, markg Subscribers: markg, anthonyfieroni, michaelh, elvisangelaccio, ngraham Differential Revision: https://phabricator.kde.org/D10085 M +3 -0 src/kitemviews/kitemlistcontroller.cpp M +17 -3 src/views/draganddrophelper.cpp M +10 -0 src/views/draganddrophelper.h https://commits.kde.org/dolphin/9fbf7a0b624aee6b116efdf69462e73f0275fab6
The reporter wrote: (Alexander Nestorov from comment #5) > I copy the entire folder, without getting inside it. but as far as I can tell, your optimizations are all only addressing the case where somebody selects many individual files and drags&drops them somewhere as a group, not the case the reporter described (a recursive copy of one single folder that just happens to contain many small files inside).
The performance improvements apply also to the problem with several thousands of files in several directories. Just to be sure, I've checked a harder test case (50.000 little files in several directories) and the performance improvements apply, but there is still room for improvement, because after moving 25.000 files, the performance drops. The problem with plasma being locked is still pending, but we are working on it. As you can see, it is not an easy bug to fix.
Oh!, I misread the data. The file operations are fast, but plasma is still showing information about half the files when the files operations has already concluded. (This could be easier to improve than the lock and could reduce the lock time).
I don't know if it would be useful for this bug but I got some problems with Dolphin (Mageia 6, KF5 16.12.3 Dolphin 16.12.3 plasma 5.8.7) It seems that Dolphin doesn't work always the same way for ntfs partitions on USB external disk: It depends on the way this partition is mounted : 1) if it is mounted with fstab (in this case it apparently DOES NOT USE udisks2) we can explore the content on the mout point choosen in fstab but it freezes eating CPU with mout.ntfs-3g and gam_server Dolphin is quite unresponsive when exploring a huge folder containing subfolders with thousands of photos 2) if it is an hotplugged partition (in this case it seems to use udisks2) a notification pop-up proposes to use Dolphin and it is OK : we explore it in /run/media/username/hddname (mount-ntfs-3g and gam_server use CPU for 3 seconds) and we get no freeze exploring the same NTFS partition https://bugs.mageia.org/show_bug.cgi?id=7985#c12
Git commit e1fe26f1fc78082fdb215cb818177541d4607d9c by Jaime Torres. Committed on 19/02/2018 at 20:09. Pushed by jtamate into branch 'master'. Reduce plasmashell frozen time to almost nothing Summary: Related: bug 358231 Even the icon with the number of tasks pending moves from time to time. To reduce the frozen time, a similar patch must be applied also to frameworks/kwindowsystem src/platforms/xcb/kxmessages.cpp frameworks/plasma-framework src/plasma/private/effectwatcher.cpp According to the documentation (and a look to the source code) http://doc.qt.io/qt-5/qabstractnativeeventfilter.html The type of event eventType is specific to the platform plugin chosen at run-time, and can be used to cast message to the right type. On X11, eventType is set to "xcb_generic_event_t", and the message can be casted to a xcb_generic_event_t pointer. The other eventType are "windows_generic_MSG" and "mac_generic_NSEvent". No other eventType starts with an 'x'. Test Plan: Cut & paste 2000 small files. Before, a freeze (plasmashell not responding) of minutes After, a freeze of seconds Reviewers: #frameworks, #plasma, davidedmundson Reviewed By: #plasma, davidedmundson Subscribers: broulik, davidedmundson, plasma-devel Tags: #plasma Differential Revision: https://phabricator.kde.org/D10627 M +1 -1 shell/screenpool.cpp https://commits.kde.org/plasma-workspace/e1fe26f1fc78082fdb215cb818177541d4607d9c
*** Bug 304136 has been marked as a duplicate of this bug. ***
*** Bug 203277 has been marked as a duplicate of this bug. ***
Hmmm, I've just posted what sounds like the same problem here https://bugs.kde.org/show_bug.cgi?id=393748 I've added attachments showing the debug output of the plasma-shell threads and dolphin threads. Hope it is useful.
*** Bug 396093 has been marked as a duplicate of this bug. ***
Git commit 10cf5aca5b3da2608eaf2967415d0b09b36121ac by Méven Car. Committed on 04/04/2020 at 18:52. Pushed by meven into branch 'master'. File ioslave : use Better setting for sendfile syscall Summary: Changes : * use sendfile when copying file bigger than 2 GB * copy 512 kB instead of previously 32 kB for each sendfile or read/write copying iteration This effectively increase the copying speed, since the filesystem is asked to copy more at once, it lets it reach higher speed before getting the next data to copy. Related: bug 402276 Test Plan: Copying large files or folders with medium sized files (for instance photos) is noticebly faster. Reviewers: #frameworks, dfaure Reviewed By: dfaure Subscribers: bruns, broulik, kde-frameworks-devel Tags: #frameworks Maniphest Tasks: T11627 Differential Revision: https://phabricator.kde.org/D28555 M +3 -5 src/ioslaves/file/file_unix.cpp https://commits.kde.org/kio/10cf5aca5b3da2608eaf2967415d0b09b36121ac
*** Bug 428073 has been marked as a duplicate of this bug. ***
I have similar issue copying to external 3.6TB USB3.0 NTFS drive in Krusader. 362 GB of data. Different files: music, video, pictures, text… It couldn’t copy this information volume during whole night O_o Now I reformatted drive in NTFS with default cluster size without any additional parameters. Before drive was formatted by vendor and had additional small partition at the beginning. At first copying started with 100MB/s and after 40 GB speed dropped to 0,5-2 MB/s while copying mp3 files. Speed is very unstable. Sometimes jumps to 30-60 MB/s, then drops to 0,5-2 MB/s. It copies better after formatting, but is it OK? Operating System: Kubuntu 21.04 KDE Plasma Version: 5.22.5 KDE Frameworks Version: 5.86.0 Qt Version: 5.15.2 Kernel Version: 5.11.0-34-generic (64-bit) Graphics Platform: X11
This problem persists. 362 GB of data. F5 list of folders in Krusader from internal NTFS HDD to external NTFS USB 3.0 HDD. 40 GB were copied very fast – 20 min or so. 180 GB were copied slowly in 3hr and I thought the rest 182 GB would be copied in 3hrs but it didn’t happen. Almost 12 hrs passed, but it copied 270 GB. 90 GB still to go according to copying status. It’s definitely not OK.
lsusb Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 002: ID 0bc2:ab30 Seagate RSS LLC BUP BK Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 002: ID 056a:033b Wacom Co., Ltd CTL-490 [Intuos Draw (S)] Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub lsusb -t /: Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M /: Bus 07.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M /: Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M /: Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 10000M |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/14p, 480M |__ Port 13: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 13: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 13: Dev 2, If 2, Class=Human Interface Device, Driver=usbhid, 12M
(In reply to empyreal from comment #31) > This problem persists. 362 GB of data. F5 list of folders in Krusader from > internal NTFS HDD to external NTFS USB 3.0 HDD. > > 40 GB were copied very fast – 20 min or so. 180 GB were copied slowly in 3hr > and I thought the rest 182 GB would be copied in 3hrs but it didn’t happen. > Almost 12 hrs passed, but it copied 270 GB. > 90 GB still to go according to copying status. > It’s definitely not OK. I feel your pain, while I love Dolphin, for me it's only capable of dealing with small amounts of data due to this 'grave' bug. And yes it is a grave bug for those that work with a lot of data. I generally use cp and rsync instead of dolphin for anything more than a few files. Having said that there is absolutely no reason why Dolphin should not be as fast as cp or rsync , we are talking I/O here, orders of magnitude slower than any CPU can operate at. The problem here is a bug or at least the way Dolphin does the copy. If that's correct that every time a file is copied some massive list is sorted then that's just not the way you do it. I only wish I could devote my time to fixing this problem. It's the one bug/feature in Dolphin that spoils an otherwise excellent program.
https://superuser.com/questions/424512/why-do-file-copy-operations-in-linux-get-slower-over-time 10 years later… I cancelled this joke copying and it stuck two cores of my CPU on 100% and freezed Krusader. Before copying I didn’t have this message: Partition 1 does not start on physical sector boundary. sudo fdisk -l Disk /dev/sdb: 3,64 TiB, 4000787029504 bytes, 7814037167 sectors Disk model: BUP BK Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: A601FD60-033F-4D1A-8E6C-BCEDD9D0D189 Device Start End Sectors Size Type /dev/sdb1 63 7814032063 7814032001 3,6T Microsoft basic data Partition 1 does not start on physical sector boundary.
Reboot stuck. Made hard reset. Rebooted. Booting slowed dramatically both on BIOS phase and Linux HDD detecting phase. After reboot to Kubuntu, it can’t automount external USB drive. I can’t mount it in Notifier. After reboot to Windows, it can’t mount external USB drive. Shutdown PC. Disconnected and booted Kubuntu. Connected external USB after boot. It didn’t mount automatically and I can’t mount it manually in Notifier. Deleted partition in Partition Manager. After that reboot became normal. Booted to Windows. Formatted partition with AOMEI. Process was very slow, which is abnormal, but finished. This copy process in Linux definitely hurts external USB drive. Start copying same data in Windows with Total Commander and got same problem. Windows couldn’t shutdown with external USB drive. It looks like canceling copy process in Linux makes external USB drive unresponsive and reformatting, rebooting do not help. Problems with this external USB drive started after canceling cp in Konsole. After that I got some weird things with mounting. I was using this drive happily without any problems till that moment. I am going through this problem second time. Copy canceling is dangerous...
Trying to put my USB drive to normal working state. Booted Linux. Connected USB drive. Notifier said that I am not authorized to mount it, so USB drive didn’t mount automatically. I mount it with this command: sudo mount -o remount,uid=1000,gid=1000,rw /dev/sdc1 USB drive mounted. Started copying 5 GB of jpg files for testing purposes. Regretted. After 2.5 GB speed dropped to 256 KiB/s. Afraid to cancel copying, because it will definitely make HDD unresponsive again.
Copying finished. 5 GB transfer took almost 1hr O_O Reboot failed. Linux was stopping Disk Manager for 90 sec and failed. Rebooting to Windows 10. Problems with loading drive. Reboot failed. Reset. Rebooting to Linux. Can’t mount USB drive. sudo mount -o remount,uid=1000,gid=1000,rw /dev/sdc1 do not work too shutdown now Started PC and booted Kubuntu. Connected USB drive. Partition 1 does not start on physical sector boundary. AGAIN!!! Can’t mount in Krusader and Notifier. sudo mount -o remount,uid=1000,gid=1000,rw /dev/sdc1 do not work too Deleted partition. Formatted to NTFS. Partition 1 does not start on physical sector boundary. GONE! shutdown now Started PC. Booted Kubuntu. I am not authorized to mount device. No automount. sudo mount -o remount,uid=1000,gid=1000,rw /dev/sdc1 WORKED!!! Unmount in Notifier. shutdown now Started PC. Booted Kubuntu. USB drive is not mounted automatically. Notifier don’t show any messages. All that nonsense went on till Partition Manager reported that it can’t create partition because of errors. I booted to Windows again and make all necessary operations in AOMEI. Then booted to Linux and got Partition 1 does not start on physical sector boundary. Installed GParted, reformatted everything and got USB drive which mounts on Linux with read-only file system and can’t mount on Windows anymore. I don’t know exactly what happened, but USB drive started working on Kubuntu after I tried it on old notebook with Windows 7 where it didn't mount. Booted Kubuntu. Then connected USB drive. At first it was automatically detected like ro file system and then system redetected it again automatically in new folder with rw files system. But now not only copying sucks in Krusader but Alt+Enter counts for eternity. Now is hard to tell whether Linux sucks or USB drive sucks, but I feel that USB controller on USB drive is glitchy, because even SeaTools on Windows do not detect it at first try. Tried rsync but with 5 GB of images but it stuck after 2,5 GB as Krusader. Cancelled and got one CPU core freezed with 100% load. That’s most annoying problem on Linux so far. Just simple cp canceling in Konsole... I wasted 3 days, figured basically nothing and made things worse. Last solution will be to open case and use external drive as internal. At least I am sure that drive inside has standard ports.
(In reply to empyreal from comment #37) > Copying finished. 5 GB transfer took almost 1hr O_O > > Reboot failed. Linux was stopping Disk Manager for 90 sec and failed. > Rebooting to Windows 10. Problems with loading drive. > Reboot failed. Reset. > Rebooting to Linux. > Can’t mount USB drive. > sudo mount -o remount,uid=1000,gid=1000,rw /dev/sdc1 do not work too > shutdown now > Started PC and booted Kubuntu. > Connected USB drive. > Partition 1 does not start on physical sector boundary. AGAIN!!! > Can’t mount in Krusader and Notifier. > sudo mount -o remount,uid=1000,gid=1000,rw /dev/sdc1 do not work too > Deleted partition. Formatted to NTFS. > Partition 1 does not start on physical sector boundary. GONE! > shutdown now > Started PC. Booted Kubuntu. > I am not authorized to mount device. No automount. > sudo mount -o remount,uid=1000,gid=1000,rw /dev/sdc1 WORKED!!! > Unmount in Notifier. > shutdown now > Started PC. Booted Kubuntu. > USB drive is not mounted automatically. > Notifier don’t show any messages. > > All that nonsense went on till Partition Manager reported that it can’t > create partition because of errors. I booted to Windows again and make all > necessary operations in AOMEI. Then booted to Linux and got Partition 1 does > not start on physical sector boundary. Installed GParted, reformatted > everything and got USB drive which mounts on Linux with read-only file > system and can’t mount on Windows anymore. > > I don’t know exactly what happened, but USB drive started working on Kubuntu > after I tried it on old notebook with Windows 7 where it didn't mount. > Booted Kubuntu. Then connected USB drive. At first it was automatically > detected like ro file system and then system redetected it again > automatically in new folder with rw files system. > > But now not only copying sucks in Krusader but Alt+Enter counts for > eternity. Now is hard to tell whether Linux sucks or USB drive sucks, but I > feel that USB controller on USB drive is glitchy, because even SeaTools on > Windows do not detect it at first try. > > Tried rsync but with 5 GB of images but it stuck after 2,5 GB as Krusader. > Cancelled and got one CPU core freezed with 100% load. > > That’s most annoying problem on Linux so far. Just simple cp canceling in > Konsole... I wasted 3 days, figured basically nothing and made things worse. > Last solution will be to open case and use external drive as internal. At > least I am sure that drive inside has standard ports. I've had very little problem with USB drives, I've been using KDE Neon for a few years now. Sounds like you have a hardware fault especially as rsync doesn't work either, I use rsync on a daily basis to check and copy thousands of files, never have a problem. If that was my system I'd probably wipe the drive with nwipe (ShredOS) https://github.com/PartialVolume/shredos.x86_64 , it will give you an idea of throughput and will check every block with verification. if that bombs out with an I/O error I'd test the drive on another system with a different USB controller in order to isolate the problem. If it's the drive scrap it, it's not worth the hassle.
Nick, thanks for help! It’s system problem. Krusader and cp worked same way. Why expect rsync work another way? I wrote that I got problem after canceling cp process in Konsole. I never had any problems with this drive, till I tried copying in Konsole. I was noticing sometimes that copying speed was slow, but I didn’t copy 200-300 GB to this USB drive, because all backups were done on Windows. I copied small volumes from time to time. I’ll test it again after I deal away with ro mount issue. I managed to awake USB drive from coma in Windows 10 with Device Manager (Populate function somehow worked out) and formatted drive in AOMEI, because Windows showed messages with errors. Now on Windows everything works as it should – 5 GB in 2 minutes! No issues with USB drive detection and mounting. On Kubuntu I still have mount problem. It always mounts USB drive as ro file system: drwxrwxrwx 1 user user 4096 sep 18 20:31 'Seagate Backup Plus Drive' /dev/sdc1 on /media/user/Seagate Backup Plus Drive type fuseblk (ro,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096,uhelper=udisks2) $ sudo mount -o remount,uid=1000,gid=1000,rw /dev/sdc1 /dev/sdc1 on /media/user/Seagate Backup Plus Drive type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096,uhelper=udisks2) rw now but still I can’t copy or create files. It’s another issue, so I won’t flame here anymore.
When there is dual boot, Offline USB drive in Windows Disk Management after use. Now USB drive mounts properly - rw. Moreover - it works properly! Copying goes as fast as on Windows and I feel dumb. But question remains: why Ctrl+C of cp process in Konsole created issue with USB drive?
Copied 379 GB from internal notebook drive to USB drive. Different files, mostly short videos. Process went successfully and took 4 hrs. Queued 783 GB from internal desktop HDD”. In 6 hrs it copied 535 GB and again stuck on mp3 (163 GB of mp3). Copying speed dropped to 2 MB/s which is strange: these are not tiny files. Assuming that average copying speed to USB 3.0 drive is 3 GB/min, these are still bad results. The only solution to make copying faster seems in queuing folders (F2 in Krusader), which is annoying pain in the ass, when you need just to copy all content from one drive to another. Paused copy process and drive became unresponsive. Can’t open anything on USB drive. Two CPU cores froze on 100% load. Krusader became unresponsive. Nothing can be done. Killed kioslave5 in System Monitor. Krusader became responsive. USB drive works too. Tried to copy 5 GB of jpegs. Very slow. 1 GB in 5 min. Cancelled. Krusader works, drive works. No freeze. Tried to copy 5 GB of mp3s. Same story. 1-2 MB/s. Reached target Reboot. Stuck. Reset.
I need to figure out whether it is hardware issue or not. It’s some sort of interface buffer problem, because changing settings in Windows helped for short period of time. Windows. Mount slow. Copy slow. Offline and disconnected drive. Waited few min. Connected drive and Online. Mount slow. Copying slow. Device Manager => Volumes => Populate. Data received but no help. Mount slow. Copy slow. Very long Shutdown. Windows. Waited few min. Connected USB drive. Mount slow. Copy slow. Noticed that folders, copied on Kubuntu open with lag. Formatted USB drive just in case. Switched buffer settings in Windows. Sometimes it helped USB drive to copy faster, but for short period of time. Speed was unstable. Made few Short DST of SAS-SCSI-FC until device passed in SeaTools. Rescanned. USB 1394 showed up. Copy slow. Disconnected drive. Restarted Windows. Connected drive. Copying started fast, then became slow. Device Manager => Volumes => Populate. Volume data received. Copying became faster. I have no idea how it works, but it shows some effect. Started copying. Starts very slow and then goes fast. Reboot with USB drive. Stuck. Unplugged drive. Rebooted Windows. Connected USB drive. Started copying video files in Total Commander. Started fast then speed dropped to 1 MB/s. Now drive steadily behaves like this: when connected, starts fast copy, the slows down to … KB/s. Drive slows boot and restart.
(In reply to empyreal from comment #42) > I need to figure out whether it is hardware issue or not. > It’s some sort of interface buffer problem, because changing settings in > Windows helped for short period of time. > > Windows. Mount slow. Copy slow. > Offline and disconnected drive. Waited few min. > Connected drive and Online. Mount slow. Copying slow. > Device Manager => Volumes => Populate. Data received but no help. Mount > slow. Copy slow. Very long Shutdown. > > Windows. Waited few min. Connected USB drive. Mount slow. Copy slow. > Noticed that folders, copied on Kubuntu open with lag. Formatted USB drive > just in case. Switched buffer settings in Windows. Sometimes it helped USB > drive to copy faster, but for short period of time. Speed was unstable. > > Made few Short DST of SAS-SCSI-FC until device passed in SeaTools. > Rescanned. USB 1394 showed up. Copy slow. > Disconnected drive. Restarted Windows. Connected drive. > Copying started fast, then became slow. > > Device Manager => Volumes => Populate. Volume data received. Copying became > faster. I have no idea how it works, but it shows some effect. > Started copying. Starts very slow and then goes fast. > > Reboot with USB drive. Stuck. Unplugged drive. Rebooted Windows. > Connected USB drive. Started copying video files in Total Commander. Started > fast then speed dropped to 1 MB/s. > > Now drive steadily behaves like this: when connected, starts fast copy, the > slows down to … KB/s. Drive slows boot and restart. That initial burst is actually normal, but dropping to 1MB/s is classic behaviour when you have a bad drive or controller that's generating I/O errors. The reason for the initial burst is that when the I/O writes first start, the operating system is filling the I/0 buffer in CPU RAM which it does very much quicker than sustained disc I/O then once there is no more space in memory you then see the true I/O speed and if you have a bad drive or controller you will see it drop way down. For a spinning disc I'd expect anything from 30MB-160MB/sec. SSD maybe 200-400MB/s. So say you have 8GB CPU memory you would expect to see that initial burst drop off after you had copied maybe 4-6GB of files. As you are seeing 1MB/s after the initial burst I would say you do seem to have a hardware fault. Like I mentioned earlier using ShredOS will confirm that. Assuming you have a backup of everything on that disk. By using ShredOS if you see the same effect i.e an initial burst (which is expected as explained above) but then it drops to 1MB/s that would confirm to me you have a hardware fault. Even on a 20 year old spinning disc you would see 20-35MB/s 30 seconds or so after that initial burst. However if you don't see 1MB/s but see a sustained throughput that is much higher, then you most likely have a software issue. Until you run ShredOS you won't be able to isolate the problem. I'd also do a grep on /var/log/ messages and also Windows event log to look for I/O errors.
You could also run smartmontools on the drive, you may have to plug it into a SATA internal connector or ESATA if you have one. For smartmontools to be able to access the drive as not all USB to SATA adapters support ATA pass through. Again ShredoS will confirm whether the controller supports ATA pass through. Smartmontools will tell you all about the general health of the drive. I don't know this but I'm assuming windows can also do this but as I don't use Windows I don't know that for sure.
I know that my approach is unprofessional but what to do? There are plenty of tools in Windows to check SMART. It is ok. Also there are no surface or file errors. SeaTools gives me hint that smth is wrong with USB controller. 99% that when I remove HDD from case, it will work properly. But I need USB drive and check copy on Windows.
USB Seagate Backup Plus Drive. I tried copy in Konsole, because I noticed slow copy in Krusader. Copy canceling in Krusader was slow too. After I canceled cp process in Konsole, I got Partition 1 does not start on physical sector boundary. Fixed issue in GParted by reformatting. It also realigned automatically. Somehow “HDD reset” on Windows and started working properly. I thought that everything is ok and left it to copy whole night on Kubuntu. Speed dropped and mess started again. It’s interesting whether it is hardware fault or software fault leading to hardware fault. Slow copying to USB drive is common problem on Linux and it should be addressed properly. It’s unacceptable that server system has issues with data transfers. GParted drive again in vain. Made things worse on Windows. Strange manipulations on Windows and I noticed that drive became responsive again. Checked it with SeaTools once again. Populated volumes info. Drive works fast but copy sucks. Offline. Disconnected USB drive. Booted Kubuntu. Waited for USB drive to cool down. Connected drive. Mounted. Tried copying. Works as expected! Booted Windows. 1 MB/s. Booted Kubuntu again. Copy on 2-3 MB/s speed. Could not Remove safely… Shutdown. Removed drive from case. Never believe SeaTools... USB controller is OK. Checked with other Seagate drive. Connected HDD to SATA port. GParted messaging about errors. dd everything. Successfully formatted drive. Problems with mounting in Kubuntu. I could create files on it as sudo su only. chown chmod remount - no result, except now sudo su can’t create anything too. Windows boots very slow with the drive. Connected hot, but copying sucks. Drive clicks like it constantly is reading smth. It behaves like this after unsuccessful mounting on Kubuntu, after copying/canceling on Kubuntu and reboot or PC hard reset from Kubuntu. On Windows it stopped clicking after a while. Windows detects hard drive as removable, despite it is SATA now. I need to Offline it as USB to mount rw file system on Kubuntu. Now I have ST4000LM024-2AN17V which can write with 1 MB/s speed at best on Windows. Kubuntu first copy starts fast, the slows down to zero. Second copy almost zero speed. And what happens next!!! //All drives are SATA. Faulty drive ST4000LM024-2AN17V. I test and copy to it. //Just booted Kubuntu. No copying sudo hdparm -Tt /dev/sdc /dev/sdc: Timing cached reads: 23030 MB in 2.00 seconds = 11531.43 MB/sec Timing buffered disk reads: 48 MB in 3.04 seconds = 15.78 MB/sec //Started copy process in Krusader. Starts fast and slows down. /dev/sdc: Timing cached reads: 24952 MB in 2.00 seconds = 12494.70 MB/sec Timing buffered disk reads: 268 MB in 3.06 seconds = 87.55 MB/sec //Started copy process in Krusader. Worked as expected O_O //Started copy process in Krusader. Worked as expected O_O sudo hdparm -Tt /dev/sdc /dev/sdc: Timing cached reads: 22382 MB in 2.00 seconds = 11206.40 MB/sec Timing buffered disk reads: 176 MB in 3.01 seconds = 58.46 MB/sec //Started copy process in Krusader. Worked as expected. 20 GB in 4 min. O_O Now I have ST4000LM024-2AN17V copying properly on Kubuntu or not? Any ideas?
> I need to figure out whether it is hardware issue or not. empyreal, would you maybe like to discuss this first in a forum until you have ruled out that this is not a _disk_ issue on your side?
I read so many forums that I got bloodshot eyes. I wrote how it all started. Slow copy and laggy copy canceling in Krusader on perfectly working 3 years old USB drive. No glitch ever! I never ever experienced such issues on Windows. No slow copying or laggy copy canceling in Total Commander with USB drives. That’s was the reason to try cp in Konsole. Canceled cp in Konsole and got this... Just finished copying with Krusader 335 GB via SATA – 2,5 hrs. I think, now this drive works as it should. I have no inclination to mess with USB controller again and continue to use it as SATA. If it fails, I’ll give notice. I do not expect slow copy problem to be addressed anytime soon, actually I do not expect anything, but maybe this info will help in some way, whatever.
For what's it's worth I think you have a bad USB controller, I've comes across plenty of poor quality USB controllers, not to mention an inadequate power supply that's supposed to power the drive. In fact I have several USB controllers that only work properly when I power the drive that's attached to USB with a ATX power supply, a lot depends on the drive power ratings. If I was you I'd throw that particular USB controller in the bin and buy a decent quality USB controller with lots of good reviews. Good luck, now you are using SATA, I'm sure everything will be fine ! I'm going to log off and stay quiet now as I don't really think this problem is solely related to KIO copying.
Seagate is to blame for supplying bad controllers for it’s USB drives? How to check that USB controller properly? I don’t want my 3.6 TB go into coma again. The only thing I can do, connect old 2.5” Seagate 80 GB and try to mess with it. Any decent utilities?
It didn’t work out that easily, till I dd whole drive with zeros. Switching to SATA makes things easier, but... Drive started glitching again after copy canceling. dd with zeros took 11 hrs starting from 140 MB/s and finished with 95 MB/s. Then drive stopped slowing boot and restart. Formatted with AOMEI from USB stick. Copying very rarely drops to 200 KB/s, but generally ok - 388 GB in 3:20 hrs. Copied more than 1,5 TB already and continue. And now I do not need to Offline disk in Windows. It mounts rw on Kubuntu. smartctl -l devstat shows few things but no time when they happened: 0x03 0x030 4 1 --- Number of Mechanical Start Failures 0x03 0x038 4 0 --- Number of Realloc. Candidate Logical Sectors 0x04 0x008 4 1 --- Number of Reported Uncorrectable Errors Nick, your advice was to the point! Easiest thing to do - wiping with zeros and then move forward. “Low level format” helped me in the past with few drives, but in this case, as it was very long process and there were no bads or relocs, I thought dd MBR will be enough. Maybe this drive was glitching long time but I didn’t noticed on small data volumes and cancel operation triggered smth... My bad...
Seagate is to blame for everything. This is shit SMR technology. This drive needs brakes and smaller packs of data. It’s busy working for 10 min after huge copy process. After brake works faster, then slows down. And if you turn it off immediately, which on USB is common thing, it's internal program may go into some sort of loop or what. https://ubuntuforums.org/archive/index.php/t-2370055.html
Any chance to hide the comments that deal with faulty hardware? They make the bugreport impossible to follow.
Indeed, just because one person was experiencing faulty hardware, doesn't mean the legitimate issue in KDE software is fixed for everyone. I will hide the non-relevant comments.
Question for those who can reproduce the issue: Is the issue fixed or improved by https://invent.kde.org/frameworks/kio/-/merge_requests/957 ?
A possibly relevant merge request was started @ https://invent.kde.org/frameworks/kio/-/merge_requests/1718