Version: 4.7 (using KDE 4.7.0) OS: Linux All going fine when I copy files from my USB device and the problem occurs only when I copy files TO it. The KDE's notification shows that copying speed is about 50Mb/s (yes, namely megabytes), so it is about 400 megabits per second and it is too fast for a full-speed USB device (12MB/s is maximum for USB 1.1). Alan Stern, one of the kernel developers, described to me what actually happens. The problem is in that actually it show the speed with which files are copied into the RAM (it could be even faster if the HDD allow it), therefore it shows that the copying operation is finished when it has not even been started. Also when target files are too small the "operation progress" popup will not appears at all. Reproducible: Always Steps to Reproduce: Plug in any USB device and try to send any files to it. Actual Results: The operation progress show the progress but for the wrong operation. It also doesn't give me a possibility to pause/cancel the operation. Expected Results: The operation progress notification should display actual data transfer progress and should also allow to user to pause/stop the operation. https://bugzilla.kernel.org/show_bug.cgi?id=41472#c7 > Yes, the information displayed in the progress window is off. This really > isn't a kernel problem so much as a problem in KDE.
> therefore it shows that the copying operation is finished when it has not even been started Do kernel developers also know a POSIX interface way to tell us when and how fast this buffered data is actually written to the disk, without causing massive slow downs that any "sync" type calls would cause?
This is sad, but seems that they do not know how to do it without sync :-( But if you agree that the progress information is wrong and there is no way to implement it correctly, we probably should send a feature request/bug report to kernel devs? How you think, should we? Actually, this is weird to have such problems after 20 years of development :)
Hello. Sorry, forgot to post this earlier. I got next mail from Alan: "Bugzilla is currently down, so I have to send this reply by regular email. On Mon, 5 Sep 2011 bugzilla-daemon@bugzilla.kernel.org wrote: >... I did some asking around, and people said to look at the sync_file_range(2) system call (see also fsync and fdatasync). It's not a POSIX interface but it should solve the KDE problem. Alan Stern" Could it be useful?
I haven't checked if it degrades performance, but looking from the description, it looks like it would be exactly what we need. Thanks for the pointer!
Sorry for noise, but is there any progress with this? I working with the different usb drives all the time and this issue makes me crazy. I trying to detect the operation progress using htop, orienting by I/O indication...
I have this issue with the latest stable release. When I transfer fairly large files, the plasma notification report that 100% have been transferred well before the actual transfer is complete. KDE 4.11.2 with OpenSuse packages.
*** This bug has been confirmed by popular vote. ***
*** Bug 335851 has been marked as a duplicate of this bug. ***
*** Bug 361224 has been marked as a duplicate of this bug. ***
On 05/04/16 13:31, Christoph Feck via KDE Bugzilla wrote: > https://bugs.kde.org/show_bug.cgi?id=281270 > > Christoph Feck <cfeck@kde.org> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |dexy@xtra.co.nz > > --- Comment #9 from Christoph Feck <cfeck@kde.org> --- > *** Bug 361224 has been marked as a duplicate of this bug. *** > HI Christoph From this I take it the KIO team has assumed the task, and it's still in progress? The companion email has it marked as resolved in the Dolpin stream. Cheers Dex (Fester :) )
> still in progress? No, work has not started yet (if it is, the bug gets status ASSIGNED).
*** Bug 383995 has been marked as a duplicate of this bug. ***
Is this still applicable with KDE Frameworks 5.45?
I am using openSUSE Tumbleweed with KDE Frameworks 5.25.0 now. This problem seems still there. I just copied a 800MB file to a Kingston USB drive that has usually 10+MB/s write-in capacity. The copying progress is showing for about several seconds at a speed of around 100MB/s. After this, the Notifier said the copy finished. But when I tried to reject the USB drive, it took about 2 minutes for removing. Then, the Notifier said the device can be removed then.
Sorry, it is 5.45.0 instead of 5.25.0.
Thanks for confirming!
Hi Nate. Firstly, how deep is your 'to-do' queue? I asked this years ago, several OS upgrades ago. Secondly, how can I tell which KDE Frameworks? I'm a user not a programmer. I'm running a fully-upgraded Kubuntu 18.04 amd64 now, and it's still this way - has been since seemingly year dot, certainly 14.04,last LTS. Thirdly, I'm also running a Raspberry Pi, and that's got the same 'gotcha'. Is this a Deeeep-OS 'feature' of Debian? or Linux itself? Not closing off a file on completion of a transfer? I transfer to a USB stick or external disk and after 'completion' go have a cuppa before the LED stops flashing. Cheers Dex On 09/05/18 05:31, Nate Graham wrote: > https://bugs.kde.org/show_bug.cgi?id=281270 > > Nate Graham <nate@kde.org> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > Status|CONFIRMED |NEEDSINFO > Resolution|--- |WAITINGFORINFO > CC| |nate@kde.org > > --- Comment #13 from Nate Graham <nate@kde.org> --- > Is this still applicable with KDE Frameworks 5.45? >
On 09/05/18 07:22, Nate Graham wrote: > https://bugs.kde.org/show_bug.cgi?id=281270 > > Nate Graham <nate@kde.org> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > Product|kio |frameworks-kio > CC| |kdelibs-bugs@kde.org > Resolution|WAITINGFORINFO |--- > Component|general |general > Status|NEEDSINFO |CONFIRMED > Summary|Show progress actually |When copying files to a > |transferred to disk instead |removable disk, speed and > |of to caches |progress reflect copying to > | |the local cache not to the > | |disk itself > > --- Comment #16 from Nate Graham <nate@kde.org> --- > Thanks for confirming! > FYI: this occurs on ALL transfers - USB, Network , and same-machine either inter-disk or inter-directory! My attempt at a semi-graphical user's view: Present system: Source Buffer Medium Destination | ---|--> |-------| | ^ v Command Report/Display 1.Command is given to transfer x(GB?) to Destination. 2. Source looks and sees Buffer- 100'of MB?. Fills Buffer. 3. Buffer reports quantity Q filled, system displays Q/x transferred. WRONG! data hasn't even got as far as the Medium. 4. Medium meanwhile reports "I accept quantity q at a time for transport". 5. Buffer delivers q to Medium, it's sent and Medium reports Q-q to system, "I need filling". 6. Source delivers q to refill Buffer, return to step 3 until Q = 0. Better system? Source Buffer Medium Destination | ---|--> |-------|-> | ^ v<- - - - -(checksum) Command <- calc/check v Report/Display 1.Command is given to transfer x(GB?) to Destination. 2. Source looks through chain to destination and sees Medium (or sequence of). 3. System creates Buffer q, enough to fill smallest (or first?) buffer in Medium chain. 4. System fills Buffer. 5. Buffer delivers q to Medium, it's sent and Medium reports "I need filling". 6. Using checksum 'back-channel', verify arrival intact. If incorrect checksum, error code sent to Buffer,Medium,Destination and transfer abandoned, else add q to 'delivered' sum D. 7. System displays D/x transferred 8. If D<x, return to step 4 ... Realistic? Helpful? Cheers, Dex
*** Bug 326744 has been marked as a duplicate of this bug. ***
*** Bug 416248 has been marked as a duplicate of this bug. ***
*** Bug 416783 has been marked as a duplicate of this bug. ***
I am still getting this bug on plasma 5.22, OS: KDE Neon
Still experiencing this issue in Kubuntu 21.04 and Manjaro 21.0.7 and I believe it affects any file transfer from one medium to another; I can replicate the issue with SATA to SATA, SATA to USB and SATA to WEBDAV. As far as I can tell, this is not an issue within other DEs I have used (mainly XFCE and Cinnamon). This is the difference between a complete, beautiful, fun-to-use D.E. and a broken system. It is a deal-breaker, for me. I don't feel like it is my place to criticise people doing things that are beyond my ken, and if it was a simple fix, it would have been done by now. Additionally, I suspect it causes large webdav file transfers to fail under KDE. My experience is, transferring files larger than 2GB by webdav fails in KDE but never demonstrates an issue in other DEs.
*** Bug 442536 has been marked as a duplicate of this bug. ***
Marking critical as this can result in data loss if the user does a move operation rather than a copy operation.
*** Bug 443126 has been marked as a duplicate of this bug. ***
Does anyone have any suggestions on how to fix this issue that has been around for nearly 10 years now?
Disabling write cache for a USB stick did show the real write rate, but it caused a 75% increase in overall transfer time. Would KDE be able to read the write rate to the USB stick instead of the write rate to the cache?
(In reply to francois072 from comment #27) > Does anyone have any suggestions on how to fix this issue that has been > around for nearly 10 years now? I have a C demonstration on how to fix this: https://github.com/RomuloPBenedetti/SaneFileTransfer Basically it consists on creating a dynamic buffer in userspace that monitors fdatasync and through this, calculate ideal user space buffer size, so we don't send the entire file to the system cache.
(In reply to Christoph Feck from comment #1) > > therefore it shows that the copying operation is finished when it has not even been started > > Do kernel developers also know a POSIX interface way to tell us when and how > fast this buffered data is actually written to the disk, without causing > massive slow downs that any "sync" type calls would cause? About sync slowdowns, not sure about back there in 2011 but I believe calling fdatasync on the destiny file descriptor will not perturb system performance.
I confirm this issue. Copying an ISO file to usb stick takes some minutes but clicking remove with security takes longer than normal. It disappears in Dolphin but in notifications continues with msg: removing device.... OpenSUSE 15.3 Dolphin 20.04.2
The message displayed in the Disks & Devices applet was just improved to handle this case: https://invent.kde.org/plasma/plasma-workspace/-/merge_requests/1367
(In reply to Nate Graham from comment #32) > The message displayed in the Disks & Devices applet was just improved to > handle this case: > https://invent.kde.org/plasma/plasma-workspace/-/merge_requests/1367 I don't see problem with the actual msg, but thus with the operation that never finishes and invariably corrupts usb content.
It should finish *eventually*, even if it takes a long time. If it doesn't, that's a separate bug.
Created attachment 147502 [details] video example As you can see at the attached video (don't pay attention to the second 11 glitch, I moved a window by mistake while recording the video), after the progress bar has arribed to the end, we must wait for a minute (or more) before can unplug the USB disk.
This happens to me in KDE neon when show to my wife how to move a cartoon film for our son to an USB drive. If don't pay attention to this notification, you cannot unplug the disk safely. Maybe can be included into 15-minute bug initiative?
Once we broaden it to include Frameworks bugs, definitely. Right now we're only doing bugs in Plasma-aligned software. None of that prevents anyone from fixing it anyway, of course. :)
*** Bug 457959 has been marked as a duplicate of this bug. ***
The Linux kernel recently got a new "cachestat" syscall. It might help to find out if pages have been written to disk.
(In reply to Christoph Feck from comment #39) > The Linux kernel recently got a new "cachestat" syscall. It might help to > find out if pages have been written to disk. https://www.phoronix.com/news/Linux-6.5-MM-cachestat Next glibc will expose it: https://sourceware.org/git/?p=glibc.git;a=commit;h=72511f539cc34681ec61c6a0dc2fe6d684760ffe
Does glibc 3.39 already have cachestat or is it pending for glibc 3.40?
__NR_cachestat is exposed with glib 2.39. I quickly wrote a test, and the syscall works (nr_dirty shrinks gradually while the data is being written). I only tested with rotating rust, not slow USB, though. Some issues: * The call is per file, not per fs. * You have to keep all files open and repeatedly call it for them as long as nr_dirty != 0. An idea could be to try the largest files first, and hope that the smaller files have "long" been written out. * Unsure if nr_dirty is set to 0 when the mm requested the last page to be written out, or if it waits for the device driver to indicate that the device has _actually_ finished writing the data. * The fs needs to update metadata, so even if the last file has nr_dirty == 0, you wouldn't know how long the fs needs to write it. So the "unmount spinning indicator" MUST STILL BE RESPECTED to avoid data loss, even if this syscall was used to monitor writeback progress.
(In reply to Christoph Feck from comment #42) > ... __NR_cachestat is exposed with glib 2.39. I quickly wrote a test ... How do you call this? Do you have a code snippet you can share? And, sorry naive question, what would happen with systems that do not support it?
io_uring can do partial fsync, https://man7.org/linux/man-pages/man3/io_uring_prep_fsync.3.html Potentially as the input file is read, that is it should have a negligeable impact on perf. io_uring should be more performant than the current copy_file_range implementation.
A possibly relevant merge request was started @ https://invent.kde.org/frameworks/kio/-/merge_requests/1718
A possibly relevant merge request was started @ https://invent.kde.org/frameworks/kio/-/merge_requests/1764
A possibly relevant merge request was started @ https://invent.kde.org/system/dolphin/-/merge_requests/862
*** Bug 497286 has been marked as a duplicate of this bug. ***
Created attachment 177238 [details] It fails to recognize that the transfer is completed. File transfer box is stuck at 99% with no transfer speed when moving massive files to an Android phone. This must be a bug on Dolphin's end because I always manage to fully extract the supposedly 1% unfinished .tar.gz file after I copy the same file from the phone back into my notebook. Curiously enough, the file transfer always concludes when transferring back. Which means that Dolphin doesn't recognize the fact that the transferring has completed, even though its transfer speed has stopped and disappeared from the box.
Created attachment 177239 [details] Reaching 100% completion doesn't mean it's 100% completed.
Reducing the cache usage for saving files to USB devices has been solving this problem for the past 2 years. Here is how we are using it, but basically it is a noudev rule that calls a simple script, where $1 is the USB device to limit the RAM cache size while copying files, we are using a 16 MB limit, which is enough so that the recording does not become too slow and at the same time when completing a recording, almost immediately the device can be ejected. echo 1 > /sys/block/$1/bdi/strict_limit echo 16777216 > /sys/block/$1/bdi/max_bytes https://github.com/biglinux/usb-dirty-pages-udev
*** Bug 500917 has been marked as a duplicate of this bug. ***