Bug 333436 - kioclient enumerates files/folders incorrectly resulting in incomplete copies and move operations
Summary: kioclient enumerates files/folders incorrectly resulting in incomplete copies...
Status: RESOLVED FIXED
Alias: None
Product: kio
Classification: Frameworks and Libraries
Component: kioslave (show other bugs)
Version: unspecified
Platform: Ubuntu Linux
: NOR critical
Target Milestone: ---
Assignee: Konqueror Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-04-15 03:07 UTC by Jason Straight
Modified: 2015-10-24 23:44 UTC (History)
7 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jason Straight 2014-04-15 03:07:45 UTC
When copying, moving, or getting properties of certain folders (usually large) from media containing filesystems such as ntfs-fuse, samba, sometimes btrfs, kioclient will not build a complete list, resulting in incomplete copies and moves.

Reproducible: Sometimes




Reproduction of this bug has proven to be a little more difficult than just copying any folder from one of the various formats it has problems with.

With my "Pictures" folder containing almost 7000 photos it happens every time. However, I've tried creating large folders containing subfolders with several thousand files on the filesystems I've found to be affected and haven't been able to reproduce it reliably.

On btrfs it happens far less.

I've found that I can exacerbate the problem if I place the source media under a load. It's possible this could affect all filesystems on a machine that is under load and returns stat calls slowly.

Choosing to view properties of a folder affected by this bug will show a different count than what's actually in the folder. Pressing "refresh" will yield random totals each time. When you copy the folder the copy will be incomplete, and because it enumerated the list incompletely in the first place it will show no errors.

Your copy or backup will be incomplete. Which is why I'm marking this critical, as I nearly lost irreplaceable data when I thought I had backed up my photos to several different media and it missed about half of them every time. It makes the basic function of copying files with KDE very untrustworthy.
Comment 2 Christoph Feck 2014-05-04 21:58:38 UTC
Reading the description, are you implying that copying/moving _to_ the external file systems works without problems? So enumerating files works OK with ext4, but not with other file systems?
Comment 3 Jason Straight 2014-05-05 01:07:02 UTC
This is totally about the source filesystem. In fact you don't need to even move or copy the files to see the error. Just right clicking a folder on one of the affected filesystems and choosing properties can give you an invalid count of files and size.

Subsequent clicks of "refresh" in the properties window will yield randomly different results each time.
Comment 4 Dawit Alemayehu 2014-06-15 18:33:11 UTC
We have similar reports like this where people actually complained about data loss. The unfortunate thing is that I personally cannot reproduce the problem. Regardless of which filesystem I mount or what ioslave I use. I mounted an NTFS partition through Dolphin and copied large directories back and forth to it many many times and not once did it not completely copy the source.

For the smb ioslave I can tell you that symlinks will not be copied if you attempt to copy a local directory with symlinks in it into an smb share through the kioslave. I have no idea what happens if you actually mount the smb share. For the record, local (read: locally mounted) and remote filesystems are handled a bit differently so not all of these problems might apply under both circumstances.
Comment 5 Jason Straight 2014-06-27 14:12:41 UTC
I've noticed that too. I've tried to recreate by creating massive collections of files and folders with a bash script but have been unable to reproduce it that way too. I wouldn't be opposed to allowing access to a remote desktop [vnc or whatever] session.
Comment 6 Reimar Imhof 2015-01-10 20:46:41 UTC
Same problem here.
I've tried to copy a directory from a server to a client.
Server: real machine
Client: some tests with a virtual machine (virtualbox), some with an real notebook
KDE: 4.14.3, openSuse 13.2, 64 bit (incl. all available patches)

Directory includes about 1500 files, size about 10 000 000 bytes
Test 1: Server running Samba, Client dolphin with smb://server/...
about 400 files missing. Lots of files got wrong timestamp (timestamp of copying, not timestamp of original file.
Test 2: Server running Samba, Client mounts samba share. Still hundrets files missing.
Test 3: Server running sshd, Client using fish://server/... Here to: hundrets of files missing.

When I want to copy huge directories from a samba-share I have to use a non-kde file manager.
Comment 7 Cacciatore 2015-09-17 20:02:40 UTC
Same here. I had to raise a bug report cause i thought this is related somehow to dolphin. 

https://bugs.kde.org/show_bug.cgi?id=352761
Comment 8 wynajem_pn 2015-10-02 10:56:08 UTC
I've just copied 50 old dvd's with some backup files to an external usb drive (using Krusader) and at the end I noticed that there are lots of missing files on the usb drive. When I checked this out it turned out that not a single DVD was completely copied to the usb drive! To be honest I was shocked that modern version of OS may have such a bug. Moreover, from my point of view it is trivial to reproduce the bug (as I said not a single dvd was properly copied), hence I cannot believe that developers didn't noticed it. 
My setup:
Ubuntu 14.04.3 LTS (with all updated installed)
krusader: 1:2.4.0~beta3-2
kdelibs-bin: 4:4.13.3-0ubuntu0.2
Now I am a bit scared how many times I didn't noticed that the 'copy' command in krusader is not reliable.
Comment 9 wynajem_pn 2015-10-02 11:50:44 UTC
I made a simple screen-cast to show you the bug in action:
http://sendvid.com/dc92o1yu
As I said, I have no problem with reproducing the bug, so feel free to ask me for help in debugging an issue.
Some additional info:
It seems that the destination of 'copy' doesn't matter - it is exactly the same with usb and sata drives (both ext4). The source is ISO 9660.
Each time I press 'copy' on the folder the command works better - it finds more files to copy. After 2 or 3 repeats it does recognize all files. After unmounting/mounting everything starts from the begging.
Comment 10 David Faure 2015-10-24 17:54:22 UTC
Git commit e60e8b96b211290e330c7ff6a3b84b20ab85850d by David Faure, on behalf of Alberto Jiménez Ruiz.
Committed on 24/10/2015 at 17:53.
Pushed by dfaure into branch 'KDE/4.14'.

Fix double-emit of result and missing warning when listing hits an inaccessible folder

REVIEW: 125613

M  +13   -1    kio/kio/job.cpp
M  +1    -0    kio/kio/jobclasses.h
M  +41   -0    kio/tests/jobtest.cpp
M  +1    -0    kio/tests/jobtest.h

http://commits.kde.org/kdelibs/e60e8b96b211290e330c7ff6a3b84b20ab85850d
Comment 11 David Faure 2015-10-24 18:02:27 UTC
Git commit 5804d7019e53bf1b551ffd4f013c4ddcbee41428 by David Faure.
Committed on 24/10/2015 at 18:02.
Pushed by dfaure into branch 'master'.

Fix double-emit of result and missing warning when listing hits an inaccessible folder

Forward-port of a kdelibs-4.14 patch by Alberto Jiménez Ruiz

Change-Id: I56eadd06c063e6e81da51599f7c1cdd18f7e7704

M  +36   -0    autotests/jobtest.cpp
M  +1    -0    autotests/jobtest.h
M  +11   -1    src/core/listjob.cpp
M  +1    -0    src/core/listjob.h

http://commits.kde.org/kio/5804d7019e53bf1b551ffd4f013c4ddcbee41428
Comment 12 Christoph Feck 2015-10-24 23:44:31 UTC
Reading comment #3 (especially the last sentence), I doubt the commit fixes the reported issue.