Bug 162211 - Copying to an external causes lots of missing files
Summary: Copying to an external causes lots of missing files
Status: CLOSED FIXED
Alias: None
Product: frameworks-kio
Classification: Frameworks and Libraries
Component: general (show other bugs)
Version: unspecified
Platform: Ubuntu Linux
: NOR critical
Target Milestone: ---
Assignee: David Faure
URL:
Keywords: triaged
: 205428 262768 320890 324606 335924 336624 352761 357875 (view as bug list)
Depends on:
Blocks:
 
Reported: 2008-05-17 15:35 UTC by Jonah Naylor
Modified: 2019-06-04 16:06 UTC (History)
63 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
output of kioclient copy (110.67 KB, text/plain)
2015-10-02 12:12 UTC, wynajem_pn
Details
Possible patch for bug 162211 (4.26 KB, patch)
2015-10-11 14:29 UTC, Alberto J. Ruiz
Details
Proposed patch (4.24 KB, patch)
2015-10-11 16:26 UTC, Alberto J. Ruiz
Details
More files than before (245.40 KB, image/png)
2018-11-06 20:22 UTC, Filip Fila
Details
DestinationList & TargetList (2.50 MB, application/x-7z-compressed)
2018-11-07 15:03 UTC, Filip Fila
Details
Archive containing the logs, and the tree, and the Seed.txt file to pass to the kiocopy script (18.56 KB, application/x-xz)
2018-12-18 02:08 UTC, bluescreenavenger
Details
Bash script of which to use to recreate the file tree that failed to copy. Pass the path to the Seed.txt to it. (17.80 KB, application/x-shellscript)
2018-12-18 02:09 UTC, bluescreenavenger
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jonah Naylor 2008-05-17 15:35:36 UTC
Version:           1.0.3 (using KDE 4.0.4)
Installed from:    Ubuntu Packages
OS:                Linux

ok this is a huge problem for me. i have an external usb 500gb hard drive. it's a toshiba and appears to work fine, mounts and shows up in the recently plugged in thing and in dolphin etc.

there is a serious issue here though, when i back up my files by selecting my home dir and copying and pasting to the usb drive it copies stuff over fine. or at least appears to. then when i go to the usb drive and click through folders and contents loads of stuff is missing and has not copied at all.

this is real bad cos you're not aware that it's happened so you think you've backed up when in fact you've lost a load of stuff.

it's the same if i just copy and paste folders within my home also, such as Documents folder. it won't copy all the documents across even though i get no warning or anything to say it can't, it just pretends it's worked.

this really needs fixing, anyway way i can help debug or help with this let me know. thanks.
Comment 1 FiNeX 2008-05-18 14:07:42 UTC
If you copy large amount of data from your internal HD to itself? Did you have the same problem?

What is the FS on the external usb HD?

Did you try to copy the same data using shell commands?

I set this as critical, because if confirmed it need to be fixed with maximum priority!!!!
Comment 2 Hanno 2008-05-18 23:15:49 UTC
USB sticks are usually FAT formated. The home folder contains e.g. the emails stored in kmail, which filenames do not go well with FAT formatted disks. Hence, all copied emails are lost. A warning during the copy process would be really nice, this problem prevented me from backing up my files for a very long time.
Comment 3 Jonah Naylor 2008-05-20 19:40:59 UTC
it's fat32 formatted.

copying copying from one partition to another seems ok on same hardrive but unsure as i can't test with loads of data as don't have enough disk space.

haven't tried shell commands, not sure of best ones to try using? can you advise on commands and i'll report back. thanks for looking into this...
Comment 4 Jonah Naylor 2008-05-21 23:36:16 UTC
ok i tested by instead using konqueror to copy the files in the same way i would in dolphin. konqueror worked great and backed up all the data to the usb as it should...
Comment 5 George Goldberg 2008-08-04 19:57:14 UTC
@Jonah:
Any chance you can try this again with KDE 4.1? Otherwise, can you give us some examples of the full paths (including names) of files which do not copy. Is it consistently the same files that do not copy if you do it multiple times, or is it different files that don't copy each time you try?
Comment 6 Richard Hartmann 2008-09-24 14:02:07 UTC
Jonah: Any update on this?
Comment 7 Markos Chandras 2008-10-13 17:14:22 UTC
I can verify this problem. I am using kde-4.1.2 and when I try to copy data to my usb flash ( kingstone data traveller 2GB ) which is formated in fat32 i get nothing. The data seems to copy fine but when I plug the flash on another computer the data is missing. The usb flash works fine when I do the same stuff from kde3 
Comment 8 Richard Hartmann 2008-10-13 17:32:10 UTC
As this is now confirmed, any objection against giving it a higher priority?
Comment 9 Richard Hartmann 2008-10-13 17:34:51 UTC
Markos: Can you provide as much information on how to reproduce this as possible? Can you reproduce this? Does exactly the same thing happen ever time?
Comment 10 Markos Chandras 2008-10-13 17:39:28 UTC
Yes this happens every time with every usb storage device formated on fat32. Havent tried ntsf or something else.

The way to reproduce it is quite simple ( for me at least ). Just plug a usb device, try to copy a file using Konqueror-4 or dolphin. Then umount the usb device and plug it again. The file is not there ... 
Comment 11 Richard Hartmann 2008-10-13 17:49:32 UTC
Does it unmount cleanly? Can you try to run

  sync

before you detach?
Comment 12 Gudy 2008-10-17 19:12:39 UTC
I confirm this problem, i got the same issue, with an EXTERNAL WD 500mb harddisk formatted ext3 filesystem  on the usb2.0....I was moving all my music from the internal HD to the EXTERNAL HD, about 200GB the moving process was smooth, no troubles. But when the next day i was seaching some music, i found out that the first few files of a subdirectory were copied well (3..4 files)but the remaining 10 files in that subdirectory had a lenght of 0 (zero)bytes (average 50mb/file)
No error messages or whatever :(

maybe we need a verify option 
Comment 13 Markos Chandras 2008-10-17 21:02:15 UTC
sync command before removing the disk seems to works most of the time but NOT always. Still no error messages
Comment 14 Richard Hartmann 2008-10-19 14:00:09 UTC
OK, I just copied some stuff to a thumb drive and thought I would try and recreate the data loss, if possible (normally, I just use CLI).

Here's what I did

* Insert thumb drive
* Click the little plasma widget which informs me that a USB drive has been connected, this opens Dolphin
* Copy stuff
* Click the volume on the left
* Select safe unmounting

I then ran mount on the CLI and lo, behold:

/dev/sda1 on /media/disk type vfat (rw,nosuid,nodev,uhelper=hal,uid=1000,codepage=437,iocharset=utf8)


Dolphin does NOT complain that it could not unmount the drive! It just fails and is silent about it. A user will, rightly, expect that it has been unmounted cleanly.

Unmounting by hand (after moving to a different location within Dolphin as Dolphin held the mount directory open) works just fine. No data loss.


Can anyone please confirm/deny that they acted in a similar way when their data loss occured? If you can confirm, we found the issue. If you can't, I will open a new bug.
Comment 15 Richard Hartmann 2008-10-19 20:21:33 UTC
Markos: When you sync manually, do you wait until the diode on your thumb drive stops blinking? It's usually a good idea to wait for two seconds after you are sure it has stopped, because, sometimes, it has not.

Please note that the sync command (same as umount, which calls sync(), as well), may return normally, i.e. without error code when the hardware itself is still busy.
Comment 16 Markos Chandras 2008-10-19 20:41:07 UTC
Yeah , indeed im waiting until the led turns off. Unfortunately i cant re-produce it always :(
Comment 17 Max 2009-05-26 13:20:29 UTC
I suppose copying itself is not the source of these problems. I've noticed that copying to my flash mp3 player is a lot faster in KDE4, but it seems that data is not actually written, but rather cached somewhere. So, when trying to umount the drive through plasma applet or dolphin panel, I have to wait for a 1-3 minutes(!) while the data (~100MB usualy) is actually being written. The device indicates writing process. Dolphin, waiting the umount, usualy gets timeout from dbus daemon and shows a message in the bottom yellow panel.
IMO copying/caching routines to such devices should be reviewed.
Comment 18 Pieter Vande Wyngaerde 2009-06-16 11:22:11 UTC
isn't there a "flush" mount option for removable media to write the stuff directly ?
Comment 19 Richard Hartmann 2009-06-16 13:45:15 UTC
Yes. Maybe it should be enabled by default..?

Problem is how do you determine when you want this or not? Especially with external drives you actually work on, this would make things _really_ slow.
Comment 20 Alexander 2009-07-22 09:19:38 UTC
I can confirm this and related problems with my Debian/SID installation (KDE 4.2.4). It seems to be a general problem of KDE4 handling files on a external USB drive.

I just lost a lot of pictures from my camera's SD card connected to my desktop PC via an USB reader. When trying to copy some files on my PC's hard drive I got error messages. The resulting files were of correct size but contained no information. Unfortunately, the original files were corrupted somehow during the process...
Comment 21 Richard Hartmann 2009-07-22 10:27:57 UTC
IMO, that either points to a general problem with the reader/SD card/partition/formatting or KDE would do something _really_ stupid.

Did you m5 the files to see if corrupted original and copy are the same?
Comment 22 Richard Hartmann 2009-07-22 10:28:17 UTC
md5, I mean..
Comment 23 Gudy 2009-08-21 17:32:19 UTC
please I wonder what is now the status of this serious bug, because its now a year ago reported, and at that time I still got kubuntu 8.04 with kde 3.5.. now 2 distro's later its still there, I checked this out again on a new laptop & desktop kubuntu 9.04 kde 4.2.. and tried to copy 50gb to my external harddisk , formatted ext3... still zerolength files without any message... You think you backup your data safely
Comment 24 Richard Hartmann 2009-08-21 17:47:05 UTC
Can you install 4.3, try to reproduce and tell us _exactly_ what you did?

Others: is there any sane way he can trace what is happening?
Comment 25 FiNeX 2009-08-28 01:28:48 UTC
*** Bug 205428 has been marked as a duplicate of this bug. ***
Comment 26 Supreme1012 2009-08-28 01:38:23 UTC
My bug was marked as a dup so here's the info from my own experience.

"I have a WD 1 tb external hard drive and I have twice now had almost all the
selected files deleted while moving from and to the external hard drive. Both
occasions involved more than 50,000 files(both times were photos), and I used
the "move to" command. The notification showed up but after a long time with
not activity it showed "completed moving x files" but the x number was only a
few hundred instead of the full selection. Since I used the "move to" command
there was no way to recover any of the files, undo move just moved the couple
hundred files back to the previous directory and the thousands of other files
disappeared as if they had never been.

running OpenSUSE 11.1 x64 with KDE 4.3 Factory"

also the 1 tb external is ntfs formatted with a mini usb connection and moving the same files within my internal hard drive works fine.
Comment 27 Patrick Stewart 2011-09-01 00:21:58 UTC
This bug still seems to exist. I just copied ~60000 files to an external (esata) backup drive, and hundreds of files weren't copied at all. The missing files were completely random (eg one file in a photo/music album). Fortunately, I happened to notice that some files were missing and recopied them all over the top (skipping overwrites). It seems that all the files have transferred now, but this is a very serious bug, and it's lucky I checked before deleting the originals.
In case it is relevant, the files were copied by drag and drop in dolphin, and both drives were ntfs and attached via sata.
Comment 28 Gudy 2011-09-01 01:27:17 UTC
At one point i'm glad you confirmed my issue
this i reported already long long time ago (i guess 2008)
Its not related to ntfs, as in my case i used ext3 as format for the  
external hd
When you use your external usb hd for home partition backup, all the time  
you get useless backups when you check it

....i wonder when there will be some action taken :).....

regards
Jazz


On Thu, 01 Sep 2011 07:21:59 +0700, Patrick Stewart <patstew@gmail.com>  
wrote:

> https://bugs.kde.org/show_bug.cgi?id=162211
>
>
> Patrick Stewart <patstew@gmail.com> changed:
>
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |patstew@gmail.com
>
>
>
>
> --- Comment #27 from Patrick Stewart <patstew gmail com>  2011-09-01  
> 00:21:58 ---
> This bug still seems to exist. I just copied ~60000 files to an external
> (esata) backup drive, and hundreds of files weren't copied at all. The  
> missing
> files were completely random (eg one file in a photo/music album).  
> Fortunately,
> I happened to notice that some files were missing and recopied them all  
> over
> the top (skipping overwrites). It seems that all the files have  
> transferred
> now, but this is a very serious bug, and it's lucky I checked before  
> deleting
> the originals.
> In case it is relevant, the files were copied by drag and drop in  
> dolphin, and
> both drives were ntfs and attached via sata.
>
Comment 29 Jaime Torres 2011-09-27 10:42:09 UTC
There is work in progress...

https://git.reviewboard.kde.org/r/102388/
Comment 30 BartOtten 2011-10-11 23:43:10 UTC
For what it's worth: Mac OS X's Finder and Windows 7 Explorer suffer the same problem. I hope it is fixed soon so we are ahead of those two ;-)
Comment 31 Gudy 2011-10-12 03:25:54 UTC
its a long standing bug already, 3...4years, it only happened with usb
attached drives
if i copy to a network attached storage (NAS) no problems at all
and its not only dolphin, also krusader, so i think its a kernel issue



On Wed, Oct 12, 2011 at 6:43 AM, BartOtten <bartotten@kde.nl> wrote:

> https://bugs.kde.org/show_bug.cgi?id=162211
>
>
>
>
>
> --- Comment #30 from BartOtten <bartotten kde nl>  2011-10-11 23:43:10 ---
> For what it's worth: Mac OS X's Finder and Windows 7 Explorer suffer the
> same
> problem. I hope it is fixed soon so we are ahead of those two ;-)
>
> --
> Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.
>
Comment 32 David Edmundson 2013-09-07 13:11:40 UTC
*** Bug 324606 has been marked as a duplicate of this bug. ***
Comment 33 Dhaval 2013-09-07 16:46:30 UTC
http://fpaste.org/37829/78572315/

This is my output from kdebugdialog when copying another batch of files. Some folders are empty, however the number of empty folders is quite less than I expected.
Comment 34 kkoksvik 2013-09-25 20:56:57 UTC
I can confirm that copying large folder-structure between to external usb disks using  Dolphin , after copying finished many files where not copied an it looks arbitrary witch files got copied or not. most folders came over(can not check them all thousands of folders)
I am running Mint 14 KDE 4.11
Comment 35 Dhaval 2013-10-02 15:41:36 UTC
It happened again, this time on a small folder. I was copying a folder with about 10 large .mkv files from an External NTFS drive into a ext4 drive, and it did not copy.
Comment 36 Martin Sandsmark 2013-10-07 08:48:31 UTC
From reading the comments it seems like this is about Plasma/Dolphin/Solid not reporting properly when it cannot unmount leading to people pulling out their USB volumes prematurely, which now works for me (in 4.11, at least, probably earlier as well); I get a warning message if the volume cannot be unmounted.
Comment 37 Gudy 2013-10-07 09:29:59 UTC
Martin, this is NOT the case, we talk about usb connected harddisks, no
flash sticks,the harddisk is near permanently connected, the harddisk led
shows no activity for long long time, the system notifier also shows copy
done,its also not the disk cache "not flushed" as it goes about many
gigabytes missing much larger then the amount of ram and all programs
dolphin,nautilus,krusader......report copy is done.Next time when you
logon, many files zero length randomly
i think this issue is already since 2008


On Mon, Oct 7, 2013 at 3:48 PM, Martin Sandsmark
<martin.sandsmark@kde.org>wrote:

> https://bugs.kde.org/show_bug.cgi?id=162211
>
> Martin Sandsmark <martin.sandsmark@kde.org> changed:
>
>            What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>              Status|CONFIRMED                   |RESOLVED
>                  CC|                            |martin.sandsmark@kde.org
>          Resolution|---                         |FIXED
>
> --- Comment #36 from Martin Sandsmark <martin.sandsmark@kde.org> ---
> From reading the comments it seems like this is about Plasma/Dolphin/Solid
> not
> reporting properly when it cannot unmount leading to people pulling out
> their
> USB volumes prematurely, which now works for me (in 4.11, at least,
> probably
> earlier as well); I get a warning message if the volume cannot be
> unmounted.
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>
Comment 38 Richard Hartmann 2013-10-07 09:53:19 UTC
Having been involved with debugging in 2008(!) and pointing out the umount issue myself, I agree that not all of these comments read that way.

In particular, https://bugs.kde.org/show_bug.cgi?id=162211#c34 specifically states that 4.11 is still affected.
Comment 39 Martin Sandsmark 2013-10-07 14:24:07 UTC
(In reply to comment #37)
> Martin, this is NOT the case, we talk about usb connected harddisks, no
> flash sticks,the harddisk is near permanently connected, the harddisk led
> shows no activity for long long time, the system notifier also shows copy
> done,its also not the disk cache "not flushed" as it goes about many
> gigabytes missing much larger then the amount of ram and all programs
> dolphin,nautilus,krusader......report copy is done.Next time when you
> logon, many files zero length randomly
> i think this issue is already since 2008

Yes, but did you actually unmount it cleanly?

(In reply to comment #38)
> Having been involved with debugging in 2008(!) and pointing out the umount
> issue myself, I agree that not all of these comments read that way.
> In particular, https://bugs.kde.org/show_bug.cgi?id=162211#c34 specifically
> states that 4.11 is still affected.

Still could just be an unclean unmount in that comment.

Can someone definitely confirm that they _cleanly unmounted_ the drive in question? Not just "I think I saw the safe to remove" message, but actually checked with for example `mount`?

I went through all the code paths that are involved yesterday, and while I may have missed something there (obviously) are no glaring errors in it. I have also tested this myself with my own removable drives and with files of all sizes and weird folder names, and I have been unable to reproduce it.
Comment 40 kkoksvik 2013-10-07 14:48:41 UTC
(In reply to comment #39)
Mint 14 KDE 4.11
I will try to explain with more details what I did when this happened.

Mounted two 1.3 TB USB2 NTFS drives using Dolphin with the intention of merging the two so I could free up one of them for a different purpose.

Selecting all files on one drive i dragging them over to the other drive. The copying takes a long time and when i return the copying process is done. I immediately go looking for a file that is supposed to be copied over and can not find it, looking at other files I recognize that many files are missing, folders seams to have been copied OK. 

Trying to reproduce this on smaller scales with only copying one folder with files that did not copy the first time ends up with same result. Only way to copy the files that did not get copied is to only copy one file at a time.

To me this seams like a problem with selection and iteration over the selection(could be specific to ntfs) 

I do not currently have access to the hardware that i used, but i will next week, would be more than happy to help investigate.
Comment 41 Martin Sandsmark 2013-10-07 21:39:16 UTC
Could you try copying with the command line utility kioclient (kioclient cp /src/ /target/)?

Also, could you share the full paths of the folders that didn't get copied, as well as the target full path?
Comment 42 Gudy 2013-10-08 02:54:29 UTC
I don't think it has to do with unmount, as by a normal system shutdown and
i not mean by pulling the plug, all drives are unmounted before the system
powers off itself.
But also by not logout, no screensaver or whatever to interfear with the
copy process it still happened
As it also seems to not makes any difference
 what file system you use ntfs, ext2 ext3 ext4,or  what application you
use..the issue is possibly deeper inside the kernel at the usb part, as
copying the same data to a NAS via ethernet 10/100, not wifi, has no
problems at all

I'm not a linux kernel expert, but i hope to be able to guide in what
direction they must search.

Its a real serious issue, as more and more people want to backup there
photo's ,movies,music and the "simple way" is get an external usb harddisk
to save the stuff and forget about it, knowing your data is there safe.....
to figure out later(hopefully the next day, and not next year that you
remember "oh yeah i had put it on my backup hd" )and that many files are
missing

Of course after the issue arrived , i never used a usb harddisk as backup
anymore
just put a new hd in the NAS (only on when i need it) copy the files to the
nas,and yes still verify it,  it never failed
So there is a clear difference in how files are handled over ethernet and
usb

It is also not a usb harddisk power supply issue as i used a 12v 7ah lead
battery to power the usb hd (it had a 12v dc external power supply)
and used a shielded 20cm usb cable on a usb2 port

I will retry in one of the next weekends  to copy by usb and post the
results including logs
Let me know what is the most usefull

best regards


On Tue, Oct 8, 2013 at 4:39 AM, Martin Sandsmark
<martin.sandsmark@kde.org>wrote:

> https://bugs.kde.org/show_bug.cgi?id=162211
>
> --- Comment #41 from Martin Sandsmark <martin.sandsmark@kde.org> ---
> Could you try copying with the command line utility kioclient (kioclient cp
> /src/ /target/)?
>
> Also, could you share the full paths of the folders that didn't get
> copied, as
> well as the target full path?
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>
Comment 43 feniksa 2013-10-21 15:11:05 UTC
Confirm, that this bug still exists:

How to reproduce:
1. Insert any removable media
2. Start copy with dolphin any folder with a lot of files. I tried to copy linux sources.
3. After some time i got such error: 
https://www.200volts.com/uploads/2013/10/21/526538c85ba02_snapshot1.png

Additional information:
1. Media free space: 
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1       7.5G  1.2G  6.3G  16% /run/media/feniks/LIVE

2. Media information: 
feniks@anna /mnt/webos_dev/linux $ sudo fdisk -l /dev/sdc

Disk /dev/sdc: 7963 MB, 7963803648 bytes, 15554304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc3072e18

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *          56    15554303     7777124    c  W95 FAT32 (LBA)

3. Kdelibs version: 4.10.5-r1
4. Linux distro: gentoo
5. kernel: 3.10.7-gentoo-r1
Comment 44 Andi 2014-01-14 19:34:37 UTC
copy result seems to be predictable

I've got a folder on a NAS with about 2.000 files (10GB). Nearly each time the folder is copied to the internal Notebook SSD the number of copied files is different. kioclient used.

If you look at the copy icon in the task bar while a copy task is running you can see how many files in total will be copied. Just click on the plus button below the progress bar. 
I made the experiance that in error case this value does not match with the total number of files in the source folder. The result is an incomplete target folder. But the target folder will always have exactly the number of files shown in the task bar info while copying - the wrong result is predictable.

A second point I recognized is that if you repeat the copy procedure the result will always be different in the number of copied files. 

It seems that the file structure of the source folder is not read completely by the KDE copy module.

The file names are quite long (2013-07-30 15.25.30 Canon EOS 650D.jpg) and the NAS is not that fast (WD MyBook - ARM CPU). This might encourage the occurcance
of the error.

Using rsync to copy the folder works without errors on same setup. Performance of NAS/network is about 30MB/sec reading for huge files. NAS is mounted as NFS. fstab: 192.168.178.100:/nfs/Public /media/public nfs rw,noauto 0 0. 

System:
Notebook i3/SSD
Debian Testing/KDE
NAS WDH1NC (ARM CPU)
FritzBox/Fixed IP
Comment 45 Oleg 2014-02-27 17:45:10 UTC
I am having this issue when copying a few folders from my ext3 home directory into USB flash drive or microSD (both Fat32) with Dolphin. I'm using Kubuntu 12.04 (all software up to date).
Comment 46 Jason Straight 2014-03-31 01:35:02 UTC
6 year old critical bug still exists. Just noticed it cost me most likely the only existing baby photos of my son. Unfortunately, my backups [to three other media] were made with KDE.

I'm currently sick over this.

A lot of good it did me to have 3 backups, when they were all incomplete. Maybe if the bug isn't going to get fixed a warning should pop up.
Comment 47 Jason Straight 2014-04-01 00:47:42 UTC
Okay, I can't imagine this would be hard to fix, after doing some more investigating.

I have been playing with this and had similar results as Andi, plus noticed the following:

Every time I copy the Pictures folder to anywhere, not just remote media, it drops more than 1/2 the files with different numbers every time.

I've also found that simply right clicking the folder and choosing properties will only give me a partial count, and every time I hit refresh - a different total of files.

In total, this folder has 6677 files.

I think this has to do with the speed the query for files vs how long the kio waits. And it has everything to do with the source, not the destination.

I have two external btrfs formatted HD's which [when checking properties in KDE] show a count of about 3400 at first, but show the right total every time afterwards hitting refresh in properties. After that count shows the actual total of files, then copying them will copy them all.

I have one copy on an NTFS partition on my SSD [fuse - slow], which never shows the right totals, no matter how many times I refresh. Anywhere between 2400 and 3000.

Another copy [mounted via cifs] on wireless only returns a count of about 1600 in properties.

Testing copying to various slow media such as flash drives and cifs mounts works fine as long as the source media is fast enough to return the proper total count.
Comment 48 Gudy 2014-04-01 03:12:00 UTC
please don't think light about this issue, its not about the number of
files , but many files become zero length without any message about why it
failed. it looks like a kind of buffer overrun, to help imagine the issue
think about the old rs232 and you connect the rts and cts pins(hardware
handshake )(which is no problem IF the receiver is faster then the sender),
now rhe receiver is slower then the sender, the receive buffer becomes full
and the sender keeps on sending more, resulting in a loss of data
Of course with usb its all software implementation, i can't figure out if
the usb harddisk not sends a command to stop  sending, or that the received
stop command is ignored,or delayed , just 1 byte missing from the file and
the checksum of the file is wrong ,possibly the harddisk controller puts
the file then at 0 length

Now the issue is why this goes wrong in linux without any message, in win
xp(since 2006 no win pc anymore,so i can't compare it with newer versions)
copying stops and you get frustrated why and waste of time , but at least
you know it wend wrong,better then thinking everything wend okay but when
you need your backup you have nothing or only a part

maybe this bug is  deeply rooted in the kernel, as i always wondered why if
i make an SD card for the raspberry pi with the dd command it mostly result
in a not working card(rpi not boot)(even adjusting block sizes and whatever
possible), but when using windiskimager never any issue

This bug is serious (your most valuable emotional data lost ) and
difficult(extreme low level hardware and software knowledge needed, and
during the years many guys tried to close this bug report without doing
anything about it




On Tue, Apr 1, 2014 at 7:47 AM, Jason Straight <j.straight-kde@straights.net
> wrote:

> https://bugs.kde.org/show_bug.cgi?id=162211
>
> --- Comment #47 from Jason Straight <j.straight-kde@straights.net> ---
> Okay, I can't imagine this would be hard to fix, after doing some more
> investigating.
>
> I have been playing with this and had similar results as Andi, plus
> noticed the
> following:
>
> Every time I copy the Pictures folder to anywhere, not just remote media,
> it
> drops more than 1/2 the files with different numbers every time.
>
> I've also found that simply right clicking the folder and choosing
> properties
> will only give me a partial count, and every time I hit refresh - a
> different
> total of files.
>
> In total, this folder has 6677 files.
>
> I think this has to do with the speed the query for files vs how long the
> kio
> waits. And it has everything to do with the source, not the destination.
>
> I have two external btrfs formatted HD's which [when checking properties in
> KDE] show a count of about 3400 at first, but show the right total every
> time
> afterwards hitting refresh in properties. After that count shows the actual
> total of files, then copying them will copy them all.
>
> I have one copy on an NTFS partition on my SSD [fuse - slow], which never
> shows
> the right totals, no matter how many times I refresh. Anywhere between
> 2400 and
> 3000.
>
> Another copy [mounted via cifs] on wireless only returns a count of about
> 1600
> in properties.
>
> Testing copying to various slow media such as flash drives and cifs mounts
> works fine as long as the source media is fast enough to return the proper
> total count.
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>
Comment 49 Jason Straight 2014-04-01 03:28:07 UTC
I think we are actually talking about two different bugs here actually.

1. A bug where KDE (kioclient?) will not recognize all the files in a folder/selection when it scans them prior to copying and when viewing properties. This leads to you trying to copy or move a folder with 6700 files and it actually only copies ~3400.

2. A different bug where you copy files and they exist on the destination, but they are 0 bytes sized.

I have not experienced #2 myself.
Comment 50 Jason Straight 2014-04-02 02:55:48 UTC
Further evidence suggests this is a time issue. Also, a way to help reproduce the issue.

On a USB3 mechanical drive formatted with btrfs filesystem I was not initially experiencing fluctuating counts with kioclient properties on the folder.

When I put that same drive under a load using dd to write a file to it from /dev/zero I started getting the problem where clicking refresh in properties would yield random results on size and file count.
Comment 51 Jason Straight 2014-04-02 21:44:16 UTC
It's not "just" speed. There's more to it.

It gets me the right counts on the NTFS fuse filesystem for Program Files and it takes a while to count them. For some reason the enumerating of files in my Pictures folder kicks out before it's done.

Is there maybe a max time kioclient will wait while stating each file? And perhaps one or more of my pictures takes a longer than usual to stat?

I'm open to try anything here. Give me some guidance on this. Hell, I'll even set up a virtual machine with ssh and vnc access for the devs with access to my photos.

I have kde debugging symbols installed, but my C skills are old and weak, and my gdb skills are even less useful.
Comment 52 Gudy 2014-04-04 03:24:55 UTC
Jason  please not only look at the number of files, verify the size of the
directory ,source and destination in case there are zero length files, the
names are in the directory, but the total files size count is different
example , source =2000 files , total 6.3 GB, destination = 2000 fles total
4.8 GB
got the same issue by ext2, ext3,ext4,ntfs formatted external usb 2.0 hd
from western digital, and seagate

After that i not used usb harddisks anymore as the backup is unreliable,
but each time there is a new distro upgrade, i still try ,it to see if the
issue still exist , and from 2008 until now it has not changed......in a
few weeks after upgrading to 14.04 i will test again



On Thu, Apr 3, 2014 at 4:44 AM, Jason Straight <j.straight-kde@straights.net
> wrote:

> https://bugs.kde.org/show_bug.cgi?id=162211
>
> --- Comment #51 from Jason Straight <j.straight-kde@straights.net> ---
> It's not "just" speed. There's more to it.
>
> It gets me the right counts on the NTFS fuse filesystem for Program Files
> and
> it takes a while to count them. For some reason the enumerating of files
> in my
> Pictures folder kicks out before it's done.
>
> Is there maybe a max time kioclient will wait while stating each file? And
> perhaps one or more of my pictures takes a longer than usual to stat?
>
> I'm open to try anything here. Give me some guidance on this. Hell, I'll
> even
> set up a virtual machine with ssh and vnc access for the devs with access
> to my
> photos.
>
> I have kde debugging symbols installed, but my C skills are old and weak,
> and
> my gdb skills are even less useful.
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>
Comment 53 Jason Straight 2014-04-15 11:02:06 UTC
I took my bug [as I think we're talking about two different ones] and created a new one: https://bugs.kde.org/show_bug.cgi?id=333436
Comment 54 drtwox 2014-04-24 23:22:03 UTC
Using whatever the version of Dolphin is in Kubuntu 13.10, I copied 3701 files in 1507 folders totaling 17.5GB to an external FAT32 drive, and Dolphin only copied 777 files and 777 folders totaling 3GB. That 777 looks very suspicious.
Comment 55 Arkadiusz 2014-05-11 07:50:16 UTC
It's a pity that this still exists (5 years without fix?)
Few days ago I lost tons of family photos in KDE 4 (Ubuntu 14.04) when doing move from internal drive to external drive (ctrl-x, ctrl-v). During that process Dolphin has crashed. After that, I noticed, that all marked files on internal drives were wiped out, but on external drive there was only some of them (as proces was interrupted by crash)

It seems that Dolphin wipes out source files without waiting for confirmation that files were successfuly moved to target location...

I think it should be treated as some kind of hot prio error. It is known since 2009, it can cause critical data loss.
Comment 56 Gudy 2014-05-13 08:15:17 UTC
maybe we have to file this by mr Linus Torvalds self or at the kernel core
development group(please give me a hint how/where) as it seems this  is not
the right place (or nobody has this specific knowledge)
....sad to see you used MOVE...so i can imagine the disaster..
.





On Sun, May 11, 2014 at 2:50 PM, Arkadiusz <arekws@gmail.com> wrote:

> https://bugs.kde.org/show_bug.cgi?id=162211
>
> Arkadiusz <arekws@gmail.com> changed:
>
>            What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>                  CC|                            |arekws@gmail.com
>
> --- Comment #55 from Arkadiusz <arekws@gmail.com> ---
> It's a pity that this still exists (5 years without fix?)
> Few days ago I lost tons of family photos in KDE 4 (Ubuntu 14.04) when
> doing
> move from internal drive to external drive (ctrl-x, ctrl-v). During that
> process Dolphin has crashed. After that, I noticed, that all marked files
> on
> internal drives were wiped out, but on external drive there was only some
> of
> them (as proces was interrupted by crash)
>
> It seems that Dolphin wipes out source files without waiting for
> confirmation
> that files were successfuly moved to target location...
>
> I think it should be treated as some kind of hot prio error. It is known
> since
> 2009, it can cause critical data loss.
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>
Comment 57 Dawit Alemayehu 2014-05-20 11:55:16 UTC
*** Bug 320890 has been marked as a duplicate of this bug. ***
Comment 58 Alex 2014-05-22 22:22:20 UTC
as I wrote  https://bugs.kde.org/show_bug.cgi?id=320890  
bug reproduced on my PC.

and this https://bugs.kde.org/show_bug.cgi?id=333436  also reproduced.
perhaps there is a connection between them.
Comment 59 Jaime Torres 2014-05-24 09:34:56 UTC
Perhaps the bug is related to the use of non-reentrant methods in kio (as signaled by cppcheck):

Non reentrant function 'readdir' called. For threadsafe applications it is recommended to use the reentrant replacement function 'readdir_r'.  (fixed in KF5).
Non reentrant function 'strtok' called. For threadsafe applications it is recommended to use the reentrant replacement function 'strtok_r'.
Non reentrant function 'getpwnam' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getpwnam_r'.
Non reentrant function 'getgrnam' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getgrnam_r'.
Non reentrant function 'getpwuid' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getpwuid_r'.
Non reentrant function 'getgrgid' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getgrgid_r'.
Non reentrant function 'getservbyname' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getservbyname_r'.
on reentrant function 'getprotobyname' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getprotobyname_r'.
Comment 60 dusijun 2014-10-31 15:59:44 UTC
It happend just now on my machine 

Ubuntu 12.10 +KDE plasma

From file manager nautlius , I copy  four folders around 180GB from PC to USB hardisk. 

After two folders, around 110GB are copied, I count the file number on the usb harddisk, there are same amount around 11k files as on my PC.

Then I stop file copying.

After that,  from  nautlius I try to copy the third folder over (around 7000+ files)  to my USB harddisk.

Cancel transmition  when there are 3000 files on the USB disk.

All looks good, counting files all, the same. So unmout, and remove USB disk.



When I open the usb harddisk on anther PC

There are only 2960 fiels lef t!!!!!!!!!!!!!!!!!!!!!!

Only a portion of first two folders.

The third folder and its subfolders DO NOT EXISTS at all.
Comment 61 Baconmon 2015-02-05 06:15:00 UTC
This bug just affected me!..
Actually I knew about this bug for a long time, but I stupidly forgot about it when I was copying files for some one!..
I am in japan and about a year ago I hacked some japanese family's wii that I am friends with.. I spent a long time downloading tons of wii games for them, and also lots of NES and SNES roms and put all the games on to a hard-drive attached to their wii..
This year they got a bigger HD for the games.. I stupidly forgot about this bug, and used KDE to transfer tons of small files from their old HD to their new HD..
It copied most (but not all) of the nes/snes roms, but only copied ONE wii game!..

I had to tell them that all of their wii games are gone (save for 1)  because of a giant bug in linux.. Do you know how bad that makes linux look?..

This bug is the ONLY reason I can't recommend linux to any one.. Who would want to use an OS that secretly erases your files when you transfer stuff via USB!??..

Tons of people can easily reproduce this bug, so why hasn't it been fixed in seven years??.. To me, this is by far the biggest bug in KDE.. I am just wondering how many more years it is going to be until I can safely transfer files using USB..

If this bug were in mac-os or windows, it would be fixed immediately, but since it is linux, and people only fix things when they feel like it, a colossal bug like this has still been unfixed for 7 years and counting..

I hope this bug is fixed this year.. I want to be able to talk about how great linux is again.. But I just can't recommend it when it has a bug this huge in it.. (And I am definitely not recommending gnome to any one though.. gnome sucks..)
Comment 62 juliaop 2015-02-12 05:12:59 UTC
use long path tool for easy solution.
Comment 63 Christoph Feck 2015-02-15 00:15:50 UTC
> Tons of people can easily reproduce this bug

But none of the developers. Also, it is still unclear if this issue is due to removing the external device too early, i.e. before all the cached contents is written out.
Comment 64 Baconmon 2015-02-15 00:51:43 UTC
> But none of the developers. Also, it is still unclear if this issue is due to removing the external device too early, i.e. before all the cached contents is written out.

I didn't realize that none of the developers can manage to reproduce it, that is really odd.. I think I am able to reproduce it on this computer, but not sure if I would be able to give any information that would help?.. I know the USB hard-drive had tons of small files in it (mainly every NES and SNES rom ever made), so not sure if that triggers the bug to happen or not..

Also, for the record, I always unmount HDs/flash-drives/SD-cards using dolphin, and I always wait until it says "you can unmount now", and actually before I do that, I usually open a console and type "sync" and wait until that is done before I unmount, because I am paranoid about the linux buffer saying files are copied already, when actually they are being copied still in the background.. I especially do this for important stuff..

Any way, I'm not sure if there is any information or tests or logs I can submit that would help, but if there is, I would be happy to do it.. I use debian stable, so I am stuck with this KDE version for a long time..
Comment 65 Nathan 2015-08-23 20:01:50 UTC
I have a stereo that takes a usb thumb drive with mp3s or whatever loaded on it. I just installed ubuntu and when i transferred some files to it everything appeared to go well. When I put it in the stereo, tracks would play but either get cut off randomly or even sometimes play one song for a few seconds, a part of another song, back to the first and so on. I thought my files were corrupt, but it plays fine on the computer. After reading this I wondered if I had made the right choice installing ubuntu. I tried several things, none of which worked, but I did learn something that might be helpful. I dropped the mp3 files into the root directory and they play fine now. if they are in folders, despite the name, they always mess up. Hope this helps someone fix this. It's really kinda annoying
Comment 66 Austin S. Hemmelgarn 2015-08-24 13:04:21 UTC
Reproduced using the following steps on Gentoo:
1. Created temporary LVM volumes for testing (test1 and test2) in a thin-provisioning pool.
2. Formatted both with BTRFS.
3. Mounted at /mnt/test1 and /mnt/test2
4. Created large number (>65536) of files and folders in semi-random structure on /mnt/test1 with very long path-name components (>200 bytes per level), and filled files to random sizes with zero bytes.
5. Using cgroups, artificially limited disk bandwidth for the current shell
6. Using kioclient, attempted to copy files recursively form /mnt/test1 to /mnt/test2.
7. Flushed all buffers to disk and unmounted /mnt/test2 (double sync followed by umount from commandline).
8. Remounted /mnt/test2 and used diff  to determine differences between /mnt/test1 and /mnt/test2.
9. Repeated steps 6-8 multiple times, each time the results from diff were different.
10. Repeated steps 6-8 using rsync instead of kioclient, and diff reported no differences.
11. Repeated steps 6-8 without the disk bandwidth limited using cgroups, and diff reported no differences.
12. Repeated steps 6-8 with shorter filenames and the disk bandwidth limited, and still reproduced the issue.

Above results lead me to believe (without looking directly at the code) that kio is timing out  trying to gather directory contents and/or calling stat on files.
Comment 67 Austin S. Hemmelgarn 2015-08-24 13:04:59 UTC
Reproduced using the following steps on Gentoo:
1. Created temporary LVM volumes for testing (test1 and test2) in a thin-provisioning pool.
2. Formatted both with BTRFS.
3. Mounted at /mnt/test1 and /mnt/test2
4. Created large number (>65536) of files and folders in semi-random structure on /mnt/test1 with very long path-name components (>200 bytes per level), and filled files to random sizes with zero bytes.
5. Using cgroups, artificially limited disk bandwidth for the current shell
6. Using kioclient, attempted to copy files recursively form /mnt/test1 to /mnt/test2.
7. Flushed all buffers to disk and unmounted /mnt/test2 (double sync followed by umount from commandline).
8. Remounted /mnt/test2 and used diff  to determine differences between /mnt/test1 and /mnt/test2.
9. Repeated steps 6-8 multiple times, each time the results from diff were different.
10. Repeated steps 6-8 using rsync instead of kioclient, and diff reported no differences.
11. Repeated steps 6-8 without the disk bandwidth limited using cgroups, and diff reported no differences.
12. Repeated steps 6-8 with shorter filenames and the disk bandwidth limited, and still reproduced the issue.

Above results lead me to believe (without looking directly at the code) that kio is timing out  trying to gather directory contents and/or calling stat on files.
Comment 68 Austin S. Hemmelgarn 2015-08-24 13:06:23 UTC
Apologies about the double comment.
Comment 69 Alberto J. Ruiz 2015-08-26 18:28:34 UTC
I'm an user that has slso been impacted by this bug. 

Can you post output of kioclient with all debug options checked?

That is, open kdebugdialog, search kio in that window and check everything and then running kioclient.

According to some comments here ( https://www.reddit.com/r/kde/comments/1ak1lz/psa_dolphin_exhibits_data_loss_do_not_use_it_for/ ) outputing that info prevents the timing out in some way.

It might be useful to redirect the output like this:

kioclient copy src dst > errorlog.txt 2>&1

Also, thanks for your great effort. 

By the way, this bug should be marked as CRITICAL.
Comment 70 Austin S. Hemmelgarn 2015-08-26 19:00:15 UTC
Yeah, but it might be a while.  As of right now I don't have most of KDE installed (out of the entirety of KDE suite, all I use personally is Okular, Calligra, and a couple of the games).  In fact I specifically installed kioclient for the sole purpose of testing this (I'm trying to help get rid of long standing bugs like this in FOSS projects whenever I stumble across them).  As I'm running Gentoo, this in turn means compiling everything, which based on past experience with KDE will take a while.  I will put it on the top of my list for tomorrow however.
Comment 71 Alberto J. Ruiz 2015-08-31 10:36:57 UTC
 (In reply to Austin S. Hemmelgarn from comment #67)

I can't reproduce the bug following your steps:


Kubuntu 14.04.2 
Linux alberto-VirtualBox 3.16.0-46-generic #62~14.04.1-Ubuntu SMP Tue Aug 11 16:28:19 UTC 2015 i686 i686 i686 GNU/Linux

Enter root and then

1.
vgcreate -s 64M vg_thin /dev/XXXX

lvcreate -L 3G --thinpool tp_tecmint_pool vg_thin

lvcreate -V 1G --thin -n thin_vol_client1 vg_thin/tp_tecmint_pool
lvcreate -V 1G --thin -n thin_vol_client2 vg_thin/tp_tecmint_pool

2.
mkfs.btrfs /dev/vg_thin/thin_vol_client1
mkfs.btrfs /dev/vg_thin/thin_vol_client2


3.

mkdir /mnt/test1 
mkdir /mnt/test2

mount /dev/vg_thin/thin_vol_client1 /mnt/test1 
mount /dev/vg_thin/thin_vol_client2 /mnt/test2

4.
rm -rf /mnt/test1/*
./create_random_files.py /mnt/test1


##BEGINNING OF SCRIPT (Python3)
#!/usr/bin/env python3 

import os
import random
import string
import sys

min_depth=3
max_depth=6

elem_by_folder_min=5
elem_by_folder_max=8

size_max_files_kb=50

letters_min=210
letters_max=220


files_and_folders_created=0

def rand_lett():
    m=random.randint(letters_min,letters_max)
    return ''.join((random.choice(string.ascii_letters) for _ in range(m)))
        
    
def create_file(name):
    open(name,'wb').write(os.urandom(random.randint(1,size_max_files_kb*1024)))
      
        

def create_folder(folder_name,depth):
    global files_and_folders_created
    
    if depth != 0:
      os.mkdir(folder_name)
    
    os.chdir(folder_name)
    for _ in range(random.randint(elem_by_folder_min,elem_by_folder_max)):
        if depth < random.randint(min_depth,max_depth):
            create_file(rand_lett())
            create_folder(rand_lett(),depth+1)
            
            files_and_folders_created=files_and_folders_created+2
            

    os.chdir("..")


create_folder(sys.argv[1],0)
print ("Folders and files created:"+str(files_and_folders_created))

##END OF SCRIPT


5.

ls -ls /dev/vg_thin/*
Those are symlinks, following them:
ls -l /dev/dm-*

Major and minor numbers are:
252:4 y 252:5

grep blkio /proc/mounts || mkdir -p /cgroup/blkio ; mount -t cgroup -o blkio none /cgroup/blkio
cgcreate -g blkio:/iothrottle

echo "252:4 5000000" > /cgroup/blkio/iothrottle/blkio.throttle.read_bps_device
echo "252:4 5000000" > /cgroup/blkio/iothrottle/blkio.throttle.write_bps_device

echo "252:5 5000000" >> /cgroup/blkio/iothrottle/blkio.throttle.read_bps_device
echo "252:5 5000000" >> /cgroup/blkio/iothrottle/blkio.throttle.write_bps_device

5.1. Put every task in iothrottle group
ps axu | awk '{print $2}' | tail -n +2  | python3 -c 'import sys;a=sys.stdin.readlines(); b=[open("/cgroup/blkio/iothrottle/tasks","a").write(val) for val in a]'


6.pre (I've doing all of that in root account)
chown -R XXX /mnt
logout


6.

rm errorlog.txt ; rm -rf /mnt/test2/* ; (kioclient copy /mnt/test1/* /mnt/test2) >&errorlog.txt


7.

sync ; sync ; echo 3 | sudo tee /proc/sys/vm/drop_caches ; echo 3 | sudo tee /proc/sys/vm/drop_caches ; sudo umount /mnt/test2

8.
Look for differences (Which files have been copied and their sizes):

sudo mount /dev/vg_thin/thin_vol_client2 /mnt/test2 && diff <(find /mnt/test1/ -printf '%P%s\n' | sort)    <(find /mnt/test2/ -printf '%P%s\n' | sort)

Am I missing something?
Comment 72 Austin S. Hemmelgarn 2015-08-31 12:14:48 UTC
Interesting, I can't seem to reproduce it now either.
Guess I'm off to run hardware tests on my system.
Comment 73 Nathan 2015-09-05 20:46:32 UTC
Would it help any if i used a different linux distribution? I am new to linux and installed ubuntu
Comment 74 Cacciatore 2015-09-23 00:07:21 UTC
There s a lot of people out there having the exact same bug all these years and it's really a pity that no action has been taken since the first reports came out. 
I recently fill a bug report https://bugs.kde.org/show_bug.cgi?id=352761 
 I'm really disappointed. Data loss is the most critical issue and after all these years we are still afraid to use a simple command such as copy. Are we being pushed in Windows?
Comment 75 Jaime Torres 2015-09-23 07:00:53 UTC
Hello,

  As you can see, if you read all the comments, this is a hard to reproduce bug, even for the developers. For example, I've tried to reproduce it several times, and I've seen it only once, and after doing the same steps, it didn't reproduce again.
It is almost impossible to fix a bug if you can not reproduce it because you have no way to know if it is fixed or not.

  Has anyone that was able to reproduce this bug with KDE SC 4 tried to reproduce it with the newest KF5 dolphin? (Just to know if it could still be there or not)
Comment 76 Cacciatore 2015-09-23 11:25:20 UTC
Hello Jaime, thank you for your answer. 

As i state in the aformentioned bug report (https://bugs.kde.org/show_bug.cgi?id=352761) this problem still exists in these versions
kdelibs version is 4.14.11
dolphin 15.0.8.0
plasma-framework 5.13.0

I agree that you can't fix something that isn't reproduced by you (the developers) and everyone clearly understand that. But, (and this is a big emphazised ''but''), with a simple google search you can see thousands of posts, all over the net, in various hardware setups, in various distros, and all of them have experienced data loss when copy or moving things around. Some of them can clearly provide more technical info that should give you insights on whats happening. 

We are willing to help you, thats why we raise all those bug reports, to contribute, from our side, to the community, to bring more stable and compete (and complete)  de. We can realize that its a difficult task, to hunt down those bugs, but the completion should be rewarding for all the community. 
Personally i lack the developer skills, though i'm willing to help, to provide the info you may want.
Comment 77 wynajem_pn 2015-10-02 10:52:51 UTC
I've just copied 50 old dvd's with some backup files to an external usb drive (using Krusader) and at the end I noticed that there are lots of missing files on the usb drive. When I checked this out it turned out that not a single DVD was completely copied to the usb drive! To be honest I was shocked that modern version of OS may have such a bug. Moreover, from my point of view it is trivial to reproduce the bug (as I said not a single dvd was properly copied), hence I cannot believe that developers didn't noticed it. 
My setup:
Ubuntu 14.04.3 LTS (with all updated installed)
krusader: 1:2.4.0~beta3-2
kdelibs-bin: 4:4.13.3-0ubuntu0.2
Now I am a bit scared how many times I didn't noticed that the 'copy' command in krusader is not reliable.
Comment 78 Jaime Torres 2015-10-02 11:25:16 UTC
Based on Alberto J. Ruiz script, I've made a similar script with infinite repetition (until the copies are not equal to the original). It has been running all week copying to an usb pendrive (I do not have an external usb hard disk) and also copying to another sata harddisk. It should have copied more than 10.000 files, and it has never reproduced this bug so far (using the latest KF5 version compiled from sources last weekend).
But we will keep trying...
Comment 79 wynajem_pn 2015-10-02 11:54:59 UTC
I made a simple screen-cast to show you the bug in action: http://sendvid.com/dc92o1yu As I said, I have no problem with reproducing the bug, so feel free to ask me for help in debugging an issue. Some additional info: It seems that the destination of 'copy' doesn't matter - it is exactly the same with usb and sata drives (both ext4). The source is ISO 9660. Each time I press 'copy' on the folder the command works better - it finds more files to copy. After 2 or 3 repeats it does recognize all files. After unmounting/mounting everything starts from the begging.
Comment 80 Alberto J. Ruiz 2015-10-02 11:59:49 UTC
(In reply to wynajem_pn from comment #79)
> I made a simple screen-cast to show you the bug in action:
> http://sendvid.com/dc92o1yu As I said, I have no problem with reproducing
> the bug, so feel free to ask me for help in debugging an issue. Some
> additional info: It seems that the destination of 'copy' doesn't matter - it
> is exactly the same with usb and sata drives (both ext4). The source is ISO
> 9660. Each time I press 'copy' on the folder the command works better - it
> finds more files to copy. After 2 or 3 repeats it does recognize all files.
> After unmounting/mounting everything starts from the begging.

All right. First, can you reproduce the bug using kioclient?    "kioclient copy source destination".

If so, open kdebugdialog. Tick Everyoption. Use kioclient to reproduce again. If you can reproduce it with debug flags, paste the output of kioclient.

Thank you
Comment 81 wynajem_pn 2015-10-02 12:12:18 UTC
Created attachment 94809 [details]
output of kioclient copy

It is exactly the same with kioclient. Log file attached.
Comment 82 Alberto J. Ruiz 2015-10-02 12:20:55 UTC
Something looks fishy :

kioclient(12516)/kio (KIOJob) KIO::SimpleJob::~SimpleJob: Killing job KIO::SimpleJob(0x17fe690) in destructor! "[
0: /usr/lib/libkdecore.so.5(kRealBacktrace(int)+0x3b) [0x7fb19cbe476b]
1: /usr/lib/libkio.so.5(KIO::SimpleJob::~SimpleJob()+0x86) [0x7fb19e393306]
2: /usr/lib/libkio.so.5(KIO::ListJob::~ListJob()+0x9) [0x7fb19e393609]
3: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QObject::event(QEvent*)+0x288) [0x7fb19c51fc58]
4: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(QApplicationPrivate::notify_helper(QObject*, QEvent*)+0x8c) [0x7fb19d174e2c]
5: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(QApplication::notify(QObject*, QEvent*)+0x270) [0x7fb19d17b4a0]
6: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QCoreApplication::notifyInternal(QObject*, QEvent*)+0x6d) [0x7fb19c5074dd]
7: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*)+0x1ed) [0x7fb19c50ab3d]
8: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(+0x1aaf83) [0x7fb19c534f83]
9: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_dispatch+0x254) [0x7fb198bd1e04]
10: /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x49048) [0x7fb198bd2048]
11: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_iteration+0x2c) [0x7fb198bd20ec]
12: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)+0x71) [0x7fb19c5347a1]
13: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(+0x26bbe6) [0x7fb19d216be6]
14: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)+0x2f) [0x7fb19c5060af]
15: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)+0x175) [0x7fb19c5063a5]
16: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QCoreApplication::exec()+0x89) [0x7fb19c50bb79]
17: kioclient() [0x4046d8]
18: kioclient() [0x405518]
19: kioclient() [0x403a58]
20: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fb19bce2ec5]
21: kioclient() [0x403fa8]
]
"

Looks like a job is destroyed before finishing.



I almost forgot, sometimes dolphin enumerates files incorrectly in the properties dialog (See bug 333436). 
Have you lost files during this kioclient copy? I mean, you navigate through folders and there aren't files that are supposed to be there. If that's the case, you have successfully reproduced this bug after 7 years and we can start working on it.
Comment 83 wynajem_pn 2015-10-02 12:24:41 UTC
In fact during the logged run of 'kioclient copy' it copied 170MB out of 3,6GB and there are more than 300 missing files.
Comment 84 Cacciatore 2015-10-03 10:30:50 UTC
At last some progress. This is very promising and we hope you can find a solution. 
A little hint
This bug occurs even if you try to copy from an external disk (either ntfs or ext4) to your internal disk. [the source is external-hdd and destination is internal sata hdd]
It doesn't matter if dolphin enumerates files incorrectly in properties, the actual problem is that it skips a tremendous amount of files to be copied.
Comment 85 Alberto J. Ruiz 2015-10-04 09:15:29 UTC
I'm not a KDE developer, only a rookie C++ developer.

In source code kdelibs-4.13.3/kio/kio/jobs.cpp :

void ListJob::slotResult( KJob * job )
{
    if (job->error()) {
	// If we can't list a subdir, the result is still ok
	// This is why we override KCompositeJob::slotResult - to not set
	// an error on parent job.
	// Let's emit a signal about this though
	emit subError(this, static_cast<KIO::ListJob*>(job));
    }
    removeSubjob(job);
    if (!hasSubjobs())
        emitResult();
}

Couldn't this be masking other errors not accounted? 
We should check for  ERR_ACCESS_DENIED and if it's not, mark the error the same way KCompositeJob::slotResult does. 
This is just a theory. The bug might be a completely unrelated.
Comment 86 Alberto J. Ruiz 2015-10-11 14:29:07 UTC
Created attachment 94949 [details]
Possible patch for bug 162211

The faulty part of the copying process seems to be in ListJob. 

1. ListJob fires the signal subError in case of error but it doesn't get upwards. (Such as cannot enter subfolder).
2. The fact that the ListJob must work even if some subfolders cannot be entered. ListJob::slotResult ommits all errors.
3. When such error happens, slotResults removes the subjob when it shouldn't. It should return immediately like this function.

void FileCopyJob::slotResult( KJob *job)
{
  .....
   if ( job->error() )
   {
     .....
      emitResult();
      return;
   }

....
}

I have reproduced the backtrace with "kioclient copy ~/src ~/dst"
~/src contains the following:
src:
total 4
drwxrwxr-x 3 alberto alberto 4096 oct  7 18:45 c

src/c:
total 548872
-rw-rw-r-- 1 alberto alberto 562036736 oct  7 14:01 a
drwx------ 2 root    alberto      4096 oct  7 18:44 b

src/c/b:
total 1024
-rw-rw-r-- 1 alberto alberto 1048576 oct  7 18:44 aleatorio

The "a" file belongs to root and chmod'ed to 700 to emulate the "Cannot access folder". The file is made of randomness.

After applying those changes, the backtrace has disappeared and kioclient does not fail silently.

Tested on Kubuntu 14.04 . Patch for KdeLibs 4.14.12.
Comment 87 Alberto J. Ruiz 2015-10-11 16:26:30 UTC
Created attachment 94951 [details]
Proposed patch

Actually, this is my proposed patch, not the otherone.
Comment 88 tromzy 2015-11-04 08:58:59 UTC
I'm affected by this bug too (well, not me directly, but my girlfriend, and as a teacher it's very problematic for her). When we try to copy a folder that contains several files and folders from her /home/Documents to a USB drive (no matter what filesystem the drive is), KDE / Dolphin will copy the folders only. Every folder will be there on the USB drive, but no file at all. We noticed though, that sometimes if we copy the content of the upper folder (but not the folder itself, just the content), it works. 

She's running up-to-date Arch Linux (Plasma 5.4.2, latest frameworks). I installed Thunar on her system and copying with Thunar works. So the problem lies somewhere in KDE...
Comment 89 Alex 2016-01-09 00:07:15 UTC
I lost the files when copying from hdd to hdd. Without using the USB drives.
Today checked on 2 PCs Archlinux in Konqueror, Krusader, Dolphin bug not reproduced.
Previously, he consistently played. Now it is fixed.
Comment 90 Christoph Feck 2016-03-10 19:05:51 UTC
*** Bug 357875 has been marked as a duplicate of this bug. ***
Comment 91 Me 2016-10-29 04:07:14 UTC
I just lost a bunch of precious pictures due to a bug in Dolphin.

I was doing maintenance on my photo collection. I had previously downloaded some jpg and raw files directly from the camera onto USB drive "A". I keep my photos organized on USB drive "B". I thus attempted to move the photos from USB drive A to USB drive B with Dolphin. Both USB A and USB B are multi GB SATA 3.5" hard drives with USB 3.0 interfaces. They are formatted EXT4.

As near as I can remember, here is what happened:

1) I opened Dolphin. I split the screen. I selected USB drive A on the left panel. I selected USB drive B on the right panel.

2) I did a bit of browsing on USB A to ascertain which folders I wanted to move. I created a new folder on USB drive B.

3) I selected about 10 folders containing several hundred images from drive A. I dragged them over to the drive B panel in Dolphin and dropped them on the folder I created. I selected Move from the pop up menu. Not copy.

4) The process started. I opened the KDE plasma notification window and expanded the info to watch the files being copied. The process took about 10 minutes. The CPU load was very high, like 100%, on the plasma shell process.

5) At the conclusion of the process I just happened to open up a jpg to have a look at it. I started scrolling through the files only to find that several sub folders had images with file sizes of 0B. They could not be opened. The file content was lost even if the handle was there.

6) When I went back to the source folders on drive A, the images are, of course, gone.

The hard drives have both been fine. I ran fsck on them and both came up clean.

At no point in this process were the drives unmounted or disconnected.  I got called away from my desk and rechecked the files again later.   The file sizes were still zero.

I've seen files lost like this using Dolphin before. I suspect there is a bug in the kio process when transferring large blocks of files to an external USB drive. 

What is interesting is that if you use mv or cp on the command line, the CPU load is way, way lower than if the files are moved from Dolphin. Like 5% versus 100% and the command line operations are quite a bit faster.

$ uname -a
Linux localhost.localdomain 4.7.9-200.fc24.x86_64 #1 SMP Thu Oct 20 14:26:16 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

kf version 5.26.0.1.fc24

Been using Linux exclusively on my workstation since Redhat 8.  I highly doubt this is user error.

Let me know if you need more information.
Comment 92 jeremy9856 2018-03-22 13:12:34 UTC
Is it fixed now as stated in this comment ?

https://www.reddit.com/r/linux/comments/8652zc/umm_gnome_shell_has_a_rather_big_memory_leak/dw2kgzy/
Comment 93 Nate Graham 2018-05-03 21:41:28 UTC
*** Bug 336624 has been marked as a duplicate of this bug. ***
Comment 94 Patrick Silva 2018-05-07 16:08:56 UTC
I think that I have been just affected by the problem reported here.
I extracted a zip file containing 104715 files and 1678 subfolders (900 MiB)
from a NTFS partition (internal hard disk) to my home (EXT4) using Ark.
I followed the steps below:

open dolphin
right click zip file, select "Extract > Extract archive to..."
browse to some place in another partition and create a new folder
click "extract" button to start extraction
when extraction is completed, open destination folder,
select all files/folders just extracted and press alt+enter to open properties window
observe amount of files/sub-folders
open zip file using Ark, press alt+enter to open zip file properties,
observe "number of entries" info
Result: amount of files/sub-folders shown by Dolphin and "number of entries"
shown by Ark do NOT match

It happned twice. Each time the zip file was extracted, Dolphin shown a different amount
of files/sub-folders in properties dialog. Now I can not reproduce anymore.

My system:
Arch Linux
plasma 5.12.5
frameworks 5.45
Dolphin 18.04
Comment 95 Nate Graham 2018-05-08 17:35:08 UTC
*** Bug 335924 has been marked as a duplicate of this bug. ***
Comment 96 jeremy9856 2018-05-12 14:17:30 UTC
I'm worried to use Kubuntu now. I wonder if there is really a chance to lose data on KDE. If it's the case I prefer not using it. So what's the state of this data loss bug ? Thank you.
Comment 97 Mahendra Tallur 2018-05-13 19:05:41 UTC
Hi Dr Chapatin,

can you double check the actual number of files with another app, from the terminal for instance ? Did you also check the total size ? It rings me a bell but I don't remember what I concluded...

(In reply to Dr. Chapatin from comment #94)
> I think that I have been just affected by the problem reported here.
> I extracted a zip file containing 104715 files and 1678 subfolders (900 MiB)
> from a NTFS partition (internal hard disk) to my home (EXT4) using Ark.
> I followed the steps below:
> 
> open dolphin
> right click zip file, select "Extract > Extract archive to..."
> browse to some place in another partition and create a new folder
> click "extract" button to start extraction
> when extraction is completed, open destination folder,
> select all files/folders just extracted and press alt+enter to open
> properties window
> observe amount of files/sub-folders
> open zip file using Ark, press alt+enter to open zip file properties,
> observe "number of entries" info
> Result: amount of files/sub-folders shown by Dolphin and "number of entries"
> shown by Ark do NOT match
> 
> It happned twice. Each time the zip file was extracted, Dolphin shown a
> different amount
> of files/sub-folders in properties dialog. Now I can not reproduce anymore.
> 
> My system:
> Arch Linux
> plasma 5.12.5
> frameworks 5.45
> Dolphin 18.04
Comment 98 Kanedias 2018-08-25 16:23:00 UTC
Was proposed patch in comment 87 applied?
Comment 99 Øystein Steffensen-Alværvik 2018-08-25 19:43:17 UTC
No. I don't think any of the developers are able to reproduce this issue.
Comment 100 FiNeX 2018-08-25 20:56:27 UTC
On Reddit another user has been hitted by this bug: https://www.reddit.com/r/linux/comments/9a7k2i/psa_on_kde_never_use_the_gui_when_copying_a_large
Comment 101 Julian Steinmann 2018-08-26 06:57:22 UTC
Can anybody reproduce this issue reliably? If yes, it'd be helpful to have a kioclient log like the one in comment 81. As mentioned in the Reddit thread, maybe an e2image which contains the file metadata could help us as well to further investigate this bug.
Comment 102 Øystein Steffensen-Alværvik 2018-08-26 08:01:42 UTC
(In reply to FiNeX from comment #100)
> On Reddit another user has been hitted by this bug:
> https://www.reddit.com/r/linux/comments/9a7k2i/
> psa_on_kde_never_use_the_gui_when_copying_a_large

I'm guessing here, but since the poster only lost files 4 levels down in the directory tree, could the bug be triggered by especially long character filenames?
Comment 103 Patrick Silva 2018-09-25 13:02:56 UTC
*** Bug 262768 has been marked as a duplicate of this bug. ***
Comment 104 Andrew Crouthamel 2018-10-21 05:02:59 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 105 Julian Steinmann 2018-10-21 12:33:45 UTC
I'll set this back to REPORTED so that this bug will not get closed for NEEDSINFO inactivity. If you can reproduce this issue reliably (preferably in a small, contained environment) we would of course be interested to investigate this bug further.
Comment 106 Filip Fila 2018-10-21 12:46:36 UTC
(In reply to Julian Schraner from comment #105)
> I'll set this back to REPORTED so that this bug will not get closed for
> NEEDSINFO inactivity. If you can reproduce this issue reliably (preferably
> in a small, contained environment) we would of course be interested to
> investigate this bug further.

That you, Julian, I was actually just about to submit a post pleading to not do this, but then I saw your comment. 

I would also use this opportunity to perhaps point out that, from my, understanding, the video proof (https://bugs.kde.org/show_bug.cgi?id=162211#c79) refutes a claim a KDE dev made recently that this bug is primarily related to not doing safe remove.

BTW, what is the consensus on Alberto's analysis and fix? Does it look like it addresses the issue, just that we would need to actually test it out in a situation where the bug is reproduced?
Comment 107 bluescreenavenger 2018-10-27 00:50:46 UTC
(In reply to wynajem_pn from comment #79)
> I made a simple screen-cast to show you the bug in action:
> http://sendvid.com/dc92o1yu As I said, I have no problem with reproducing
> the bug, so feel free to ask me for help in debugging an issue. Some
> additional info: It seems that the destination of 'copy' doesn't matter - it
> is exactly the same with usb and sata drives (both ext4). The source is ISO
> 9660. Each time I press 'copy' on the folder the command works better - it
> finds more files to copy. After 2 or 3 repeats it does recognize all files.
> After unmounting/mounting everything starts from the begging.

Are you able to maybe try this?
run 


tree /path/to/source > ~/origfiles.txt
export QT_LOGGING_RULES=kf5.kio.*=true
kioclient5 copy /path/to/source /path/to/destination &> ~/copylog.txt
tree /path/to/destination > ~/copiedfiles.txt
diff -rq /path/to/source /path/to/destination > ~/copydiff.txt

and then attach origfiles.txt copylog.txt copiedfiles.txt copydiff.txt to the report?
that might help the devs, provided that the issue replicates with using kioclient to copy the files
Comment 108 Filip Fila 2018-11-06 20:22:05 UTC
Created attachment 116131 [details]
More files than before

When trying to reproduce this bug Dolphin tells me I get more files than I originally had.

The screenshots depicts what happens when I copy the WinSxS folder from the Windows NTFS partition to a Linux home ext4 partition.

Both the number of files as well as the size on the disk increases. No idea is this is relevant at all, but it may point to some numeration problems that still do exist.
Comment 109 bluescreenavenger 2018-11-07 03:20:59 UTC
(In reply to Filip F. from comment #108)
> Created attachment 116131 [details]
> More files than before
> 
> When trying to reproduce this bug Dolphin tells me I get more files than I
> originally had.
> 
> The screenshots depicts what happens when I copy the WinSxS folder from the
> Windows NTFS partition to a Linux home ext4 partition.
> 
> Both the number of files as well as the size on the disk increases. No idea
> is this is relevant at all, but it may point to some numeration problems
> that still do exist.

I know WinSxS is full of hard links, and NTFS junctions....
Maybe run 
cd  /run/media/filip/System/Windows/WinSxS
find  > ~/TargetList.txt
cd /home/filip/Testing/WinSxs
find > ~/DestinationList.txt

and attach the two files that will then be in your $HOME?
Comment 110 Filip Fila 2018-11-07 14:53:24 UTC
(In reply to bluescreenavenger from comment #109)
> (In reply to Filip F. from comment #108)
> > Created attachment 116131 [details]
> > More files than before
> > 
> > When trying to reproduce this bug Dolphin tells me I get more files than I
> > originally had.
> > 
> > The screenshots depicts what happens when I copy the WinSxS folder from the
> > Windows NTFS partition to a Linux home ext4 partition.
> > 
> > Both the number of files as well as the size on the disk increases. No idea
> > is this is relevant at all, but it may point to some numeration problems
> > that still do exist.
> 
> I know WinSxS is full of hard links, and NTFS junctions....
> Maybe run 
> cd  /run/media/filip/System/Windows/WinSxS
> find  > ~/TargetList.txt
> cd /home/filip/Testing/WinSxs
> find > ~/DestinationList.txt
> 
> and attach the two files that will then be in your $HOME?

It's not letting me attach the .txt files because both are 15 MiB each, but the number of lines is identical in both of them so what you said must be true. Thanks for the clarification, will keep on trying to reproduce this bug here and there.
Comment 111 realf 2018-11-07 14:55:54 UTC
(In reply to Filip F. from comment #110)
> (In reply to bluescreenavenger from comment #109)
> > (In reply to Filip F. from comment #108)
> > > Created attachment 116131 [details]
> > > More files than before
> > > 
> > > When trying to reproduce this bug Dolphin tells me I get more files than I
> > > originally had.
> > > 
> > > The screenshots depicts what happens when I copy the WinSxS folder from the
> > > Windows NTFS partition to a Linux home ext4 partition.
> > > 
> > > Both the number of files as well as the size on the disk increases. No idea
> > > is this is relevant at all, but it may point to some numeration problems
> > > that still do exist.
> > 
> > I know WinSxS is full of hard links, and NTFS junctions....
> > Maybe run 
> > cd  /run/media/filip/System/Windows/WinSxS
> > find  > ~/TargetList.txt
> > cd /home/filip/Testing/WinSxs
> > find > ~/DestinationList.txt
> > 
> > and attach the two files that will then be in your $HOME?
> 
> It's not letting me attach the .txt files because both are 15 MiB each, but
> the number of lines is identical in both of them so what you said must be
> true. Thanks for the clarification, will keep on trying to reproduce this
> bug here and there.

Can you please archive them? Text files can usually be compressed effectively.
Comment 112 Filip Fila 2018-11-07 15:03:13 UTC
Created attachment 116152 [details]
DestinationList & TargetList

Didn't know that, here are the files in a .7z archive
Comment 113 realf 2018-11-07 15:19:11 UTC
(In reply to Filip F. from comment #112)
> Created attachment 116152 [details]
> DestinationList & TargetList
> 
> Didn't know that, here are the files in a .7z archive

The lists seem to be identical.
Comment 114 don.vhs 2018-11-14 12:55:43 UTC
*** This bug has been confirmed by popular vote. ***
Comment 115 Patrick Silva 2018-11-15 14:32:27 UTC
(In reply to Filip F. from comment #108)
> Both the number of files as well as the size on the disk increases. No idea
> is this is relevant at all, but it may point to some numeration problems
> that still do exist.

probably because bug 238424
Comment 116 bluescreenavenger 2018-12-04 01:13:12 UTC
I made a script that makes 1,000,000 randomly placed files, with random names, with various sizes from 1kb to 1gb, into 1,000,000 randomly placed folders, with random names. It selects 3 random other directories to fill them with 100,000 additional files each as well.

Then it uses kioclient5 to copy the items.  

https://raw.githubusercontent.com/n3rdopolis/otherstuff/master/kiocopy


The names have kinds of random numbers, symbols, some are 20 chars long, some have weird symbols that try to break parsing.

It then lets me diff the trees. I
I tried it on my KDE frameworks 5.44 box... 

...but after all that, I was NOT able to replicate it. the trees match. 
I have a log, and a seed file for the script, but the .tar.xz is 83MB, and it's quite worthless because it worked perfectly anyway.

I am at a loss at how to replicate it. maybe I should change the outlier generated maximum file name size to 220 chars?
Comment 117 bluescreenavenger 2018-12-04 01:55:48 UTC
One odd thing I notice during the test, is that the kioslaves have the high CPU usage, kioclient5 has the high memory usage. I assume that means kioclient5 stores the list of files to copy, and then it must make the kioslave do the copying.

My theory was maybe a process (possibly a helper is crashing on the reporters computers, such as kio_file.so)

So I tried a smaller, manual test with Dolphin. Copying a smaller directory of ~240000 files, Dolphin's memory usage grows a bit (not a lot). So that confirms my theory that Dolphin must stay alive to tell the kioslaves what file to copy. Killing the kioslave that the CPU usage spiked with did NOT result in a silent failure. I got a dialog warning that the file protocol died, and it gave me some options to recover which it recovered from fully. So it's not silently failing because of that.

HOWEVER trying again, I went after Dolphin this time. I closed the Dolphin window. The Dolphin process smartly stayed running to continue the copy (so the issue is not the Dolphin window being clsed). As a test, to simulate a crash or abrupt exit of Dolphin, I killed the Dolphin process. Sure enough the copy stopped. ...and I got a "The copy has completed successfully" notification. I wonder if any of the reports are because of Dolphin going down?
Comment 118 David Edmundson 2018-12-04 23:38:20 UTC
*** Bug 352761 has been marked as a duplicate of this bug. ***
Comment 119 David Edmundson 2018-12-04 23:50:50 UTC
Your correct in your analysis of how KIO works.


I did KIO speedups last week that should somewhat reduce the CPU and memory on the file.so slave for many files.

>I wonder if any of the reports are because of Dolphin going down?

Note that was fixed in 2beb1a0ad23177f7dc2e5ee622bed3a70f671278 in plasma-workspace. I CC'd a different bug.

----

The history of this bug is quite messy, most reports date back to before the potentially important e60e8b96b211290e330c7ff6a3b84b20ab85850d on kdelibs handling of inaccessible folders.

This bug was reopened primarily from someone citing copying from a zip which is an entirely different story.
There's not a lot to suggest we still have the same issue as we might have had when that bug was opened.

Then there's also noise from people not unmounting drives and potentially some reports from kuiserver handling of the host app crashing that I just mentioned which I fixed only very recently. 

I've tested copying a million files repeatedly, bluescreenavenger has done the same.
Comment 120 bluescreenavenger 2018-12-06 13:33:24 UTC
Cool. I will also add that ran a second test with my large tree generating script script to check all the files have all the correct sizes. 
At first I saw some differences, but then I realized that I was copying from a test ext4 image to a test btrfs image. The lines with the different sizes in my diff were all *directories* and no files were clipped... I guess not all dirs in BTRFS are 4096 bytes like in EXT4
Comment 121 David Edmundson 2018-12-07 14:42:43 UTC
I like how you've ended up going down the same paths and patterns as I did !

Running stat on a directory is pretty FS independent, it's 0 bytes on NTFS. ls sometimes puts different results too 

I did propose something https://phabricator.kde.org/D17071 but I'm not very convinced.
Comment 122 bluescreenavenger 2018-12-08 00:47:51 UTC
(In reply to David Edmundson from comment #121)
> I like how you've ended up going down the same paths and patterns as I did !
> 
> Running stat on a directory is pretty FS independent, it's 0 bytes on NTFS.
> ls sometimes puts different results too 
> 
> I did propose something https://phabricator.kde.org/D17071 but I'm not very
> convinced.

I used the tree command and the -s flag to show the sizes instead of the Properties dialog however. I saw all kinds of lines that were different on the BTRFS side. Instead of being fixed at 4096, there were lines that where coming up when I diffed the files, at say 0 bytes, 72 bytes, 54 bytes; etc.
I got really concerned that I might have replicated it, but then I realized when I added the -F flag to append a / to directories, that every single line that was different was a directory, and all the files where the same

So as it turns out, it looks like BTRFS also varies from EXT4 on how it measures the size of directories.
Comment 123 wg9rffujwz8y276u 2018-12-09 05:21:37 UTC
(In reply to David Edmundson from comment #119)
> 
> >I wonder if any of the reports are because of Dolphin going down?
> 
> Note that was fixed in 2beb1a0ad23177f7dc2e5ee622bed3a70f671278 in
> plasma-workspace. I CC'd a different bug.

https://bugs.kde.org/show_bug.cgi?id=401055
Comment 124 David Edmundson 2018-12-17 10:18:47 UTC
As per #119 I am closing this report.

There was an issue in kdelibs that was fixed many many years ago. Most comments here relate to that. It was reopened with unrelated reasons.

There was also some mishandling in kuiserver which is fixed quite recently which could give misleading messages which confuses this bug.

No attempts to reproduce a bug with up-to-date code has proved successful despite multiple efforts from multiple people.


That isn't to claim kio is bug free, but it will be much more productive to treat that as a new bug with new information than to add data here which is full of fixed outdated noise.
Comment 125 bluescreenavenger 2018-12-18 02:04:03 UTC
I THINK I JUST REPLICATED IT!!!!!


I should have been MUCH more aggressive with the File Sizes, but I was was worried about off-by-1 errors causing my Script to attempt to create invalid files, and not create enough.

So the script is somewhat incomplete trying to make 255 char files, it doesn't generate as many files as I was before. (as Unicode chars were causing too large names) 

Copying with kioclient, with all the KIO logs on, there is a big line

"Could not make folder /tmp/kiocopy/1545096048/folder_dest/testdir/0l>╚>/1H{n.4🐧N5>2ä<╚#uDJjà57vÉ7BFI*;9jÄ>w@Vdäâ2@Op*8w>i5-by4P{èåhÄÄDw9î1RÆ4x3©h23uÉè)<W?&&,🐧WÄ31Hbv5î,7VTDCrÉï'So🐧D#\"}127z🐧4ij8f®uZ58®pÄJHê®9ÄÅ4Æ-Dj zpN0èg1JB𖡒?Zh{Hom6𖡒B9i1\"èEms2.1.5¾ch@ÅWÆ<äU})2CW1h\t𖡒V9g�."

at the end, and then kioclient5 exits with the status of 1!

Diffing the trees the final lines:
-17 directories, 413 files
+7 directories, 0 files


I will attach the report that the script made, containing the tree listings, and containing the log for the kioclient5, and containing the Seed file. Passing the path to the Seed file, causes it to generate the exact same files that it did for me. The script will fail to create SOME files before it runs the actual test (because of the mentioned byte count issue). I will also attach the version of the script that replicated the issue

Let me know if I need to provide any info.
This was kio 5.44 admittedly, ...but I also get similar results from a version from Mid October 2018, built from Git.

But it looks like it fails to create that folder, and quits.
Comment 126 bluescreenavenger 2018-12-18 02:08:02 UTC
Created attachment 116980 [details]
Archive containing the logs, and the tree, and the Seed.txt file to pass to the kiocopy script
Comment 127 bluescreenavenger 2018-12-18 02:09:16 UTC
Created attachment 116981 [details]
Bash script of which to use to recreate the file tree that failed to copy. Pass the path to the Seed.txt to it.
Comment 128 Christoph Feck 2018-12-18 02:13:59 UTC
The filename limit is 255 bytes, not 255 characters. In UTF-8, any non-ASCII character needs more than 1 byte. Additionally, the last character looks cropped, causing an illegal UTF-8 name, which Qt does not handle.
Comment 129 bluescreenavenger 2018-12-18 02:24:19 UTC
ALSO, for what it's worth, if I leave the two 
LC_ALL=C 
LANG=C
variables set when running kioclient5, instead of setting them back with the script I get... 

-17 directories, 413 files
+8 directories, 221 files
 
Setting it back to 
LANG=en_US.UTF-8 
LC_ALL=
and I get the apparent silent failure


(I set those variables in hopes of working around trying to fit Unicode chars in 255 bytes in the bash/generation portion...)

I REALLY hope this info is helpful
Comment 130 bluescreenavenger 2018-12-18 02:29:46 UTC
(In reply to Christoph Feck from comment #128)
> The filename limit is 255 bytes, not 255 characters. In UTF-8, any non-ASCII
> character needs more than 1 byte. Additionally, the last character looks
> cropped, causing an illegal UTF-8 name, which Qt does not handle.

Yeah, that's the issue I was trying to fix in the script. Trying to find a way to count the BYTES in a variable with bash...

as far as that unreadable char, this is what appears in Hexdump
nerdopolis@nerdopolis:/tmp/kiocopy/1545096048$ echo -n �| hexdump
0000000 bfef 00bd                              
0000003
It could be the fact that the char is lopped, since that is not one of the random ones the script creates... ...but it apparently might have triggered the silent failure ...maybe?
Comment 131 Christoph Feck 2018-12-18 02:42:56 UTC
That character is just the replacement character for a non-UTF-8 sequence when converting to UTF-16 (which is what QString uses internally). You would need to raw bytes before that conversion.
Comment 132 Christoph Feck 2018-12-18 02:44:44 UTC
EDIT: ... need to see the raw bytes ...
Comment 133 bluescreenavenger 2018-12-18 03:26:38 UTC
(In reply to Christoph Feck from comment #132)
> EDIT: ... need to see the raw bytes ...

That should be in my 
"Archive containing the logs, and the tree, and the Seed.txt file to pass to the kiocopy script"
attachment: and then /tmp/kiocopy/1545096048/logs/kioclient.log, 
second to the last line... 
I tried to make sure I had the right byte copied and pasted, when piping into  hexdump
Comment 134 Christoph Feck 2018-12-18 03:55:30 UTC
You copied the replacement character, not the invalid raw byte that caused it. You might want to write the list of filenames from your script to a file, and inspect that with Okteta. It might be easier if you use one file per filename, e.g. FILENAME-000001.txt contains 17 bytes, if the first filename is 16 bytes long etc.
Comment 135 bluescreenavenger 2018-12-18 04:19:27 UTC
I have the full logs, and listings in the mentioned attachment, however the 3 bytes seem to be
EF BF BD
between the valid lower case 'g' and the '.'
Comment 136 David Edmundson 2018-12-18 12:01:11 UTC
Please make a new bug for a specific issue that we're trying to track.
Comment 137 David Edmundson 2018-12-18 12:59:21 UTC
LANG definitely can make a difference to the way things run. 

Filesystems don't have a concept of UTF-8 or anything else just a concept of a raw bytestream. We then put that into a QUrl which obviously follows the locale it runs in. Though the worst this should do is (correctly) fail on a file conflict where one didn't previously exist.

I think your script is creating a byte array that isn't valid text in any format. I can understand KIO failing to copy that because we don't treat things as a byte stream all the way through.

However in your logs KIO isn't silent.

Your kioclient log ends with:

"Could not make folder /tmp/kiocopy/1545096048/folder_dest/testdir/0l>╚>/1H{n.4🐧N5>2ä<╚#uDJjà57vÉ7BFI*;9jÄ>w@Vdäâ2@Op*8w>i5-by4P{èåhÄÄDw9î1RÆ4x3©h23uÉè)<W?&&,🐧WÄ31Hbv5î,7VTDCrÉï'So🐧D#\"}127z🐧4ij8f®uZ58®pÄJHê®9ÄÅ4Æ-Dj zpN0èg1JB𖡒?Zh{Hom6𖡒B9i1\"èEms2.1.5¾ch@ÅWÆ<äU})2CW1h\t𖡒V9g�."
1

that 1 being the error code 

So kio did catch the issue and raised an issue. If that was in dolphin etc. that would be an error prompt.
Comment 138 bluescreenavenger 2018-12-18 13:17:18 UTC
I opened https://bugs.kde.org/show_bug.cgi?id=402298
If I set LANG differently, I DO get error prompts though. and I get more files copied. I'm not running kioclient5 headless, but in a case where it's running under a display, and I CAN get other dialogs... such as, when I forgot to set LANG back, I got actual messages telling me that it can't access those files...