Version: 1.0.3 (using KDE 4.0.4) Installed from: Ubuntu Packages OS: Linux ok this is a huge problem for me. i have an external usb 500gb hard drive. it's a toshiba and appears to work fine, mounts and shows up in the recently plugged in thing and in dolphin etc. there is a serious issue here though, when i back up my files by selecting my home dir and copying and pasting to the usb drive it copies stuff over fine. or at least appears to. then when i go to the usb drive and click through folders and contents loads of stuff is missing and has not copied at all. this is real bad cos you're not aware that it's happened so you think you've backed up when in fact you've lost a load of stuff. it's the same if i just copy and paste folders within my home also, such as Documents folder. it won't copy all the documents across even though i get no warning or anything to say it can't, it just pretends it's worked. this really needs fixing, anyway way i can help debug or help with this let me know. thanks.
If you copy large amount of data from your internal HD to itself? Did you have the same problem? What is the FS on the external usb HD? Did you try to copy the same data using shell commands? I set this as critical, because if confirmed it need to be fixed with maximum priority!!!!
USB sticks are usually FAT formated. The home folder contains e.g. the emails stored in kmail, which filenames do not go well with FAT formatted disks. Hence, all copied emails are lost. A warning during the copy process would be really nice, this problem prevented me from backing up my files for a very long time.
it's fat32 formatted. copying copying from one partition to another seems ok on same hardrive but unsure as i can't test with loads of data as don't have enough disk space. haven't tried shell commands, not sure of best ones to try using? can you advise on commands and i'll report back. thanks for looking into this...
ok i tested by instead using konqueror to copy the files in the same way i would in dolphin. konqueror worked great and backed up all the data to the usb as it should...
@Jonah: Any chance you can try this again with KDE 4.1? Otherwise, can you give us some examples of the full paths (including names) of files which do not copy. Is it consistently the same files that do not copy if you do it multiple times, or is it different files that don't copy each time you try?
Jonah: Any update on this?
I can verify this problem. I am using kde-4.1.2 and when I try to copy data to my usb flash ( kingstone data traveller 2GB ) which is formated in fat32 i get nothing. The data seems to copy fine but when I plug the flash on another computer the data is missing. The usb flash works fine when I do the same stuff from kde3
As this is now confirmed, any objection against giving it a higher priority?
Markos: Can you provide as much information on how to reproduce this as possible? Can you reproduce this? Does exactly the same thing happen ever time?
Yes this happens every time with every usb storage device formated on fat32. Havent tried ntsf or something else. The way to reproduce it is quite simple ( for me at least ). Just plug a usb device, try to copy a file using Konqueror-4 or dolphin. Then umount the usb device and plug it again. The file is not there ...
Does it unmount cleanly? Can you try to run sync before you detach?
I confirm this problem, i got the same issue, with an EXTERNAL WD 500mb harddisk formatted ext3 filesystem on the usb2.0....I was moving all my music from the internal HD to the EXTERNAL HD, about 200GB the moving process was smooth, no troubles. But when the next day i was seaching some music, i found out that the first few files of a subdirectory were copied well (3..4 files)but the remaining 10 files in that subdirectory had a lenght of 0 (zero)bytes (average 50mb/file) No error messages or whatever :( maybe we need a verify option
sync command before removing the disk seems to works most of the time but NOT always. Still no error messages
OK, I just copied some stuff to a thumb drive and thought I would try and recreate the data loss, if possible (normally, I just use CLI). Here's what I did * Insert thumb drive * Click the little plasma widget which informs me that a USB drive has been connected, this opens Dolphin * Copy stuff * Click the volume on the left * Select safe unmounting I then ran mount on the CLI and lo, behold: /dev/sda1 on /media/disk type vfat (rw,nosuid,nodev,uhelper=hal,uid=1000,codepage=437,iocharset=utf8) Dolphin does NOT complain that it could not unmount the drive! It just fails and is silent about it. A user will, rightly, expect that it has been unmounted cleanly. Unmounting by hand (after moving to a different location within Dolphin as Dolphin held the mount directory open) works just fine. No data loss. Can anyone please confirm/deny that they acted in a similar way when their data loss occured? If you can confirm, we found the issue. If you can't, I will open a new bug.
Markos: When you sync manually, do you wait until the diode on your thumb drive stops blinking? It's usually a good idea to wait for two seconds after you are sure it has stopped, because, sometimes, it has not. Please note that the sync command (same as umount, which calls sync(), as well), may return normally, i.e. without error code when the hardware itself is still busy.
Yeah , indeed im waiting until the led turns off. Unfortunately i cant re-produce it always :(
I suppose copying itself is not the source of these problems. I've noticed that copying to my flash mp3 player is a lot faster in KDE4, but it seems that data is not actually written, but rather cached somewhere. So, when trying to umount the drive through plasma applet or dolphin panel, I have to wait for a 1-3 minutes(!) while the data (~100MB usualy) is actually being written. The device indicates writing process. Dolphin, waiting the umount, usualy gets timeout from dbus daemon and shows a message in the bottom yellow panel. IMO copying/caching routines to such devices should be reviewed.
isn't there a "flush" mount option for removable media to write the stuff directly ?
Yes. Maybe it should be enabled by default..? Problem is how do you determine when you want this or not? Especially with external drives you actually work on, this would make things _really_ slow.
I can confirm this and related problems with my Debian/SID installation (KDE 4.2.4). It seems to be a general problem of KDE4 handling files on a external USB drive. I just lost a lot of pictures from my camera's SD card connected to my desktop PC via an USB reader. When trying to copy some files on my PC's hard drive I got error messages. The resulting files were of correct size but contained no information. Unfortunately, the original files were corrupted somehow during the process...
IMO, that either points to a general problem with the reader/SD card/partition/formatting or KDE would do something _really_ stupid. Did you m5 the files to see if corrupted original and copy are the same?
md5, I mean..
please I wonder what is now the status of this serious bug, because its now a year ago reported, and at that time I still got kubuntu 8.04 with kde 3.5.. now 2 distro's later its still there, I checked this out again on a new laptop & desktop kubuntu 9.04 kde 4.2.. and tried to copy 50gb to my external harddisk , formatted ext3... still zerolength files without any message... You think you backup your data safely
Can you install 4.3, try to reproduce and tell us _exactly_ what you did? Others: is there any sane way he can trace what is happening?
*** Bug 205428 has been marked as a duplicate of this bug. ***
My bug was marked as a dup so here's the info from my own experience. "I have a WD 1 tb external hard drive and I have twice now had almost all the selected files deleted while moving from and to the external hard drive. Both occasions involved more than 50,000 files(both times were photos), and I used the "move to" command. The notification showed up but after a long time with not activity it showed "completed moving x files" but the x number was only a few hundred instead of the full selection. Since I used the "move to" command there was no way to recover any of the files, undo move just moved the couple hundred files back to the previous directory and the thousands of other files disappeared as if they had never been. running OpenSUSE 11.1 x64 with KDE 4.3 Factory" also the 1 tb external is ntfs formatted with a mini usb connection and moving the same files within my internal hard drive works fine.
This bug still seems to exist. I just copied ~60000 files to an external (esata) backup drive, and hundreds of files weren't copied at all. The missing files were completely random (eg one file in a photo/music album). Fortunately, I happened to notice that some files were missing and recopied them all over the top (skipping overwrites). It seems that all the files have transferred now, but this is a very serious bug, and it's lucky I checked before deleting the originals. In case it is relevant, the files were copied by drag and drop in dolphin, and both drives were ntfs and attached via sata.
At one point i'm glad you confirmed my issue this i reported already long long time ago (i guess 2008) Its not related to ntfs, as in my case i used ext3 as format for the external hd When you use your external usb hd for home partition backup, all the time you get useless backups when you check it ....i wonder when there will be some action taken :)..... regards Jazz On Thu, 01 Sep 2011 07:21:59 +0700, Patrick Stewart <patstew@gmail.com> wrote: > https://bugs.kde.org/show_bug.cgi?id=162211 > > > Patrick Stewart <patstew@gmail.com> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |patstew@gmail.com > > > > > --- Comment #27 from Patrick Stewart <patstew gmail com> 2011-09-01 > 00:21:58 --- > This bug still seems to exist. I just copied ~60000 files to an external > (esata) backup drive, and hundreds of files weren't copied at all. The > missing > files were completely random (eg one file in a photo/music album). > Fortunately, > I happened to notice that some files were missing and recopied them all > over > the top (skipping overwrites). It seems that all the files have > transferred > now, but this is a very serious bug, and it's lucky I checked before > deleting > the originals. > In case it is relevant, the files were copied by drag and drop in > dolphin, and > both drives were ntfs and attached via sata. >
There is work in progress... https://git.reviewboard.kde.org/r/102388/
For what it's worth: Mac OS X's Finder and Windows 7 Explorer suffer the same problem. I hope it is fixed soon so we are ahead of those two ;-)
its a long standing bug already, 3...4years, it only happened with usb attached drives if i copy to a network attached storage (NAS) no problems at all and its not only dolphin, also krusader, so i think its a kernel issue On Wed, Oct 12, 2011 at 6:43 AM, BartOtten <bartotten@kde.nl> wrote: > https://bugs.kde.org/show_bug.cgi?id=162211 > > > > > > --- Comment #30 from BartOtten <bartotten kde nl> 2011-10-11 23:43:10 --- > For what it's worth: Mac OS X's Finder and Windows 7 Explorer suffer the > same > problem. I hope it is fixed soon so we are ahead of those two ;-) > > -- > Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are on the CC list for the bug. >
*** Bug 324606 has been marked as a duplicate of this bug. ***
http://fpaste.org/37829/78572315/ This is my output from kdebugdialog when copying another batch of files. Some folders are empty, however the number of empty folders is quite less than I expected.
I can confirm that copying large folder-structure between to external usb disks using Dolphin , after copying finished many files where not copied an it looks arbitrary witch files got copied or not. most folders came over(can not check them all thousands of folders) I am running Mint 14 KDE 4.11
It happened again, this time on a small folder. I was copying a folder with about 10 large .mkv files from an External NTFS drive into a ext4 drive, and it did not copy.
From reading the comments it seems like this is about Plasma/Dolphin/Solid not reporting properly when it cannot unmount leading to people pulling out their USB volumes prematurely, which now works for me (in 4.11, at least, probably earlier as well); I get a warning message if the volume cannot be unmounted.
Martin, this is NOT the case, we talk about usb connected harddisks, no flash sticks,the harddisk is near permanently connected, the harddisk led shows no activity for long long time, the system notifier also shows copy done,its also not the disk cache "not flushed" as it goes about many gigabytes missing much larger then the amount of ram and all programs dolphin,nautilus,krusader......report copy is done.Next time when you logon, many files zero length randomly i think this issue is already since 2008 On Mon, Oct 7, 2013 at 3:48 PM, Martin Sandsmark <martin.sandsmark@kde.org>wrote: > https://bugs.kde.org/show_bug.cgi?id=162211 > > Martin Sandsmark <martin.sandsmark@kde.org> changed: > > What |Removed |Added > > ---------------------------------------------------------------------------- > Status|CONFIRMED |RESOLVED > CC| |martin.sandsmark@kde.org > Resolution|--- |FIXED > > --- Comment #36 from Martin Sandsmark <martin.sandsmark@kde.org> --- > From reading the comments it seems like this is about Plasma/Dolphin/Solid > not > reporting properly when it cannot unmount leading to people pulling out > their > USB volumes prematurely, which now works for me (in 4.11, at least, > probably > earlier as well); I get a warning message if the volume cannot be > unmounted. > > -- > You are receiving this mail because: > You are on the CC list for the bug. >
Having been involved with debugging in 2008(!) and pointing out the umount issue myself, I agree that not all of these comments read that way. In particular, https://bugs.kde.org/show_bug.cgi?id=162211#c34 specifically states that 4.11 is still affected.
(In reply to comment #37) > Martin, this is NOT the case, we talk about usb connected harddisks, no > flash sticks,the harddisk is near permanently connected, the harddisk led > shows no activity for long long time, the system notifier also shows copy > done,its also not the disk cache "not flushed" as it goes about many > gigabytes missing much larger then the amount of ram and all programs > dolphin,nautilus,krusader......report copy is done.Next time when you > logon, many files zero length randomly > i think this issue is already since 2008 Yes, but did you actually unmount it cleanly? (In reply to comment #38) > Having been involved with debugging in 2008(!) and pointing out the umount > issue myself, I agree that not all of these comments read that way. > In particular, https://bugs.kde.org/show_bug.cgi?id=162211#c34 specifically > states that 4.11 is still affected. Still could just be an unclean unmount in that comment. Can someone definitely confirm that they _cleanly unmounted_ the drive in question? Not just "I think I saw the safe to remove" message, but actually checked with for example `mount`? I went through all the code paths that are involved yesterday, and while I may have missed something there (obviously) are no glaring errors in it. I have also tested this myself with my own removable drives and with files of all sizes and weird folder names, and I have been unable to reproduce it.
(In reply to comment #39) Mint 14 KDE 4.11 I will try to explain with more details what I did when this happened. Mounted two 1.3 TB USB2 NTFS drives using Dolphin with the intention of merging the two so I could free up one of them for a different purpose. Selecting all files on one drive i dragging them over to the other drive. The copying takes a long time and when i return the copying process is done. I immediately go looking for a file that is supposed to be copied over and can not find it, looking at other files I recognize that many files are missing, folders seams to have been copied OK. Trying to reproduce this on smaller scales with only copying one folder with files that did not copy the first time ends up with same result. Only way to copy the files that did not get copied is to only copy one file at a time. To me this seams like a problem with selection and iteration over the selection(could be specific to ntfs) I do not currently have access to the hardware that i used, but i will next week, would be more than happy to help investigate.
Could you try copying with the command line utility kioclient (kioclient cp /src/ /target/)? Also, could you share the full paths of the folders that didn't get copied, as well as the target full path?
I don't think it has to do with unmount, as by a normal system shutdown and i not mean by pulling the plug, all drives are unmounted before the system powers off itself. But also by not logout, no screensaver or whatever to interfear with the copy process it still happened As it also seems to not makes any difference what file system you use ntfs, ext2 ext3 ext4,or what application you use..the issue is possibly deeper inside the kernel at the usb part, as copying the same data to a NAS via ethernet 10/100, not wifi, has no problems at all I'm not a linux kernel expert, but i hope to be able to guide in what direction they must search. Its a real serious issue, as more and more people want to backup there photo's ,movies,music and the "simple way" is get an external usb harddisk to save the stuff and forget about it, knowing your data is there safe..... to figure out later(hopefully the next day, and not next year that you remember "oh yeah i had put it on my backup hd" )and that many files are missing Of course after the issue arrived , i never used a usb harddisk as backup anymore just put a new hd in the NAS (only on when i need it) copy the files to the nas,and yes still verify it, it never failed So there is a clear difference in how files are handled over ethernet and usb It is also not a usb harddisk power supply issue as i used a 12v 7ah lead battery to power the usb hd (it had a 12v dc external power supply) and used a shielded 20cm usb cable on a usb2 port I will retry in one of the next weekends to copy by usb and post the results including logs Let me know what is the most usefull best regards On Tue, Oct 8, 2013 at 4:39 AM, Martin Sandsmark <martin.sandsmark@kde.org>wrote: > https://bugs.kde.org/show_bug.cgi?id=162211 > > --- Comment #41 from Martin Sandsmark <martin.sandsmark@kde.org> --- > Could you try copying with the command line utility kioclient (kioclient cp > /src/ /target/)? > > Also, could you share the full paths of the folders that didn't get > copied, as > well as the target full path? > > -- > You are receiving this mail because: > You are on the CC list for the bug. >
Confirm, that this bug still exists: How to reproduce: 1. Insert any removable media 2. Start copy with dolphin any folder with a lot of files. I tried to copy linux sources. 3. After some time i got such error: https://www.200volts.com/uploads/2013/10/21/526538c85ba02_snapshot1.png Additional information: 1. Media free space: Filesystem Size Used Avail Use% Mounted on /dev/sdc1 7.5G 1.2G 6.3G 16% /run/media/feniks/LIVE 2. Media information: feniks@anna /mnt/webos_dev/linux $ sudo fdisk -l /dev/sdc Disk /dev/sdc: 7963 MB, 7963803648 bytes, 15554304 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xc3072e18 Device Boot Start End Blocks Id System /dev/sdc1 * 56 15554303 7777124 c W95 FAT32 (LBA) 3. Kdelibs version: 4.10.5-r1 4. Linux distro: gentoo 5. kernel: 3.10.7-gentoo-r1
copy result seems to be predictable I've got a folder on a NAS with about 2.000 files (10GB). Nearly each time the folder is copied to the internal Notebook SSD the number of copied files is different. kioclient used. If you look at the copy icon in the task bar while a copy task is running you can see how many files in total will be copied. Just click on the plus button below the progress bar. I made the experiance that in error case this value does not match with the total number of files in the source folder. The result is an incomplete target folder. But the target folder will always have exactly the number of files shown in the task bar info while copying - the wrong result is predictable. A second point I recognized is that if you repeat the copy procedure the result will always be different in the number of copied files. It seems that the file structure of the source folder is not read completely by the KDE copy module. The file names are quite long (2013-07-30 15.25.30 Canon EOS 650D.jpg) and the NAS is not that fast (WD MyBook - ARM CPU). This might encourage the occurcance of the error. Using rsync to copy the folder works without errors on same setup. Performance of NAS/network is about 30MB/sec reading for huge files. NAS is mounted as NFS. fstab: 192.168.178.100:/nfs/Public /media/public nfs rw,noauto 0 0. System: Notebook i3/SSD Debian Testing/KDE NAS WDH1NC (ARM CPU) FritzBox/Fixed IP
I am having this issue when copying a few folders from my ext3 home directory into USB flash drive or microSD (both Fat32) with Dolphin. I'm using Kubuntu 12.04 (all software up to date).
6 year old critical bug still exists. Just noticed it cost me most likely the only existing baby photos of my son. Unfortunately, my backups [to three other media] were made with KDE. I'm currently sick over this. A lot of good it did me to have 3 backups, when they were all incomplete. Maybe if the bug isn't going to get fixed a warning should pop up.
Okay, I can't imagine this would be hard to fix, after doing some more investigating. I have been playing with this and had similar results as Andi, plus noticed the following: Every time I copy the Pictures folder to anywhere, not just remote media, it drops more than 1/2 the files with different numbers every time. I've also found that simply right clicking the folder and choosing properties will only give me a partial count, and every time I hit refresh - a different total of files. In total, this folder has 6677 files. I think this has to do with the speed the query for files vs how long the kio waits. And it has everything to do with the source, not the destination. I have two external btrfs formatted HD's which [when checking properties in KDE] show a count of about 3400 at first, but show the right total every time afterwards hitting refresh in properties. After that count shows the actual total of files, then copying them will copy them all. I have one copy on an NTFS partition on my SSD [fuse - slow], which never shows the right totals, no matter how many times I refresh. Anywhere between 2400 and 3000. Another copy [mounted via cifs] on wireless only returns a count of about 1600 in properties. Testing copying to various slow media such as flash drives and cifs mounts works fine as long as the source media is fast enough to return the proper total count.
please don't think light about this issue, its not about the number of files , but many files become zero length without any message about why it failed. it looks like a kind of buffer overrun, to help imagine the issue think about the old rs232 and you connect the rts and cts pins(hardware handshake )(which is no problem IF the receiver is faster then the sender), now rhe receiver is slower then the sender, the receive buffer becomes full and the sender keeps on sending more, resulting in a loss of data Of course with usb its all software implementation, i can't figure out if the usb harddisk not sends a command to stop sending, or that the received stop command is ignored,or delayed , just 1 byte missing from the file and the checksum of the file is wrong ,possibly the harddisk controller puts the file then at 0 length Now the issue is why this goes wrong in linux without any message, in win xp(since 2006 no win pc anymore,so i can't compare it with newer versions) copying stops and you get frustrated why and waste of time , but at least you know it wend wrong,better then thinking everything wend okay but when you need your backup you have nothing or only a part maybe this bug is deeply rooted in the kernel, as i always wondered why if i make an SD card for the raspberry pi with the dd command it mostly result in a not working card(rpi not boot)(even adjusting block sizes and whatever possible), but when using windiskimager never any issue This bug is serious (your most valuable emotional data lost ) and difficult(extreme low level hardware and software knowledge needed, and during the years many guys tried to close this bug report without doing anything about it On Tue, Apr 1, 2014 at 7:47 AM, Jason Straight <j.straight-kde@straights.net > wrote: > https://bugs.kde.org/show_bug.cgi?id=162211 > > --- Comment #47 from Jason Straight <j.straight-kde@straights.net> --- > Okay, I can't imagine this would be hard to fix, after doing some more > investigating. > > I have been playing with this and had similar results as Andi, plus > noticed the > following: > > Every time I copy the Pictures folder to anywhere, not just remote media, > it > drops more than 1/2 the files with different numbers every time. > > I've also found that simply right clicking the folder and choosing > properties > will only give me a partial count, and every time I hit refresh - a > different > total of files. > > In total, this folder has 6677 files. > > I think this has to do with the speed the query for files vs how long the > kio > waits. And it has everything to do with the source, not the destination. > > I have two external btrfs formatted HD's which [when checking properties in > KDE] show a count of about 3400 at first, but show the right total every > time > afterwards hitting refresh in properties. After that count shows the actual > total of files, then copying them will copy them all. > > I have one copy on an NTFS partition on my SSD [fuse - slow], which never > shows > the right totals, no matter how many times I refresh. Anywhere between > 2400 and > 3000. > > Another copy [mounted via cifs] on wireless only returns a count of about > 1600 > in properties. > > Testing copying to various slow media such as flash drives and cifs mounts > works fine as long as the source media is fast enough to return the proper > total count. > > -- > You are receiving this mail because: > You are on the CC list for the bug. >
I think we are actually talking about two different bugs here actually. 1. A bug where KDE (kioclient?) will not recognize all the files in a folder/selection when it scans them prior to copying and when viewing properties. This leads to you trying to copy or move a folder with 6700 files and it actually only copies ~3400. 2. A different bug where you copy files and they exist on the destination, but they are 0 bytes sized. I have not experienced #2 myself.
Further evidence suggests this is a time issue. Also, a way to help reproduce the issue. On a USB3 mechanical drive formatted with btrfs filesystem I was not initially experiencing fluctuating counts with kioclient properties on the folder. When I put that same drive under a load using dd to write a file to it from /dev/zero I started getting the problem where clicking refresh in properties would yield random results on size and file count.
It's not "just" speed. There's more to it. It gets me the right counts on the NTFS fuse filesystem for Program Files and it takes a while to count them. For some reason the enumerating of files in my Pictures folder kicks out before it's done. Is there maybe a max time kioclient will wait while stating each file? And perhaps one or more of my pictures takes a longer than usual to stat? I'm open to try anything here. Give me some guidance on this. Hell, I'll even set up a virtual machine with ssh and vnc access for the devs with access to my photos. I have kde debugging symbols installed, but my C skills are old and weak, and my gdb skills are even less useful.
Jason please not only look at the number of files, verify the size of the directory ,source and destination in case there are zero length files, the names are in the directory, but the total files size count is different example , source =2000 files , total 6.3 GB, destination = 2000 fles total 4.8 GB got the same issue by ext2, ext3,ext4,ntfs formatted external usb 2.0 hd from western digital, and seagate After that i not used usb harddisks anymore as the backup is unreliable, but each time there is a new distro upgrade, i still try ,it to see if the issue still exist , and from 2008 until now it has not changed......in a few weeks after upgrading to 14.04 i will test again On Thu, Apr 3, 2014 at 4:44 AM, Jason Straight <j.straight-kde@straights.net > wrote: > https://bugs.kde.org/show_bug.cgi?id=162211 > > --- Comment #51 from Jason Straight <j.straight-kde@straights.net> --- > It's not "just" speed. There's more to it. > > It gets me the right counts on the NTFS fuse filesystem for Program Files > and > it takes a while to count them. For some reason the enumerating of files > in my > Pictures folder kicks out before it's done. > > Is there maybe a max time kioclient will wait while stating each file? And > perhaps one or more of my pictures takes a longer than usual to stat? > > I'm open to try anything here. Give me some guidance on this. Hell, I'll > even > set up a virtual machine with ssh and vnc access for the devs with access > to my > photos. > > I have kde debugging symbols installed, but my C skills are old and weak, > and > my gdb skills are even less useful. > > -- > You are receiving this mail because: > You are on the CC list for the bug. >
I took my bug [as I think we're talking about two different ones] and created a new one: https://bugs.kde.org/show_bug.cgi?id=333436
Using whatever the version of Dolphin is in Kubuntu 13.10, I copied 3701 files in 1507 folders totaling 17.5GB to an external FAT32 drive, and Dolphin only copied 777 files and 777 folders totaling 3GB. That 777 looks very suspicious.
It's a pity that this still exists (5 years without fix?) Few days ago I lost tons of family photos in KDE 4 (Ubuntu 14.04) when doing move from internal drive to external drive (ctrl-x, ctrl-v). During that process Dolphin has crashed. After that, I noticed, that all marked files on internal drives were wiped out, but on external drive there was only some of them (as proces was interrupted by crash) It seems that Dolphin wipes out source files without waiting for confirmation that files were successfuly moved to target location... I think it should be treated as some kind of hot prio error. It is known since 2009, it can cause critical data loss.
maybe we have to file this by mr Linus Torvalds self or at the kernel core development group(please give me a hint how/where) as it seems this is not the right place (or nobody has this specific knowledge) ....sad to see you used MOVE...so i can imagine the disaster.. . On Sun, May 11, 2014 at 2:50 PM, Arkadiusz <arekws@gmail.com> wrote: > https://bugs.kde.org/show_bug.cgi?id=162211 > > Arkadiusz <arekws@gmail.com> changed: > > What |Removed |Added > > ---------------------------------------------------------------------------- > CC| |arekws@gmail.com > > --- Comment #55 from Arkadiusz <arekws@gmail.com> --- > It's a pity that this still exists (5 years without fix?) > Few days ago I lost tons of family photos in KDE 4 (Ubuntu 14.04) when > doing > move from internal drive to external drive (ctrl-x, ctrl-v). During that > process Dolphin has crashed. After that, I noticed, that all marked files > on > internal drives were wiped out, but on external drive there was only some > of > them (as proces was interrupted by crash) > > It seems that Dolphin wipes out source files without waiting for > confirmation > that files were successfuly moved to target location... > > I think it should be treated as some kind of hot prio error. It is known > since > 2009, it can cause critical data loss. > > -- > You are receiving this mail because: > You are on the CC list for the bug. >
*** Bug 320890 has been marked as a duplicate of this bug. ***
as I wrote https://bugs.kde.org/show_bug.cgi?id=320890 bug reproduced on my PC. and this https://bugs.kde.org/show_bug.cgi?id=333436 also reproduced. perhaps there is a connection between them.
Perhaps the bug is related to the use of non-reentrant methods in kio (as signaled by cppcheck): Non reentrant function 'readdir' called. For threadsafe applications it is recommended to use the reentrant replacement function 'readdir_r'. (fixed in KF5). Non reentrant function 'strtok' called. For threadsafe applications it is recommended to use the reentrant replacement function 'strtok_r'. Non reentrant function 'getpwnam' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getpwnam_r'. Non reentrant function 'getgrnam' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getgrnam_r'. Non reentrant function 'getpwuid' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getpwuid_r'. Non reentrant function 'getgrgid' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getgrgid_r'. Non reentrant function 'getservbyname' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getservbyname_r'. on reentrant function 'getprotobyname' called. For threadsafe applications it is recommended to use the reentrant replacement function 'getprotobyname_r'.
It happend just now on my machine Ubuntu 12.10 +KDE plasma From file manager nautlius , I copy four folders around 180GB from PC to USB hardisk. After two folders, around 110GB are copied, I count the file number on the usb harddisk, there are same amount around 11k files as on my PC. Then I stop file copying. After that, from nautlius I try to copy the third folder over (around 7000+ files) to my USB harddisk. Cancel transmition when there are 3000 files on the USB disk. All looks good, counting files all, the same. So unmout, and remove USB disk. When I open the usb harddisk on anther PC There are only 2960 fiels lef t!!!!!!!!!!!!!!!!!!!!!! Only a portion of first two folders. The third folder and its subfolders DO NOT EXISTS at all.
This bug just affected me!.. Actually I knew about this bug for a long time, but I stupidly forgot about it when I was copying files for some one!.. I am in japan and about a year ago I hacked some japanese family's wii that I am friends with.. I spent a long time downloading tons of wii games for them, and also lots of NES and SNES roms and put all the games on to a hard-drive attached to their wii.. This year they got a bigger HD for the games.. I stupidly forgot about this bug, and used KDE to transfer tons of small files from their old HD to their new HD.. It copied most (but not all) of the nes/snes roms, but only copied ONE wii game!.. I had to tell them that all of their wii games are gone (save for 1) because of a giant bug in linux.. Do you know how bad that makes linux look?.. This bug is the ONLY reason I can't recommend linux to any one.. Who would want to use an OS that secretly erases your files when you transfer stuff via USB!??.. Tons of people can easily reproduce this bug, so why hasn't it been fixed in seven years??.. To me, this is by far the biggest bug in KDE.. I am just wondering how many more years it is going to be until I can safely transfer files using USB.. If this bug were in mac-os or windows, it would be fixed immediately, but since it is linux, and people only fix things when they feel like it, a colossal bug like this has still been unfixed for 7 years and counting.. I hope this bug is fixed this year.. I want to be able to talk about how great linux is again.. But I just can't recommend it when it has a bug this huge in it.. (And I am definitely not recommending gnome to any one though.. gnome sucks..)
use long path tool for easy solution.
> Tons of people can easily reproduce this bug But none of the developers. Also, it is still unclear if this issue is due to removing the external device too early, i.e. before all the cached contents is written out.
> But none of the developers. Also, it is still unclear if this issue is due to removing the external device too early, i.e. before all the cached contents is written out. I didn't realize that none of the developers can manage to reproduce it, that is really odd.. I think I am able to reproduce it on this computer, but not sure if I would be able to give any information that would help?.. I know the USB hard-drive had tons of small files in it (mainly every NES and SNES rom ever made), so not sure if that triggers the bug to happen or not.. Also, for the record, I always unmount HDs/flash-drives/SD-cards using dolphin, and I always wait until it says "you can unmount now", and actually before I do that, I usually open a console and type "sync" and wait until that is done before I unmount, because I am paranoid about the linux buffer saying files are copied already, when actually they are being copied still in the background.. I especially do this for important stuff.. Any way, I'm not sure if there is any information or tests or logs I can submit that would help, but if there is, I would be happy to do it.. I use debian stable, so I am stuck with this KDE version for a long time..
I have a stereo that takes a usb thumb drive with mp3s or whatever loaded on it. I just installed ubuntu and when i transferred some files to it everything appeared to go well. When I put it in the stereo, tracks would play but either get cut off randomly or even sometimes play one song for a few seconds, a part of another song, back to the first and so on. I thought my files were corrupt, but it plays fine on the computer. After reading this I wondered if I had made the right choice installing ubuntu. I tried several things, none of which worked, but I did learn something that might be helpful. I dropped the mp3 files into the root directory and they play fine now. if they are in folders, despite the name, they always mess up. Hope this helps someone fix this. It's really kinda annoying
Reproduced using the following steps on Gentoo: 1. Created temporary LVM volumes for testing (test1 and test2) in a thin-provisioning pool. 2. Formatted both with BTRFS. 3. Mounted at /mnt/test1 and /mnt/test2 4. Created large number (>65536) of files and folders in semi-random structure on /mnt/test1 with very long path-name components (>200 bytes per level), and filled files to random sizes with zero bytes. 5. Using cgroups, artificially limited disk bandwidth for the current shell 6. Using kioclient, attempted to copy files recursively form /mnt/test1 to /mnt/test2. 7. Flushed all buffers to disk and unmounted /mnt/test2 (double sync followed by umount from commandline). 8. Remounted /mnt/test2 and used diff to determine differences between /mnt/test1 and /mnt/test2. 9. Repeated steps 6-8 multiple times, each time the results from diff were different. 10. Repeated steps 6-8 using rsync instead of kioclient, and diff reported no differences. 11. Repeated steps 6-8 without the disk bandwidth limited using cgroups, and diff reported no differences. 12. Repeated steps 6-8 with shorter filenames and the disk bandwidth limited, and still reproduced the issue. Above results lead me to believe (without looking directly at the code) that kio is timing out trying to gather directory contents and/or calling stat on files.
Apologies about the double comment.
I'm an user that has slso been impacted by this bug. Can you post output of kioclient with all debug options checked? That is, open kdebugdialog, search kio in that window and check everything and then running kioclient. According to some comments here ( https://www.reddit.com/r/kde/comments/1ak1lz/psa_dolphin_exhibits_data_loss_do_not_use_it_for/ ) outputing that info prevents the timing out in some way. It might be useful to redirect the output like this: kioclient copy src dst > errorlog.txt 2>&1 Also, thanks for your great effort. By the way, this bug should be marked as CRITICAL.
Yeah, but it might be a while. As of right now I don't have most of KDE installed (out of the entirety of KDE suite, all I use personally is Okular, Calligra, and a couple of the games). In fact I specifically installed kioclient for the sole purpose of testing this (I'm trying to help get rid of long standing bugs like this in FOSS projects whenever I stumble across them). As I'm running Gentoo, this in turn means compiling everything, which based on past experience with KDE will take a while. I will put it on the top of my list for tomorrow however.
(In reply to Austin S. Hemmelgarn from comment #67) I can't reproduce the bug following your steps: Kubuntu 14.04.2 Linux alberto-VirtualBox 3.16.0-46-generic #62~14.04.1-Ubuntu SMP Tue Aug 11 16:28:19 UTC 2015 i686 i686 i686 GNU/Linux Enter root and then 1. vgcreate -s 64M vg_thin /dev/XXXX lvcreate -L 3G --thinpool tp_tecmint_pool vg_thin lvcreate -V 1G --thin -n thin_vol_client1 vg_thin/tp_tecmint_pool lvcreate -V 1G --thin -n thin_vol_client2 vg_thin/tp_tecmint_pool 2. mkfs.btrfs /dev/vg_thin/thin_vol_client1 mkfs.btrfs /dev/vg_thin/thin_vol_client2 3. mkdir /mnt/test1 mkdir /mnt/test2 mount /dev/vg_thin/thin_vol_client1 /mnt/test1 mount /dev/vg_thin/thin_vol_client2 /mnt/test2 4. rm -rf /mnt/test1/* ./create_random_files.py /mnt/test1 ##BEGINNING OF SCRIPT (Python3) #!/usr/bin/env python3 import os import random import string import sys min_depth=3 max_depth=6 elem_by_folder_min=5 elem_by_folder_max=8 size_max_files_kb=50 letters_min=210 letters_max=220 files_and_folders_created=0 def rand_lett(): m=random.randint(letters_min,letters_max) return ''.join((random.choice(string.ascii_letters) for _ in range(m))) def create_file(name): open(name,'wb').write(os.urandom(random.randint(1,size_max_files_kb*1024))) def create_folder(folder_name,depth): global files_and_folders_created if depth != 0: os.mkdir(folder_name) os.chdir(folder_name) for _ in range(random.randint(elem_by_folder_min,elem_by_folder_max)): if depth < random.randint(min_depth,max_depth): create_file(rand_lett()) create_folder(rand_lett(),depth+1) files_and_folders_created=files_and_folders_created+2 os.chdir("..") create_folder(sys.argv[1],0) print ("Folders and files created:"+str(files_and_folders_created)) ##END OF SCRIPT 5. ls -ls /dev/vg_thin/* Those are symlinks, following them: ls -l /dev/dm-* Major and minor numbers are: 252:4 y 252:5 grep blkio /proc/mounts || mkdir -p /cgroup/blkio ; mount -t cgroup -o blkio none /cgroup/blkio cgcreate -g blkio:/iothrottle echo "252:4 5000000" > /cgroup/blkio/iothrottle/blkio.throttle.read_bps_device echo "252:4 5000000" > /cgroup/blkio/iothrottle/blkio.throttle.write_bps_device echo "252:5 5000000" >> /cgroup/blkio/iothrottle/blkio.throttle.read_bps_device echo "252:5 5000000" >> /cgroup/blkio/iothrottle/blkio.throttle.write_bps_device 5.1. Put every task in iothrottle group ps axu | awk '{print $2}' | tail -n +2 | python3 -c 'import sys;a=sys.stdin.readlines(); b=[open("/cgroup/blkio/iothrottle/tasks","a").write(val) for val in a]' 6.pre (I've doing all of that in root account) chown -R XXX /mnt logout 6. rm errorlog.txt ; rm -rf /mnt/test2/* ; (kioclient copy /mnt/test1/* /mnt/test2) >&errorlog.txt 7. sync ; sync ; echo 3 | sudo tee /proc/sys/vm/drop_caches ; echo 3 | sudo tee /proc/sys/vm/drop_caches ; sudo umount /mnt/test2 8. Look for differences (Which files have been copied and their sizes): sudo mount /dev/vg_thin/thin_vol_client2 /mnt/test2 && diff <(find /mnt/test1/ -printf '%P%s\n' | sort) <(find /mnt/test2/ -printf '%P%s\n' | sort) Am I missing something?
Interesting, I can't seem to reproduce it now either. Guess I'm off to run hardware tests on my system.
Would it help any if i used a different linux distribution? I am new to linux and installed ubuntu
There s a lot of people out there having the exact same bug all these years and it's really a pity that no action has been taken since the first reports came out. I recently fill a bug report https://bugs.kde.org/show_bug.cgi?id=352761 I'm really disappointed. Data loss is the most critical issue and after all these years we are still afraid to use a simple command such as copy. Are we being pushed in Windows?
Hello, As you can see, if you read all the comments, this is a hard to reproduce bug, even for the developers. For example, I've tried to reproduce it several times, and I've seen it only once, and after doing the same steps, it didn't reproduce again. It is almost impossible to fix a bug if you can not reproduce it because you have no way to know if it is fixed or not. Has anyone that was able to reproduce this bug with KDE SC 4 tried to reproduce it with the newest KF5 dolphin? (Just to know if it could still be there or not)
Hello Jaime, thank you for your answer. As i state in the aformentioned bug report (https://bugs.kde.org/show_bug.cgi?id=352761) this problem still exists in these versions kdelibs version is 4.14.11 dolphin 15.0.8.0 plasma-framework 5.13.0 I agree that you can't fix something that isn't reproduced by you (the developers) and everyone clearly understand that. But, (and this is a big emphazised ''but''), with a simple google search you can see thousands of posts, all over the net, in various hardware setups, in various distros, and all of them have experienced data loss when copy or moving things around. Some of them can clearly provide more technical info that should give you insights on whats happening. We are willing to help you, thats why we raise all those bug reports, to contribute, from our side, to the community, to bring more stable and compete (and complete) de. We can realize that its a difficult task, to hunt down those bugs, but the completion should be rewarding for all the community. Personally i lack the developer skills, though i'm willing to help, to provide the info you may want.
I've just copied 50 old dvd's with some backup files to an external usb drive (using Krusader) and at the end I noticed that there are lots of missing files on the usb drive. When I checked this out it turned out that not a single DVD was completely copied to the usb drive! To be honest I was shocked that modern version of OS may have such a bug. Moreover, from my point of view it is trivial to reproduce the bug (as I said not a single dvd was properly copied), hence I cannot believe that developers didn't noticed it. My setup: Ubuntu 14.04.3 LTS (with all updated installed) krusader: 1:2.4.0~beta3-2 kdelibs-bin: 4:4.13.3-0ubuntu0.2 Now I am a bit scared how many times I didn't noticed that the 'copy' command in krusader is not reliable.
Based on Alberto J. Ruiz script, I've made a similar script with infinite repetition (until the copies are not equal to the original). It has been running all week copying to an usb pendrive (I do not have an external usb hard disk) and also copying to another sata harddisk. It should have copied more than 10.000 files, and it has never reproduced this bug so far (using the latest KF5 version compiled from sources last weekend). But we will keep trying...
I made a simple screen-cast to show you the bug in action: http://sendvid.com/dc92o1yu As I said, I have no problem with reproducing the bug, so feel free to ask me for help in debugging an issue. Some additional info: It seems that the destination of 'copy' doesn't matter - it is exactly the same with usb and sata drives (both ext4). The source is ISO 9660. Each time I press 'copy' on the folder the command works better - it finds more files to copy. After 2 or 3 repeats it does recognize all files. After unmounting/mounting everything starts from the begging.
(In reply to wynajem_pn from comment #79) > I made a simple screen-cast to show you the bug in action: > http://sendvid.com/dc92o1yu As I said, I have no problem with reproducing > the bug, so feel free to ask me for help in debugging an issue. Some > additional info: It seems that the destination of 'copy' doesn't matter - it > is exactly the same with usb and sata drives (both ext4). The source is ISO > 9660. Each time I press 'copy' on the folder the command works better - it > finds more files to copy. After 2 or 3 repeats it does recognize all files. > After unmounting/mounting everything starts from the begging. All right. First, can you reproduce the bug using kioclient? "kioclient copy source destination". If so, open kdebugdialog. Tick Everyoption. Use kioclient to reproduce again. If you can reproduce it with debug flags, paste the output of kioclient. Thank you
Created attachment 94809 [details] output of kioclient copy It is exactly the same with kioclient. Log file attached.
Something looks fishy : kioclient(12516)/kio (KIOJob) KIO::SimpleJob::~SimpleJob: Killing job KIO::SimpleJob(0x17fe690) in destructor! "[ 0: /usr/lib/libkdecore.so.5(kRealBacktrace(int)+0x3b) [0x7fb19cbe476b] 1: /usr/lib/libkio.so.5(KIO::SimpleJob::~SimpleJob()+0x86) [0x7fb19e393306] 2: /usr/lib/libkio.so.5(KIO::ListJob::~ListJob()+0x9) [0x7fb19e393609] 3: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QObject::event(QEvent*)+0x288) [0x7fb19c51fc58] 4: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(QApplicationPrivate::notify_helper(QObject*, QEvent*)+0x8c) [0x7fb19d174e2c] 5: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(QApplication::notify(QObject*, QEvent*)+0x270) [0x7fb19d17b4a0] 6: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QCoreApplication::notifyInternal(QObject*, QEvent*)+0x6d) [0x7fb19c5074dd] 7: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*)+0x1ed) [0x7fb19c50ab3d] 8: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(+0x1aaf83) [0x7fb19c534f83] 9: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_dispatch+0x254) [0x7fb198bd1e04] 10: /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x49048) [0x7fb198bd2048] 11: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_iteration+0x2c) [0x7fb198bd20ec] 12: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)+0x71) [0x7fb19c5347a1] 13: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(+0x26bbe6) [0x7fb19d216be6] 14: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)+0x2f) [0x7fb19c5060af] 15: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)+0x175) [0x7fb19c5063a5] 16: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QCoreApplication::exec()+0x89) [0x7fb19c50bb79] 17: kioclient() [0x4046d8] 18: kioclient() [0x405518] 19: kioclient() [0x403a58] 20: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fb19bce2ec5] 21: kioclient() [0x403fa8] ] " Looks like a job is destroyed before finishing. I almost forgot, sometimes dolphin enumerates files incorrectly in the properties dialog (See bug 333436). Have you lost files during this kioclient copy? I mean, you navigate through folders and there aren't files that are supposed to be there. If that's the case, you have successfully reproduced this bug after 7 years and we can start working on it.
In fact during the logged run of 'kioclient copy' it copied 170MB out of 3,6GB and there are more than 300 missing files.
At last some progress. This is very promising and we hope you can find a solution. A little hint This bug occurs even if you try to copy from an external disk (either ntfs or ext4) to your internal disk. [the source is external-hdd and destination is internal sata hdd] It doesn't matter if dolphin enumerates files incorrectly in properties, the actual problem is that it skips a tremendous amount of files to be copied.
I'm not a KDE developer, only a rookie C++ developer. In source code kdelibs-4.13.3/kio/kio/jobs.cpp : void ListJob::slotResult( KJob * job ) { if (job->error()) { // If we can't list a subdir, the result is still ok // This is why we override KCompositeJob::slotResult - to not set // an error on parent job. // Let's emit a signal about this though emit subError(this, static_cast<KIO::ListJob*>(job)); } removeSubjob(job); if (!hasSubjobs()) emitResult(); } Couldn't this be masking other errors not accounted? We should check for ERR_ACCESS_DENIED and if it's not, mark the error the same way KCompositeJob::slotResult does. This is just a theory. The bug might be a completely unrelated.
Created attachment 94949 [details] Possible patch for bug 162211 The faulty part of the copying process seems to be in ListJob. 1. ListJob fires the signal subError in case of error but it doesn't get upwards. (Such as cannot enter subfolder). 2. The fact that the ListJob must work even if some subfolders cannot be entered. ListJob::slotResult ommits all errors. 3. When such error happens, slotResults removes the subjob when it shouldn't. It should return immediately like this function. void FileCopyJob::slotResult( KJob *job) { ..... if ( job->error() ) { ..... emitResult(); return; } .... } I have reproduced the backtrace with "kioclient copy ~/src ~/dst" ~/src contains the following: src: total 4 drwxrwxr-x 3 alberto alberto 4096 oct 7 18:45 c src/c: total 548872 -rw-rw-r-- 1 alberto alberto 562036736 oct 7 14:01 a drwx------ 2 root alberto 4096 oct 7 18:44 b src/c/b: total 1024 -rw-rw-r-- 1 alberto alberto 1048576 oct 7 18:44 aleatorio The "a" file belongs to root and chmod'ed to 700 to emulate the "Cannot access folder". The file is made of randomness. After applying those changes, the backtrace has disappeared and kioclient does not fail silently. Tested on Kubuntu 14.04 . Patch for KdeLibs 4.14.12.
Created attachment 94951 [details] Proposed patch Actually, this is my proposed patch, not the otherone.
I'm affected by this bug too (well, not me directly, but my girlfriend, and as a teacher it's very problematic for her). When we try to copy a folder that contains several files and folders from her /home/Documents to a USB drive (no matter what filesystem the drive is), KDE / Dolphin will copy the folders only. Every folder will be there on the USB drive, but no file at all. We noticed though, that sometimes if we copy the content of the upper folder (but not the folder itself, just the content), it works. She's running up-to-date Arch Linux (Plasma 5.4.2, latest frameworks). I installed Thunar on her system and copying with Thunar works. So the problem lies somewhere in KDE...
I lost the files when copying from hdd to hdd. Without using the USB drives. Today checked on 2 PCs Archlinux in Konqueror, Krusader, Dolphin bug not reproduced. Previously, he consistently played. Now it is fixed.
*** Bug 357875 has been marked as a duplicate of this bug. ***
I just lost a bunch of precious pictures due to a bug in Dolphin. I was doing maintenance on my photo collection. I had previously downloaded some jpg and raw files directly from the camera onto USB drive "A". I keep my photos organized on USB drive "B". I thus attempted to move the photos from USB drive A to USB drive B with Dolphin. Both USB A and USB B are multi GB SATA 3.5" hard drives with USB 3.0 interfaces. They are formatted EXT4. As near as I can remember, here is what happened: 1) I opened Dolphin. I split the screen. I selected USB drive A on the left panel. I selected USB drive B on the right panel. 2) I did a bit of browsing on USB A to ascertain which folders I wanted to move. I created a new folder on USB drive B. 3) I selected about 10 folders containing several hundred images from drive A. I dragged them over to the drive B panel in Dolphin and dropped them on the folder I created. I selected Move from the pop up menu. Not copy. 4) The process started. I opened the KDE plasma notification window and expanded the info to watch the files being copied. The process took about 10 minutes. The CPU load was very high, like 100%, on the plasma shell process. 5) At the conclusion of the process I just happened to open up a jpg to have a look at it. I started scrolling through the files only to find that several sub folders had images with file sizes of 0B. They could not be opened. The file content was lost even if the handle was there. 6) When I went back to the source folders on drive A, the images are, of course, gone. The hard drives have both been fine. I ran fsck on them and both came up clean. At no point in this process were the drives unmounted or disconnected. I got called away from my desk and rechecked the files again later. The file sizes were still zero. I've seen files lost like this using Dolphin before. I suspect there is a bug in the kio process when transferring large blocks of files to an external USB drive. What is interesting is that if you use mv or cp on the command line, the CPU load is way, way lower than if the files are moved from Dolphin. Like 5% versus 100% and the command line operations are quite a bit faster. $ uname -a Linux localhost.localdomain 4.7.9-200.fc24.x86_64 #1 SMP Thu Oct 20 14:26:16 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux kf version 5.26.0.1.fc24 Been using Linux exclusively on my workstation since Redhat 8. I highly doubt this is user error. Let me know if you need more information.
Is it fixed now as stated in this comment ? https://www.reddit.com/r/linux/comments/8652zc/umm_gnome_shell_has_a_rather_big_memory_leak/dw2kgzy/
*** Bug 336624 has been marked as a duplicate of this bug. ***
I think that I have been just affected by the problem reported here. I extracted a zip file containing 104715 files and 1678 subfolders (900 MiB) from a NTFS partition (internal hard disk) to my home (EXT4) using Ark. I followed the steps below: open dolphin right click zip file, select "Extract > Extract archive to..." browse to some place in another partition and create a new folder click "extract" button to start extraction when extraction is completed, open destination folder, select all files/folders just extracted and press alt+enter to open properties window observe amount of files/sub-folders open zip file using Ark, press alt+enter to open zip file properties, observe "number of entries" info Result: amount of files/sub-folders shown by Dolphin and "number of entries" shown by Ark do NOT match It happned twice. Each time the zip file was extracted, Dolphin shown a different amount of files/sub-folders in properties dialog. Now I can not reproduce anymore. My system: Arch Linux plasma 5.12.5 frameworks 5.45 Dolphin 18.04
*** Bug 335924 has been marked as a duplicate of this bug. ***
I'm worried to use Kubuntu now. I wonder if there is really a chance to lose data on KDE. If it's the case I prefer not using it. So what's the state of this data loss bug ? Thank you.
Hi Dr Chapatin, can you double check the actual number of files with another app, from the terminal for instance ? Did you also check the total size ? It rings me a bell but I don't remember what I concluded... (In reply to Dr. Chapatin from comment #94) > I think that I have been just affected by the problem reported here. > I extracted a zip file containing 104715 files and 1678 subfolders (900 MiB) > from a NTFS partition (internal hard disk) to my home (EXT4) using Ark. > I followed the steps below: > > open dolphin > right click zip file, select "Extract > Extract archive to..." > browse to some place in another partition and create a new folder > click "extract" button to start extraction > when extraction is completed, open destination folder, > select all files/folders just extracted and press alt+enter to open > properties window > observe amount of files/sub-folders > open zip file using Ark, press alt+enter to open zip file properties, > observe "number of entries" info > Result: amount of files/sub-folders shown by Dolphin and "number of entries" > shown by Ark do NOT match > > It happned twice. Each time the zip file was extracted, Dolphin shown a > different amount > of files/sub-folders in properties dialog. Now I can not reproduce anymore. > > My system: > Arch Linux > plasma 5.12.5 > frameworks 5.45 > Dolphin 18.04
Was proposed patch in comment 87 applied?
No. I don't think any of the developers are able to reproduce this issue.
On Reddit another user has been hitted by this bug: https://www.reddit.com/r/linux/comments/9a7k2i/psa_on_kde_never_use_the_gui_when_copying_a_large
Can anybody reproduce this issue reliably? If yes, it'd be helpful to have a kioclient log like the one in comment 81. As mentioned in the Reddit thread, maybe an e2image which contains the file metadata could help us as well to further investigate this bug.
(In reply to FiNeX from comment #100) > On Reddit another user has been hitted by this bug: > https://www.reddit.com/r/linux/comments/9a7k2i/ > psa_on_kde_never_use_the_gui_when_copying_a_large I'm guessing here, but since the poster only lost files 4 levels down in the directory tree, could the bug be triggered by especially long character filenames?
*** Bug 262768 has been marked as a duplicate of this bug. ***
Dear Bug Submitter, This bug has been in NEEDSINFO status with no change for at least 15 days. Please provide the requested information as soon as possible and set the bug status as REPORTED. Due to regular bug tracker maintenance, if the bug is still in NEEDSINFO status with no change in 30 days the bug will be closed as RESOLVED > WORKSFORME due to lack of needed information. For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging If you have already provided the requested information, please mark the bug as REPORTED so that the KDE team knows that the bug is ready to be confirmed. Thank you for helping us make KDE software even better for everyone!
I'll set this back to REPORTED so that this bug will not get closed for NEEDSINFO inactivity. If you can reproduce this issue reliably (preferably in a small, contained environment) we would of course be interested to investigate this bug further.
(In reply to Julian Schraner from comment #105) > I'll set this back to REPORTED so that this bug will not get closed for > NEEDSINFO inactivity. If you can reproduce this issue reliably (preferably > in a small, contained environment) we would of course be interested to > investigate this bug further. That you, Julian, I was actually just about to submit a post pleading to not do this, but then I saw your comment. I would also use this opportunity to perhaps point out that, from my, understanding, the video proof (https://bugs.kde.org/show_bug.cgi?id=162211#c79) refutes a claim a KDE dev made recently that this bug is primarily related to not doing safe remove. BTW, what is the consensus on Alberto's analysis and fix? Does it look like it addresses the issue, just that we would need to actually test it out in a situation where the bug is reproduced?
(In reply to wynajem_pn from comment #79) > I made a simple screen-cast to show you the bug in action: > http://sendvid.com/dc92o1yu As I said, I have no problem with reproducing > the bug, so feel free to ask me for help in debugging an issue. Some > additional info: It seems that the destination of 'copy' doesn't matter - it > is exactly the same with usb and sata drives (both ext4). The source is ISO > 9660. Each time I press 'copy' on the folder the command works better - it > finds more files to copy. After 2 or 3 repeats it does recognize all files. > After unmounting/mounting everything starts from the begging. Are you able to maybe try this? run tree /path/to/source > ~/origfiles.txt export QT_LOGGING_RULES=kf5.kio.*=true kioclient5 copy /path/to/source /path/to/destination &> ~/copylog.txt tree /path/to/destination > ~/copiedfiles.txt diff -rq /path/to/source /path/to/destination > ~/copydiff.txt and then attach origfiles.txt copylog.txt copiedfiles.txt copydiff.txt to the report? that might help the devs, provided that the issue replicates with using kioclient to copy the files
Created attachment 116131 [details] More files than before When trying to reproduce this bug Dolphin tells me I get more files than I originally had. The screenshots depicts what happens when I copy the WinSxS folder from the Windows NTFS partition to a Linux home ext4 partition. Both the number of files as well as the size on the disk increases. No idea is this is relevant at all, but it may point to some numeration problems that still do exist.
(In reply to Filip F. from comment #108) > Created attachment 116131 [details] > More files than before > > When trying to reproduce this bug Dolphin tells me I get more files than I > originally had. > > The screenshots depicts what happens when I copy the WinSxS folder from the > Windows NTFS partition to a Linux home ext4 partition. > > Both the number of files as well as the size on the disk increases. No idea > is this is relevant at all, but it may point to some numeration problems > that still do exist. I know WinSxS is full of hard links, and NTFS junctions.... Maybe run cd /run/media/filip/System/Windows/WinSxS find > ~/TargetList.txt cd /home/filip/Testing/WinSxs find > ~/DestinationList.txt and attach the two files that will then be in your $HOME?
(In reply to bluescreenavenger from comment #109) > (In reply to Filip F. from comment #108) > > Created attachment 116131 [details] > > More files than before > > > > When trying to reproduce this bug Dolphin tells me I get more files than I > > originally had. > > > > The screenshots depicts what happens when I copy the WinSxS folder from the > > Windows NTFS partition to a Linux home ext4 partition. > > > > Both the number of files as well as the size on the disk increases. No idea > > is this is relevant at all, but it may point to some numeration problems > > that still do exist. > > I know WinSxS is full of hard links, and NTFS junctions.... > Maybe run > cd /run/media/filip/System/Windows/WinSxS > find > ~/TargetList.txt > cd /home/filip/Testing/WinSxs > find > ~/DestinationList.txt > > and attach the two files that will then be in your $HOME? It's not letting me attach the .txt files because both are 15 MiB each, but the number of lines is identical in both of them so what you said must be true. Thanks for the clarification, will keep on trying to reproduce this bug here and there.
(In reply to Filip F. from comment #110) > (In reply to bluescreenavenger from comment #109) > > (In reply to Filip F. from comment #108) > > > Created attachment 116131 [details] > > > More files than before > > > > > > When trying to reproduce this bug Dolphin tells me I get more files than I > > > originally had. > > > > > > The screenshots depicts what happens when I copy the WinSxS folder from the > > > Windows NTFS partition to a Linux home ext4 partition. > > > > > > Both the number of files as well as the size on the disk increases. No idea > > > is this is relevant at all, but it may point to some numeration problems > > > that still do exist. > > > > I know WinSxS is full of hard links, and NTFS junctions.... > > Maybe run > > cd /run/media/filip/System/Windows/WinSxS > > find > ~/TargetList.txt > > cd /home/filip/Testing/WinSxs > > find > ~/DestinationList.txt > > > > and attach the two files that will then be in your $HOME? > > It's not letting me attach the .txt files because both are 15 MiB each, but > the number of lines is identical in both of them so what you said must be > true. Thanks for the clarification, will keep on trying to reproduce this > bug here and there. Can you please archive them? Text files can usually be compressed effectively.
Created attachment 116152 [details] DestinationList & TargetList Didn't know that, here are the files in a .7z archive
(In reply to Filip F. from comment #112) > Created attachment 116152 [details] > DestinationList & TargetList > > Didn't know that, here are the files in a .7z archive The lists seem to be identical.
*** This bug has been confirmed by popular vote. ***
(In reply to Filip F. from comment #108) > Both the number of files as well as the size on the disk increases. No idea > is this is relevant at all, but it may point to some numeration problems > that still do exist. probably because bug 238424
I made a script that makes 1,000,000 randomly placed files, with random names, with various sizes from 1kb to 1gb, into 1,000,000 randomly placed folders, with random names. It selects 3 random other directories to fill them with 100,000 additional files each as well. Then it uses kioclient5 to copy the items. https://raw.githubusercontent.com/n3rdopolis/otherstuff/master/kiocopy The names have kinds of random numbers, symbols, some are 20 chars long, some have weird symbols that try to break parsing. It then lets me diff the trees. I I tried it on my KDE frameworks 5.44 box... ...but after all that, I was NOT able to replicate it. the trees match. I have a log, and a seed file for the script, but the .tar.xz is 83MB, and it's quite worthless because it worked perfectly anyway. I am at a loss at how to replicate it. maybe I should change the outlier generated maximum file name size to 220 chars?
One odd thing I notice during the test, is that the kioslaves have the high CPU usage, kioclient5 has the high memory usage. I assume that means kioclient5 stores the list of files to copy, and then it must make the kioslave do the copying. My theory was maybe a process (possibly a helper is crashing on the reporters computers, such as kio_file.so) So I tried a smaller, manual test with Dolphin. Copying a smaller directory of ~240000 files, Dolphin's memory usage grows a bit (not a lot). So that confirms my theory that Dolphin must stay alive to tell the kioslaves what file to copy. Killing the kioslave that the CPU usage spiked with did NOT result in a silent failure. I got a dialog warning that the file protocol died, and it gave me some options to recover which it recovered from fully. So it's not silently failing because of that. HOWEVER trying again, I went after Dolphin this time. I closed the Dolphin window. The Dolphin process smartly stayed running to continue the copy (so the issue is not the Dolphin window being clsed). As a test, to simulate a crash or abrupt exit of Dolphin, I killed the Dolphin process. Sure enough the copy stopped. ...and I got a "The copy has completed successfully" notification. I wonder if any of the reports are because of Dolphin going down?
*** Bug 352761 has been marked as a duplicate of this bug. ***
Your correct in your analysis of how KIO works. I did KIO speedups last week that should somewhat reduce the CPU and memory on the file.so slave for many files. >I wonder if any of the reports are because of Dolphin going down? Note that was fixed in 2beb1a0ad23177f7dc2e5ee622bed3a70f671278 in plasma-workspace. I CC'd a different bug. ---- The history of this bug is quite messy, most reports date back to before the potentially important e60e8b96b211290e330c7ff6a3b84b20ab85850d on kdelibs handling of inaccessible folders. This bug was reopened primarily from someone citing copying from a zip which is an entirely different story. There's not a lot to suggest we still have the same issue as we might have had when that bug was opened. Then there's also noise from people not unmounting drives and potentially some reports from kuiserver handling of the host app crashing that I just mentioned which I fixed only very recently. I've tested copying a million files repeatedly, bluescreenavenger has done the same.
Cool. I will also add that ran a second test with my large tree generating script script to check all the files have all the correct sizes. At first I saw some differences, but then I realized that I was copying from a test ext4 image to a test btrfs image. The lines with the different sizes in my diff were all *directories* and no files were clipped... I guess not all dirs in BTRFS are 4096 bytes like in EXT4
I like how you've ended up going down the same paths and patterns as I did ! Running stat on a directory is pretty FS independent, it's 0 bytes on NTFS. ls sometimes puts different results too I did propose something https://phabricator.kde.org/D17071 but I'm not very convinced.
(In reply to David Edmundson from comment #121) > I like how you've ended up going down the same paths and patterns as I did ! > > Running stat on a directory is pretty FS independent, it's 0 bytes on NTFS. > ls sometimes puts different results too > > I did propose something https://phabricator.kde.org/D17071 but I'm not very > convinced. I used the tree command and the -s flag to show the sizes instead of the Properties dialog however. I saw all kinds of lines that were different on the BTRFS side. Instead of being fixed at 4096, there were lines that where coming up when I diffed the files, at say 0 bytes, 72 bytes, 54 bytes; etc. I got really concerned that I might have replicated it, but then I realized when I added the -F flag to append a / to directories, that every single line that was different was a directory, and all the files where the same So as it turns out, it looks like BTRFS also varies from EXT4 on how it measures the size of directories.
(In reply to David Edmundson from comment #119) > > >I wonder if any of the reports are because of Dolphin going down? > > Note that was fixed in 2beb1a0ad23177f7dc2e5ee622bed3a70f671278 in > plasma-workspace. I CC'd a different bug. https://bugs.kde.org/show_bug.cgi?id=401055
As per #119 I am closing this report. There was an issue in kdelibs that was fixed many many years ago. Most comments here relate to that. It was reopened with unrelated reasons. There was also some mishandling in kuiserver which is fixed quite recently which could give misleading messages which confuses this bug. No attempts to reproduce a bug with up-to-date code has proved successful despite multiple efforts from multiple people. That isn't to claim kio is bug free, but it will be much more productive to treat that as a new bug with new information than to add data here which is full of fixed outdated noise.
I THINK I JUST REPLICATED IT!!!!! I should have been MUCH more aggressive with the File Sizes, but I was was worried about off-by-1 errors causing my Script to attempt to create invalid files, and not create enough. So the script is somewhat incomplete trying to make 255 char files, it doesn't generate as many files as I was before. (as Unicode chars were causing too large names) Copying with kioclient, with all the KIO logs on, there is a big line "Could not make folder /tmp/kiocopy/1545096048/folder_dest/testdir/0l>╚>/1H{n.4🐧N5>2ä<╚#uDJjà57vÉ7BFI*;9jÄ>w@Vdäâ2@Op*8w>i5-by4P{èåhÄÄDw9î1RÆ4x3©h23uÉè)<W?&&,🐧WÄ31Hbv5î,7VTDCrÉï'So🐧D#\"}127z🐧4ij8f®uZ58®pÄJHê®9ÄÅ4Æ-Dj zpN0èg1JB𖡒?Zh{Hom6𖡒B9i1\"èEms2.1.5¾ch@ÅWÆ<äU})2CW1h\t𖡒V9g�." at the end, and then kioclient5 exits with the status of 1! Diffing the trees the final lines: -17 directories, 413 files +7 directories, 0 files I will attach the report that the script made, containing the tree listings, and containing the log for the kioclient5, and containing the Seed file. Passing the path to the Seed file, causes it to generate the exact same files that it did for me. The script will fail to create SOME files before it runs the actual test (because of the mentioned byte count issue). I will also attach the version of the script that replicated the issue Let me know if I need to provide any info. This was kio 5.44 admittedly, ...but I also get similar results from a version from Mid October 2018, built from Git. But it looks like it fails to create that folder, and quits.
Created attachment 116980 [details] Archive containing the logs, and the tree, and the Seed.txt file to pass to the kiocopy script
Created attachment 116981 [details] Bash script of which to use to recreate the file tree that failed to copy. Pass the path to the Seed.txt to it.
The filename limit is 255 bytes, not 255 characters. In UTF-8, any non-ASCII character needs more than 1 byte. Additionally, the last character looks cropped, causing an illegal UTF-8 name, which Qt does not handle.
ALSO, for what it's worth, if I leave the two LC_ALL=C LANG=C variables set when running kioclient5, instead of setting them back with the script I get... -17 directories, 413 files +8 directories, 221 files Setting it back to LANG=en_US.UTF-8 LC_ALL= and I get the apparent silent failure (I set those variables in hopes of working around trying to fit Unicode chars in 255 bytes in the bash/generation portion...) I REALLY hope this info is helpful
(In reply to Christoph Feck from comment #128) > The filename limit is 255 bytes, not 255 characters. In UTF-8, any non-ASCII > character needs more than 1 byte. Additionally, the last character looks > cropped, causing an illegal UTF-8 name, which Qt does not handle. Yeah, that's the issue I was trying to fix in the script. Trying to find a way to count the BYTES in a variable with bash... as far as that unreadable char, this is what appears in Hexdump nerdopolis@nerdopolis:/tmp/kiocopy/1545096048$ echo -n �| hexdump 0000000 bfef 00bd 0000003 It could be the fact that the char is lopped, since that is not one of the random ones the script creates... ...but it apparently might have triggered the silent failure ...maybe?
That character is just the replacement character for a non-UTF-8 sequence when converting to UTF-16 (which is what QString uses internally). You would need to raw bytes before that conversion.
EDIT: ... need to see the raw bytes ...
(In reply to Christoph Feck from comment #132) > EDIT: ... need to see the raw bytes ... That should be in my "Archive containing the logs, and the tree, and the Seed.txt file to pass to the kiocopy script" attachment: and then /tmp/kiocopy/1545096048/logs/kioclient.log, second to the last line... I tried to make sure I had the right byte copied and pasted, when piping into hexdump
You copied the replacement character, not the invalid raw byte that caused it. You might want to write the list of filenames from your script to a file, and inspect that with Okteta. It might be easier if you use one file per filename, e.g. FILENAME-000001.txt contains 17 bytes, if the first filename is 16 bytes long etc.
I have the full logs, and listings in the mentioned attachment, however the 3 bytes seem to be EF BF BD between the valid lower case 'g' and the '.'
Please make a new bug for a specific issue that we're trying to track.
LANG definitely can make a difference to the way things run. Filesystems don't have a concept of UTF-8 or anything else just a concept of a raw bytestream. We then put that into a QUrl which obviously follows the locale it runs in. Though the worst this should do is (correctly) fail on a file conflict where one didn't previously exist. I think your script is creating a byte array that isn't valid text in any format. I can understand KIO failing to copy that because we don't treat things as a byte stream all the way through. However in your logs KIO isn't silent. Your kioclient log ends with: "Could not make folder /tmp/kiocopy/1545096048/folder_dest/testdir/0l>╚>/1H{n.4🐧N5>2ä<╚#uDJjà57vÉ7BFI*;9jÄ>w@Vdäâ2@Op*8w>i5-by4P{èåhÄÄDw9î1RÆ4x3©h23uÉè)<W?&&,🐧WÄ31Hbv5î,7VTDCrÉï'So🐧D#\"}127z🐧4ij8f®uZ58®pÄJHê®9ÄÅ4Æ-Dj zpN0èg1JB𖡒?Zh{Hom6𖡒B9i1\"èEms2.1.5¾ch@ÅWÆ<äU})2CW1h\t𖡒V9g�." 1 that 1 being the error code So kio did catch the issue and raised an issue. If that was in dolphin etc. that would be an error prompt.
I opened https://bugs.kde.org/show_bug.cgi?id=402298 If I set LANG differently, I DO get error prompts though. and I get more files copied. I'm not running kioclient5 headless, but in a case where it's running under a display, and I CAN get other dialogs... such as, when I forgot to set LANG back, I got actual messages telling me that it can't access those files...