Bug 203277 - can't move large directories
Summary: can't move large directories
Status: RESOLVED DUPLICATE of bug 342056
Alias: None
Product: kio
Classification: Frameworks and Libraries
Component: general (show other bugs)
Version: unspecified
Platform: Fedora RPMs Linux
: NOR normal
Target Milestone: ---
Assignee: David Faure
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-08-10 09:41 UTC by kavol
Modified: 2018-04-25 21:58 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description kavol 2009-08-10 09:41:06 UTC
Version:           2.0.0 (using KDE 4.3.0)
OS:                Linux
Installed from:    Fedora RPMs

I needed to move a large backup directory containing about 450000 files. I had the stupid idea to try that in Krusader ... After more than hour of disk activity, Krusader had allocated over 1.5 GiB of memory, yet without moving a single file. Then I left the computer, and coming back after the weekend, I was greeted by two error message boxes "Proces pro protokol file neočekávaně skončil." (Process for the protocol file has exited unexpectly - or something like that.) Needless to say, no files were moved ...
Comment 1 Marcin Gryszkalis 2011-03-05 12:29:40 UTC
Did you try to move the directory itself or all files (like: enter directory, select all, move)?
Comment 2 kavol 2011-03-05 15:38:50 UTC
(In reply to comment #1)
> Did you try to move the directory itself or all files (like: enter directory,
> select all, move)?

all files

I've just retried with krusader-2.3.0-0.1.beta1.fc14.x86_64

at first, I tried with 100000 files - everything seemed okay (except that moving the files took about two minutes, while Midnight Commander could do the same in about 15 seconds)

then I added another 400000 files, half a million in total - trying to move the files in Krusader then led to my computer being very unresponsive, the memory consumption of Krusader grew up to 1.3 GiB before I've managed to kill it

but in this case, unlike the original report, some files were moved
Comment 3 Jan Lepper 2013-03-20 09:24:30 UTC
I belive that's a bug in kio.
Comment 4 Dawit Alemayehu 2013-06-16 23:36:05 UTC
Moves should be much much faster in KIO nowadays. Do you still see this issue with the latest version, v4.10 or higher?
Comment 5 kavol 2013-08-18 17:47:21 UTC
(In reply to comment #4)
> Moves should be much much faster in KIO nowadays. Do you still see this
> issue with the latest version, v4.10 or higher?

so, I tried to re-test with krusader-2.4.0-0.9.beta3.fc19.x86_64 and kdelibs-4.10.5-1.fc19.x86_64

I've prepared the testing directory like this:

$ time for i in `seq 500000` ; do touch $i ; done

real    28m56.211s
user    2m36.167s
sys     23m49.398s

then I've opened krusader, went to that directory, chosen all, switched to target directory in the second panel and started the move ...

it took all the cpu (just 1 core, to be exact) and there were big troubles re-drawing Krusader window and the progress box

during the operation, top said krusader has VIRT about 2G and RES peaked at 1.483g

after finishing, the memory didn't seem to be freed, RES still taking 1.431g

and it took 5 and 1/2 hours ... not exactly what I'd expect if the file creation using a simple scripting language with calls to external programs took less than half an hour (not to mention the cpu being used by Krusader and not the system, exactly opposite to the file creation process as shown above)
Comment 6 Dawit Alemayehu 2013-08-28 13:15:51 UTC
(In reply to comment #5)
> (In reply to comment #4)
> > Moves should be much much faster in KIO nowadays. Do you still see this
> > issue with the latest version, v4.10 or higher?
> 
> so, I tried to re-test with krusader-2.4.0-0.9.beta3.fc19.x86_64 and
> kdelibs-4.10.5-1.fc19.x86_64
> 
> I've prepared the testing directory like this:
> 
> $ time for i in `seq 500000` ; do touch $i ; done
> 
> real    28m56.211s
> user    2m36.167s
> sys     23m49.398s
> 
> then I've opened krusader, went to that directory, chosen all, switched to
> target directory in the second panel and started the move ...
> 
> it took all the cpu (just 1 core, to be exact) and there were big troubles
> re-drawing Krusader window and the progress box
> 
> during the operation, top said krusader has VIRT about 2G and RES peaked at
> 1.483g

Sorry but what top says is meaningless. You should look at what ksysguard (CTRL+ESC) tells you instead which correctly reflects the memory usage and even provides a detailed information if you right click on the desired process.

> after finishing, the memory didn't seem to be freed, RES still taking 1.431g
> 
> and it took 5 and 1/2 hours ... not exactly what I'd expect if the file
> creation using a simple scripting language with calls to external programs
> took less than half an hour (not to mention the cpu being used by Krusader
> and not the system, exactly opposite to the file creation process as shown
> above)

As a practical matter, that is simply a horrible test. You will never have that many zero sized files in any folder in real life. And filesystems are notoriously bad at dealing with such small files to begin with. Anyhow, to provide the correct context you should provide the time it takes to copy those same files from the command line ; not how long it took to create the files from the command line.
Comment 7 Nate Graham 2018-04-25 21:58:21 UTC
Getting better all the time...

*** This bug has been marked as a duplicate of bug 342056 ***