Bug 280088

Summary:	KIO::copy or KIO::move too slow if list of files is long
Product:	[Unmaintained] kio	Reporter:	Eduard <karbofos>
Component:	general	Assignee:	David Faure <faure>
Status:	RESOLVED NOT A BUG
Severity:	normal	CC:	ahartmetz, karbofos
Priority:	NOR
Version:	unspecified
Target Milestone:	---
Platform:	Unlisted Binaries
OS:	Linux
Latest Commit:		Version Fixed In:
Sentry Crash Report:

Description Eduard 2011-08-14 20:03:08 UTC

Version:           unspecified (using KDE 4.6.5) 
OS:                Linux

if i work with my files in dolphin or krusader, operations KIO::copy or KIO::move are to slow. my lists of files are sometimes long, so about 500 (today was 7000 and more). files are very different: from cpp source code fies to binaries (archive files to). 

if files a relative small (cpp, h), file operations are very slow, 500kB/s, files bigger (rar, zip, tar.gz, binaries) runs little bit faster, so about 2MB/s..4Mb/s.
For very big files the IO speed is on my HDD better: about 15..20MB/s. I have external USB2.0 SATA device

the second problem with long lists is followed: very high WAIT/IO in top, 30-40%

third problem: during IO operations with long lists free memory will be smaller and if for example move operation stops, then it seems like memory leak, memory usage jump up very fast. 

Reproducible: Always

Steps to Reproduce:
copy or move files from one hdd device to other. the list of files needs to be long, if longer, better to see

Actual Results:  
removing of source code (cpp, h, files up to 100 kB): ~0.5..1MB/s
removing of archives (files from 100 kB to 50MB): 1.5MB/s..4.3MB/s
removing of big files (from 500MB to 1.5GB): 15MB/s

Expected Results:  
IO speed for any files should be similar

Comment 1 Andreas Hartmetz 2011-09-21 02:02:11 UTC

Please don't mix several issues in one report.
The third problem sounds like it might be an actual KDE bug, but there are not enough details. I'm about to fix a bug that slows down processing several tens of thousands of files due a bad algorithm using too much CPU time, so forget about that one if you have it.
Generally though, on traditional harddisks it is perfectly normal that reading many small files is much slower than reading one large file, due to files being scattered on the disk, which requires (slow) disk seeks.

Comment 2 Andreas Hartmetz 2011-09-21 02:20:40 UTC

Git commit 2cd2d1a4cfa1226f959047bd4a0a429382056f19 by Andreas Hartmetz.
Committed on 21/09/2011 at 04:03.
Pushed by ahartmetz into branch 'KDE/4.7'.

Undo manager: Don't prepend() to a QVector which can contain many, many items.

Use a QList instead.
A QStack is a QVector, which has an O(n) prepend method.
Conceptually it is a stack, but not always used like one, so I
haven't renamed the typedefs, so the code looks a bit (more) confusing
now. Sorry about that.
I noticed that copying several 100k files slowed down to just a few
files a second and found that most of the time was spent in
QVector<KIO::BasicOperation>::prepend(), called from
KIO::CommandRecorder::slotCopyingDone().
Another problem is memory usage, which was a few hundred megabytes.
This patch does not fix memory usage because I have a lot of memory
and it's a difficult problem :)

BUG: 257907
BUG: 258497
CCBUG: 280088

M  +9    -9    kio/kio/fileundomanager.cpp
M  +2    -2    kio/kio/fileundomanager_p.h

http://commits.kde.org/kdelibs/2cd2d1a4cfa1226f959047bd4a0a429382056f19