Bug 257907 - File copy operations slow to a stop and bog down computer for large numbers of small files and folders
Summary: File copy operations slow to a stop and bog down computer for large numbers o...
Status: RESOLVED FIXED
Alias: None
Product: kio
Classification: Frameworks and Libraries
Component: file (show other bugs)
Version: 4.5
Platform: openSUSE Linux
: NOR normal
Target Milestone: ---
Assignee: David Faure
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-11-25 20:38 UTC by Todd
Modified: 2011-09-24 03:21 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In: 4.7.2


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Todd 2010-11-25 20:38:04 UTC
Version:           4.5 (using KDE 4.5.3) 
OS:                Linux

When I transfer very large numbers (tens of thousands) of small (few kilobyte) files in a large number of folders (tens of thousands), the transfer slows down to a crawl and the computer becomes essentially unusable.  It always tranfers the folders before the files, and it seems that the more folders it has gone through the slower it goes, to the point that it seems to eventually stop entirely (before any files have copied).  

cp works just fine, it takes a long time but it successfully transfers everything.  It makes the KDE file copy system completely unusable under such situations, I have to use cp or it never moves a single file, even after waiting for a couple of days.

Reproducible: Always
Comment 1 Andreas Hartmetz 2011-09-21 02:20:40 UTC
Git commit 2cd2d1a4cfa1226f959047bd4a0a429382056f19 by Andreas Hartmetz.
Committed on 21/09/2011 at 04:03.
Pushed by ahartmetz into branch 'KDE/4.7'.

Undo manager: Don't prepend() to a QVector which can contain many, many items.

Use a QList instead.
A QStack is a QVector, which has an O(n) prepend method.
Conceptually it is a stack, but not always used like one, so I
haven't renamed the typedefs, so the code looks a bit (more) confusing
now. Sorry about that.
I noticed that copying several 100k files slowed down to just a few
files a second and found that most of the time was spent in
QVector<KIO::BasicOperation>::prepend(), called from
KIO::CommandRecorder::slotCopyingDone().
Another problem is memory usage, which was a few hundred megabytes.
This patch does not fix memory usage because I have a lot of memory
and it's a difficult problem :)

BUG: 257907
BUG: 258497
CCBUG: 280088

M  +9    -9    kio/kio/fileundomanager.cpp
M  +2    -2    kio/kio/fileundomanager_p.h

http://commits.kde.org/kdelibs/2cd2d1a4cfa1226f959047bd4a0a429382056f19