Bug 43356 - deleting files in directories with more than 3000 files gets painfully slow
Summary: deleting files in directories with more than 3000 files gets painfully slow
Status: RESOLVED FIXED
Alias: None
Product: kio
Classification: Unclassified
Component: general (show other bugs)
Version: 2.0
Platform: openSUSE RPMs Linux
: NOR normal with 250 votes (vote)
Target Milestone: ---
Assignee: David Faure
URL:
Keywords:
: 93156 112877 150885 150887 170062 (view as bug list)
Depends on:
Blocks:
 
Reported: 2002-06-01 19:48 UTC by Janne Karhunen
Modified: 2008-11-30 18:48 UTC (History)
6 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
konqui_test.sh - test case (156 bytes, text/plain)
2004-11-12 11:58 UTC, Waldo Bastian
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Janne Karhunen 2002-06-01 19:38:35 UTC
(*** This bug was imported into bugs.kde.org ***)

Package:           konqueror
Version:           2.0 (kfmclient) (using KDE 3.0.1 )
Severity:          wishlist
Installed from:    SuSE RPMs
Compiler:          gcc-2.95.3-216
OS:                Linux
OS/Compiler notes: SuSE 8.0; absolutely latest KDE 3.0.1 packages

Directories that contain few thousand files get kfmclient almost hang (without previes or anything; just plain file listing).  Approx. 3000 files seems to be some mystical limit after which kfmclient is pretty much useless. Example: try to delete 5000 files in a single directory. It starts by deleting files one by one which takes ages. After it has deleted them down to ~3000 it's a blaze. 

Oh machine should not a problem it's 'fast enough'. Plus system configuration is fine as well (disk IO is as expected).

(Submitted via bugs.kde.org)
Comment 1 Janne Karhunen 2002-10-26 18:12:36 UTC
Seems to work now ( 3.1 head as of 23.10.2002 ). 
Comment 2 Dik Takken 2003-07-06 20:06:13 UTC
Deleting a directory tree with tens of thousands of files still is slow, because konqueror 
builds a list of all files that have to be deleted. Try downloading all packages of KDE 
3.1 into the same container directory, extract them all, and delete the container 
directory. Takes almost a full minute to complete, while "rm -f -R" does the trick in 
about 3 seconds. 
 
This problem may be related to bug 41305. 
Comment 3 Dik Takken 2003-07-06 20:07:16 UTC
Forgot to mention: I use KDE 3.1.1 now. 
Comment 4 Maksim Orlovich 2003-07-06 20:14:13 UTC
Are you using the iconview or the listview? If the former, I suggeste trying the 
latter. 
 
Comment 5 Waldo Bastian 2004-11-12 11:53:22 UTC
Still an issue.
Comment 6 Waldo Bastian 2004-11-12 11:58:10 UTC
Created attachment 8262 [details]
konqui_test.sh - test case

Attached script creates 5000 files in a temp directory.
Select all in Konqueror and delete the files. Updating the iconview or the
listview takes way too much time.

The slowness is caused among others by kdirlister using O^2 algorithms in
KDirListerCache::FilesRemoved.

See also bug 90473 

Note that if the progress stops at 3900 files for you with the attached test
script then you need to update kdirwatcher.cpp to CVS HEAD.
Comment 7 Stephan Kulow 2004-11-12 12:08:15 UTC
*** Bug 93156 has been marked as a duplicate of this bug. ***
Comment 8 Maksim Orlovich 2004-11-12 15:32:36 UTC
QIconView also has some quadratic behavior when removing lots of icons, if my memory serves me right (and it's hard to fix, since the data structures are screwey; it may be something that we can only fix for KDE4, if Qt4's iconview is better, not sure if it is)

Comment 9 Thiago Macieira 2005-09-19 12:34:20 UTC
*** Bug 112877 has been marked as a duplicate of this bug. ***
Comment 10 Tommi Tervo 2007-10-16 13:03:52 UTC
*** Bug 150887 has been marked as a duplicate of this bug. ***
Comment 11 Jaime Torres 2008-06-10 16:54:39 UTC
*** Bug 150885 has been marked as a duplicate of this bug. ***
Comment 12 David Faure 2008-09-17 14:28:53 UTC
I just committed a 25-times speedup for the case of deleting 1000 local files from the same directory, see Bug 150885 for details. This fix will be in KDE-4.2 (I might backport it for 4.1.2 if nobody detects regressions that my unit test didn't catch).
Comment 13 David Faure 2008-09-17 15:41:33 UTC
The GUI part is still slow though - this hits a performance issue in KDirModel. Working on this now.
Comment 14 Maciej Pilichowski 2008-09-17 15:56:44 UTC
Just a reminder -- in k3b (AFAIR) it helped to check if there is a difference in progress units -- if not, do not change anything in progress bar. But this my not be relevant in this case.
Comment 15 David Faure 2008-09-18 12:02:04 UTC
Fixed, finally, r862165.

Deleting 5000 files with konqueror-4.1 took 8 minutes (30 seconds stat'ing, 5 seconds deleting, and then 7'25" until the view refreshes).

Comment #12 was about speeding up the stat'ing part, but that was nothing compared to the long time updating the view (as Waldo had noted in kde3 too).

With model/view we can finally update the view in a better way: by chunks instead of item-after-item.

Deleting 5000 files with konqueror-trunk now takes only 15 seconds (2 seconds stat+deleting, 13 seconds until the view refreshes).

Don't worry about the progressbar, it has no time to even appear in the 2 seconds that the kio job takes :)
Comment 16 Maciej Pilichowski 2008-09-18 13:54:22 UTC
Absolute great speed-up! You are great :-)

Btw. stat+deleting I guess means counting + actual deleting. But what this 
"the view refreshes" means (i.e. what view?).
Comment 17 David Faure 2008-09-18 14:44:02 UTC
The view is the dolphin or konqueror view, where you see those files, and which has to be updated whenever files are created or deleted. The graphical part of this all.
Comment 18 Maciej Pilichowski 2008-09-18 17:35:52 UTC
Does refreshing also take place _during_ deletion? If yes, maybe it could be "frozen"? I am asking because one can say 12 seconds is not big deal, but if comparing to actual I/O it is 6 times slower. 
Comment 19 David Faure 2008-09-18 17:51:37 UTC
Refreshing happens after the actual deletion. Otherwise it would be chunked, and that means slower in this case (more signalling, more computing etc). I have no idea what you mean by frozen. The refreshing is about removing 5000 rows in one go in a Qt itemview, now that it's done in one operation, I can hardly make it faster. Yes it takes 6 times the time of the actual deletion (for 5000 items). GUIs obviously have a price compared to command-line tools, we have GUI stuff to handle in addition to the actual files :). But this is nitpicking. You don't delete 5000 files every day, and 15 seconds isn't that much for so many files.
Comment 20 Peter Penz 2008-11-30 18:48:42 UTC
*** Bug 170062 has been marked as a duplicate of this bug. ***