Version: 0.12.90 (using KDE 3.4.90 (alpha1, >= 20050806), compiled sources) Compiler: Target: i586-suse-linux OS: Linux (i686) release 2.6.13-3-default This is an addition to 102932. The symptoms are the same: adding many files to a project is so slow, that adding a checkout of the KDE SVN modules takes almost a hour! Even if now there is a Cancel button, in some cases it simply cannot be used. And adding lots of files causes some kind of invalid loop in some cases that cannot be canceled. It was visible from the progress bar that once it shown that 1.1GB was added and after some time only 900MB and it never (=couldn't wait that long) finished.
Hi, The actual problem is not that it takes too long, but it takes forever. It stuck in KIO::NetAccess, which calls enter_loop for every operations. Due to some (unknown) bad coincidence, sometimes it cannot leave the loop and the application becomes useless. It *might* be that this happens only when dragging the items to the project, that operation also re-enters the event loop. And this also means that the problem is random, sometimes it appears, sometimes it not. But there is a solution. In k3bdataurladdingdialog.cpp, line 156, don't use K3b::filesize(url), but f.size(). You don't need network aware stat here as at this point "url" is a local file! Problem solved. ;-) If you have nothing against it, I will commit. Related to this, I'm wondering if you could get rid of some operations that are not really needed at the time of the addition, thus speeding up the process. You lstat anyway every item, which is very time consuming. Doing it twice, is even worse. The above stat for big files could be removed and the check for big files moved to the K3bFileItem constructor (or that one removed from there). Andras
> use K3b::filesize(url), but f.size(). You don't need network aware stat > here as at this point "url" is a local file! Problem solved. ;-) > If you have nothing against it, I will commit. please don't. the problem is that qt does not support big files. The better solution would be to change K3b::filesize to use system calls instead of KIO. > Related to this, I'm wondering if you could get rid of some operations > that are not really needed at the time of the addition, thus speeding > up the process. You lstat anyway every item, which is very time > consuming. Doing it twice, is even worse. > The above stat for big files could be removed and the check for big > files moved to the K3bFileItem constructor (or that one removed from > there). If it was in the constructor how do you intend to filter the unusable files out?
On Wednesday 09 November 2005 22:02, Sebastian Trueg wrote: > please don't. the problem is that qt does not support big files. The > better solution would be to change K3b::filesize to use system calls > instead of KIO. Ok, I did not know it. But you can just use ::lstat (KIO uses KDE_stat (::stat) and KDE_lstat (::lstat) for local files, so you don't loose anything) , like in the item constructor than. I don't know why do you need K3b::filesize, unless it is used for non-local files as well. Hm, even more strange. You do 4 stats: - twice before creating the item (QFileInfo::isSymLink does a stat and in some cases even two) - in K3bFileItem::K3bFileItem: "if( ::lstat( QFile::encodeName(filePath), &statBuf )" - if lstat fails, you call filesize. Is there a case when ::lstat can fail, but filesize works? > If it was in the constructor how do you intend to filter the unusable > files out? Do it before the constructor. You already do it, so keep the results from there and pass to the constructor. I even would suggest to use KFileItem instead of QFileInfo and the manual ::lstat. It will not solve the slowness completely, but it will help. I've made some test on a 921.4MB big directory with 2135 folders and 20766 files. Konqueror calculates the size of the directory (this includes finding the files sizes as well) in 45 seconds. K3B needs 2 minutes 11 seconds to add the same directory to the project. That's about 3 times that much! The original version with K3b::filesize takes even more: 2:43. Now imagine trying to add a checkout of the KDE sources to a project, which is about twice as big, but has much more files. I stopped Konqueror after 12 minutes. K3b surely needs more than half an hour. Andras
The following patch gets rid of QFileInfo and buffers the result of the first stat and tries to reuse it as much as possible. It has also some other caching operations of the variables. The result is 1:33 minutes for the same test. This is more than 1 minute win over the original and now only about the double of what Konqueror requires. As a bonus it catches and reports some broken links which were missed by the original. The patch optimizes only the K3BFileItem and K3bDataUrlAddingDialog::slotAddUrls(). The rest of the files are only modified so they compile. They should also work, altough in an unoptimized way. And I did not tested as well. I hope you will like this. Andras Created an attachment (id=13374) k3b_speedup.diff
your patch pointed me in the correct direction. thanks.
You are welcome. I cannot wait to see the result. ;-)
I just committed another little improvement through fewer gui updates. Adding kdelibs svn now takes about a minute on my system. still too much, but so far I don't know how to improve. Using dcop is not much faster and I don't think we can get faster than that...
*** This bug has been marked as a duplicate of 76438 ***