Bug 112546

Hi,

 The actual problem is not that it takes too long, but it takes forever. 
It stuck in KIO::NetAccess, which calls enter_loop for every 
operations. Due to some (unknown) bad coincidence, sometimes it cannot 
leave the loop and the application becomes useless. It *might* be that 
this happens only when dragging the items to the project, that 
operation also re-enters the event loop. And this also means that the 
problem is random, sometimes it appears, sometimes it not.
But there is a solution. In k3bdataurladdingdialog.cpp, line 156, don't 
use K3b::filesize(url), but f.size(). You don't need network aware stat 
here as at this point "url" is a local file! Problem solved. ;-)
If you have nothing against it, I will commit.

Related to this, I'm wondering if you could get rid of some operations 
that are not really needed at the time of the addition, thus speeding 
up the process. You lstat anyway every item, which is very time 
consuming. Doing it twice, is even worse.
 The above stat for big files could be removed and the check for big 
files moved to the K3bFileItem constructor (or that one removed from 
there).

Andras

> use K3b::filesize(url), but f.size(). You don't need network aware stat
> here as at this point "url" is a local file! Problem solved. ;-)
> If you have nothing against it, I will commit.


please don't. the problem is that qt does not support big files. The better 
solution would be to change K3b::filesize to use system calls instead of KIO.

> Related to this, I'm wondering if you could get rid of some operations
> that are not really needed at the time of the addition, thus speeding
> up the process. You lstat anyway every item, which is very time
> consuming. Doing it twice, is even worse.
>  The above stat for big files could be removed and the check for big
> files moved to the K3bFileItem constructor (or that one removed from
> there).


If it was in the constructor how do you intend to filter the unusable files 
out?

On Wednesday 09 November 2005 22:02, Sebastian Trueg wrote:
> please don't. the problem is that qt does not support big files. The
> better solution would be to change K3b::filesize to use system calls
> instead of KIO.

Ok, I did not know it. But you can just use ::lstat (KIO uses KDE_stat 
(::stat) and KDE_lstat (::lstat) for local files, so you don't loose 
anything) , like in the item constructor than. I don't know why do you 
need  K3b::filesize, unless it is used for non-local files as well. Hm, 
even more strange. You do 4 stats:
- twice before creating the item (QFileInfo::isSymLink does a stat and 
in some cases even two)
- in K3bFileItem::K3bFileItem: 
"if( ::lstat( QFile::encodeName(filePath), &statBuf )"
- if lstat fails, you call filesize. 

Is there a case when ::lstat can fail, but filesize works?

> If it was in the constructor how do you intend to filter the unusable
> files out?

Do it before the constructor. You already do it, so keep the results 
from there and pass to the constructor. I even would suggest to use 
KFileItem instead of QFileInfo and the manual ::lstat.
 It will not solve the slowness completely, but it will help.
 I've made some test on a 921.4MB big directory with 2135 folders and 
20766 files.  Konqueror calculates the size of the directory (this 
includes finding the files sizes as well) in  45 seconds. K3B needs 2 
minutes 11 seconds to add the same directory to the project. That's 
about 3 times that much! The original version with K3b::filesize takes 
even more: 2:43.
 Now imagine trying to add a checkout of the KDE sources to a project, 
which is about twice as big, but has much more files. I stopped 
Konqueror after 12 minutes. K3b surely needs more than half an hour.


Andras

The following patch gets rid of QFileInfo and buffers the result of the 
first stat and tries to reuse it as much as possible. It has also some 
other caching operations of the variables. The result is 1:33 minutes 
for the same test. This is more than 1 minute win over the original and 
now only about the double of what Konqueror requires.
As a bonus it catches and reports some broken links which were missed by 
the original.
The patch optimizes only the K3BFileItem and 
K3bDataUrlAddingDialog::slotAddUrls(). The rest of the files are only 
modified so they compile. They should also work, altough in an 
unoptimized way. And I did not tested as well.
I hope you will like this.

Andras


Created an attachment (id=13374)
k3b_speedup.diff

your patch pointed me in the correct direction. thanks.

You are welcome. I cannot wait to see the result. ;-)

I just committed another little improvement through fewer gui updates. Adding kdelibs svn now takes about a minute on my system. still too much, but so far I don't know how to improve. Using dcop is not much faster and I don't think we can get faster than that...

*** This bug has been marked as a duplicate of 76438 ***