Bug 244215

Summary: Downloading big files causes crash by filling up temp directory...
Product: [Frameworks and Libraries] kio Reporter: gurkan gur <seqizz>
Component: httpAssignee: kdelibs bugs <kdelibs-bugs>
Severity: major CC: adawit, lukas.karas, mat69, web
Priority: HI    
Version: unspecified   
Target Milestone: ---   
Platform: Gentoo Packages   
OS: Linux   
Latest Commit: Version Fixed In: 4.6

Description gurkan gur 2010-07-11 13:21:09 UTC
Version:           unspecified (using KDE 4.4.4) 
OS:                Linux

Kget in kde 4.4.5 here (wasn't on the list).

When i try to download a big file (e.g >4gb) to an external drive (usb hdd), before kget finishes the download, it's starting to download under /var/tmp/kdecache-<user>/http folder i guess. So my root (/) folder has not enough space, almost everything is starting to crash because of insufficient space, includes kde. If i can stop downloading before the root folder become full, the http folder's size is not increasing.

(e.g. For now, problematic downloads are from ftp. I didn't see this when i download via torrent, yet.)

Reproducible: Sometimes

Expected Results:  
Download to specified location from start to end?

I don't know what kind of debug info you need. ~amd64 gentoo. Activated plugins: Bittorrent, checksum search, KIO, Metalink, Mirror search, Multi segment KIO.

CFLAGS="-march=native -O3 -fomit-frame-pointer -pipe"
Comment 1 Matthias Fuchs 2010-07-11 16:54:28 UTC
How fast was the connection in your case, especially in comparison with the usb-speed?
And also how much free space was on root?
Comment 2 gurkan gur 2010-07-11 17:17:00 UTC
8Mbit (downloads ~840 kb per second). I don't think usb is slower than this (seems like using ehci_hcd), but probably when the problem occurs, Kget just stops to transfer files to usb, because tmp directory is getting bigger almost equal to download speed. Root directory has ~2.8gb free space.
Comment 3 Matthias Fuchs 2010-07-12 21:13:22 UTC
I haven't tried to reproduce with a usb hdd yet but it worked without problems for normal hdds.

Could you please post what filesystem you used and also the speed of copying files to the the usb hdd could maybe be useful?
Comment 4 gurkan gur 2010-07-12 22:21:28 UTC
reiserfs on usb hdd (not 4), hdparm results:

> for i in 1 2 3; do hdparm -tT /dev/sdb3; done                                      

 Timing cached reads:   3566 MB in  2.00 seconds = 1784.50 MB/sec
 Timing buffered disk reads:  100 MB in  3.04 seconds =  32.92 MB/sec

 Timing cached reads:   5990 MB in  2.00 seconds = 2999.03 MB/sec
 Timing buffered disk reads:  100 MB in  3.02 seconds =  33.07 MB/sec

 Timing cached reads:   3952 MB in  2.00 seconds = 1977.87 MB/sec
 Timing buffered disk reads:   96 MB in  3.01 seconds =  31.93 MB/sec

(probably irrelevant, but ext4 on root)

I've expanded my root filesystem, so i've lots of space now. I'm trying to reproduce also, cause i wonder what will kget do after finishing to download under /var/tmp.
Comment 5 Matthias Fuchs 2010-07-15 19:37:33 UTC
Have you changed the number of segments in the multiseg kio config?
Comment 6 Matthias Fuchs 2010-07-15 19:57:00 UTC
I think this might be a bug in the caching of KIO.
Comment 7 gurkan gur 2010-07-15 20:28:08 UTC
I didn't touch any plugins yet, they are in default position, at least for Gentoo. Multisegment KIO is set to "5". Will try "3". (Guessing decreasing will help)?
Comment 8 Dawit Alemayehu 2011-01-14 18:12:30 UTC
*** Bug 262906 has been marked as a duplicate of this bug. ***
Comment 9 Dawit Alemayehu 2011-01-17 02:23:34 UTC
Increased the severity of the bug because of the details in the similar report marked duplicate of this one.
Comment 10 Dawit Alemayehu 2011-01-23 07:58:18 UTC
*** Bug 263872 has been marked as a duplicate of this bug. ***
Comment 11 Dawit Alemayehu 2011-01-23 09:18:24 UTC
SVN commit 1216412 by adawit:

Don't cache files that are bigger than the maximum cache size. 

NOTE: this fix only works for cases where the content size is known and
as such further improvements are needed in the future. Until then this 
should fix all the problems reported in BR 244215.

CCBUG: 244215

 M  +14 -3     http.cpp  

WebSVN link: http://websvn.kde.org/?view=rev&revision=1216412
Comment 12 Matthias Fuchs 2011-01-23 18:25:54 UTC
Could you please also backport this to the 4.6 branch?
Comment 13 Dawit Alemayehu 2011-01-24 06:18:59 UTC
SVN commit 1216579 by adawit:

Backport the fix for BR#244215.


 M  +14 -3     http.cpp  

WebSVN link: http://websvn.kde.org/?view=rev&revision=1216579