Bug 160227 - CPU at ~100% and memory usage very high when downloading a large file (>4GB)
Summary: CPU at ~100% and memory usage very high when downloading a large file (>4GB)
Status: RESOLVED WAITINGFORINFO
Alias: None
Product: kget
Classification: Applications
Component: Multisegkio (show other bugs)
Version: unspecified
Platform: Ubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: KGet authors
URL:
Keywords:
: 213050 (view as bug list)
Depends on:
Blocks:
 
Reported: 2008-04-01 21:19 UTC by Praveen Srinivasan
Modified: 2010-02-15 21:04 UTC (History)
6 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Praveen Srinivasan 2008-04-01 21:19:57 UTC
Version:            (using KDE 4.0.2)
Installed from:    Ubuntu Packages

Hi,
I was trying to download to large ISO files, e.g.
http://download.opensuse.org/distribution/10.3/iso/dvd/openSUSE-10.3-GM-DVD-x86_64.iso
and http://download.opensuse.org/distribution/10.3/iso/dvd/openSUSE-10.3-GM-DVD-i386.iso

simultaneously with KGet in KDE 4.0.2 in Kubuntu, and the CPU usage was near 100%, and the memory
usage was steadily increasing.
Comment 1 Urs Wolfer 2008-04-01 21:43:04 UTC
I assume you are using KGet from KDE 4.0.2, right?
Comment 2 Praveen Srinivasan 2008-04-01 23:15:34 UTC
Yes, I am.

On 1 Apr 2008 19:43:04 -0000, Urs Wolfer <uwolfer@kde.org> wrote:
[bugs.kde.org quoted mail]
Comment 3 Kolik 2008-04-02 15:48:37 UTC
Don't know whether it's the same bug, but I also get high CPU usage (not 100%, but high) using kde4's kget, though it doesn't depend on the size of file, but on download speed - e.g. when downloading at 10Mbit/s. However, cpu load is coming from kget and kio_file processes. I don't have that kind of problem with old, kde3 kget.
Comment 4 Praveen Srinivasan 2008-04-02 15:56:47 UTC
For comparison, wget has almost 0% CPU usage when downloading the same
files on my machine.
I didn't test with smaller files at high speed, so that may well be it.

-Praveen

On Wed, Apr 2, 2008 at 9:48 AM, Kolik <kolik777@gmail.com> wrote:
[bugs.kde.org quoted mail]
Comment 5 Lukas Appelhans 2008-04-02 16:52:44 UTC
Mmh, 10 MBit/s is very much =)
Anyway, that could be, cause of a function in KDELibs which takes uch cpu =( We have to optimize that for 4.1 :)

Lukas
Comment 6 Praveen Srinivasan 2008-04-02 17:27:34 UTC
In my case, it was downloading at ~2MB/sec (megabytes/sec) (over 100MBit link).
Comment 7 Thomas Kamps 2008-06-07 22:36:23 UTC
Th problem still exists with kde 4.0.4
I am downloading with 1.6MB/s (Megabytes per second)
And i have also ~73% us-load

kio_file uses ~60% and kget ~14%
Comment 8 Lukas Appelhans 2008-07-16 20:15:12 UTC
Can someone test it with the latest KGet from 4.1rc1?

LUkas
Comment 9 Dario Andres 2008-08-01 19:17:13 UTC
Version 2.1.0 Using KDE 4.1.00 (KDE 4.1.0) (KDEmod) in ArchLinux i686:

After 8 minutes of downloading these two files, these values were constant:
"ps uax" output:

dario     5239  5.8  2.7  72344 28660 ?        R    13:59   0:24 /usr/bin/kget -caption KGet -icon kget

While downloading (copying) a file from another computer in my network (using my wifi connection) at 1.4MB per second, these values were obtained ("ps uax"):

At start:

dario     5366  2.4  2.4  78424 25580 ?        S    14:12   0:01 /usr/bin/kget -caption KGet -icon kget

At end:

dario     5366  5.8  2.5  78424 25740 ?        R    14:12   0:05 /usr/bin/kget -caption KGet -icon kget

(the CPU usage was slowly increasing during the transfer)

After the download KGet CPU usage is slowly decreasing
Comment 10 Praveen Srinivasan 2008-08-07 02:37:34 UTC
I still have the bug in KDE 4.1; kget was maxing-out the CPU (there were other things running, so it couldn't get to 100%, but most likely would have).
Comment 11 Marton Drotos 2008-11-01 17:18:14 UTC
I experience this problem mainly when using the torrent plugin. For smaller files (~200MB) everything works fine, but for large files (>=1GB) the CPU usage of KGet is nearly 100% and the memory consumption is constantly growing (after 6 hours it's around 1.5GB, and still growing).

This behavior does not depend on the download/upload speed for me, the resource usage does not decrease even if I pause the download.

System: Debian amd64, unstable/experimental
KGet version: 4.1.2-1 (Debian versioning), 2.1.2 ('about' dialog)
Comment 12 Lukas Appelhans 2008-11-01 18:36:43 UTC
So I think the Torrent-Plugin-Issue is more the fault of libbtcore, which is handling all that stuff for us... can u test if ktorrent has the same problem?

Lukas
Comment 13 Marton Drotos 2008-11-02 20:38:38 UTC
I've installed ktorrent (KDE 4.1.2, ktorrent 3.1.4, Debian unstable/experimental), and I downloaded a 2GB file with it. The CPU- and memory usage of ktorrent was low all the time (max 5-6% CPU, ~100MB mem; the download took about 7 hours).

Interestingly after my last comment, when I restarted KGet to get my resources back, the resource consumption stayed normal. However earlier I've experienced the opposite, too.

From my point of view I can't decide if this is a problem with libbtcore, and I've only seen it using KGet because I've only tested ktorrent once; or it's due to a bug in KGet.

From now on I will use both tools for torrent downloading and I'll keep an eye on the resource usage, and I'll report if I've found anything useful.
Comment 14 Lukas Appelhans 2008-11-02 21:35:52 UTC
Ok great! Hope you find something :)

Lukas
Comment 15 Søren Holm 2008-12-10 23:29:11 UTC
I am having this exact problem when downloading a 21G file over http. Top looks like this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10090 sgh       20   0 79516  25m  19m R   59  2.9  11:12.08 kget
10879 sgh       20   0  103m  73m 3880 S   23  8.3   1:31.59 kio_file
10880 sgh       20   0 39132 9684 6452 S   16  1.1   1:10.36 kio_http


Comment 16 Søren Holm 2008-12-10 23:39:41 UTC
Well ... kio_file also has a huge memory-consumption. That is just crazy - 149 Megs just for fetching a file!

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10090 sgh       20   0 80020  26m  19m R   56  2.9  15:41.85 kget
10879 sgh       20   0  180m 149m 3880 S   23 16.9   3:15.56 kio_file
10880 sgh       20   0 39132 9684 6452 S   17  1.1   2:23.41 kio_http
Comment 17 Tim Burrell 2009-01-30 22:24:18 UTC
With KGet 2.2.0 (KDE 4.2) from Ubuntu PPA packages I still have the extremely high cpu usage problem with KGet.  I've had this same problem with KGet from every version I've tried.

CPU usage is always high even when downloading over a slow 10 Mbit / sec link.  Over gigabit lan it's almost impossible to max out the transfer rate because CPU usage hits 100%.

With wget on the same files the cpu usage is at roughly zero percent.

It's a shame because KGet could be so good!  It seems silly that a file downloader would bring a modern cpu to its knees.
Comment 18 Tim Burrell 2009-01-30 22:27:08 UTC
I should report that while downloading at approx 1MB/sec I see the following cpu usage:
Comment 19 Tim Burrell 2009-01-30 22:30:26 UTC
I should report that while downloading at approx 1MB/sec I see the following cpu usage:
    - kget: 37%
    - plasma: 9%
    - kio_file: 6%
    - kio_http: 6%

With KGet disabled the plasma usage goes way down (I do not have any kget widgets active).
Comment 20 Marton Drotos 2009-02-03 15:10:11 UTC
(In reply to comment #13)
> 
> From now on I will use both tools for torrent downloading and I'll keep an eye
> on the resource usage, and I'll report if I've found anything useful.
> 

In KDE >=4.1.3 kget works fine for me even when a really huge file (8GB) is downloaded via torrent (low CPU, reasonable memory usage), so my problem is solved. It seems that this was a different issue, it was not related to this bug.
Comment 21 Lukas Appelhans 2009-12-01 21:00:28 UTC
btw please retest in latest trunk... there might've been some improvements.

Lukas
Comment 22 Lukas Appelhans 2010-01-02 01:06:37 UTC
*** Bug 213050 has been marked as a duplicate of this bug. ***
Comment 23 Matthias Fuchs 2010-02-15 21:04:38 UTC
Please reopen if this problem still exists in KDE 4.4.