Bug 192625 - kio_http spawns excessive number of kio_http processes
Summary: kio_http spawns excessive number of kio_http processes
Status: RESOLVED FIXED
Alias: None
Product: kio
Classification: Frameworks and Libraries
Component: general (show other bugs)
Version: unspecified
Platform: Compiled Sources All
: NOR normal
Target Milestone: ---
Assignee: David Faure
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-05-14 02:55 UTC by Josh Berry
Modified: 2011-05-01 19:23 UTC (History)
12 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.5


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Josh Berry 2009-05-14 02:55:10 UTC
Version:            (using Devel)
OS:                Linux
Installed from:    Compiled sources

Every few minutes, Plasma seems to make a lot of HTTP requests (I believe to update the Twitter plasmoid).  These requests spawn roughly 20 kio_http processes, which together consume roughly 100MB of RAM.  (I measured this by taking the difference in overall memory usage before and after I killed the processes.)

The processes themselves then sit around, doing nothing, until Twitter refreshes and the next batch of HTTP requests is made.

100MB of RAM and 20 processes, solely for issuing a few HTTP requests, seems excessive to me.  Is there some way we can avoid having this many processes/RAM usage in the first place?

Unfortunately, I don't know exactly where this bug belongs -- I'm surprised KIO even allows this to happen, but this behavior is only consistently reproducible in Plasma.  My apologies if I've misdirected it.
Comment 1 Dario Andres 2009-05-14 15:04:56 UTC
BTW: I have found another way to reproduce:
Open Choqok and press F5.. for every time you press it, 4 kio_http process will appear.. (If you have a short delay for updating the messages, you will get a bunch of this processes in no time). They don't finish nor are closed.
((BTW: choqok never shows the new messages))

So, we have some connection points: kio and twitterApi services (I'm using identi.ca here)

Could this be a connection problem with the microblogging site ?

My versions:

Qt: 4.5.1 (qt-copy  958974)
KDE: 4.2.85 (KDE 4.2.85 (KDE 4.3 Beta1))
kdelibs svn rev. 967040 / kdebase svn rev. 967041
choqok svn rev. 967186
on ArchLinux i686 - Kernel 2.6.29.2
Comment 2 Dario Andres 2009-05-17 03:52:38 UTC
At least in choqok: disabling the secure connection (use http instead of https), only four "kio_http" process will be spawned... When updating the message lists, more kio_http instances won't appear. And Choqok now works OK

Twitter plasmoid also uses secure connection by default

so this is a https error related with kio_http and twitterApi sites
Comment 3 Panagiotis Papadopoulos 2009-05-17 13:59:56 UTC
I did some testing with the microblogging plasmoid:
I removed the plasmoid, stopped all kio_http instances (~23), then added the plasmoid again. The next step I did was to change the identi.ca service URL from https: to http:, I entered my user credentials and clicked on apply.

The plasmoid started "fetching" data, the KDE system monitor showed me 3 instances in the beginning, but quickly rose to the amount of 21 instances again.

After 30 minutes it still has 21 instances, although I set the update timer to 2 minutes.
Comment 4 illogic-al 2009-05-18 16:13:15 UTC
Changed OSes to all as I see this on OS X. Also switched component to kio.
And confirming as new.
Comment 5 Marco Martin 2009-06-02 20:01:26 UTC
SVN commit 976814 by mart:

explicitly limit the number of http connection here, but this is just a
temporaty workaround to bug 192625, to be reverted as soon as it's
fixed.

BUG: 194345
CCBUG: 192625


 M  +22 -6     imagesource.cpp  
 M  +2 -0      imagesource.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=976814
Comment 6 thomas coopman 2009-12-08 09:06:51 UTC
I saw this behaviour with choqok just now, when I started choqok.  It looked like none of my accounts were remembered so I added them again.  After that, the add account dialog wouldn't respond and a huge number off kio_http processes were started, so much that I couldn't start a new program anymore (I got this when I started something new: "The system may have reached the maximum number of open files possible or the maximum number of open files that you are allowed to use has been reached.").  Some time later choqok crashed and all processes were gone. After restarting choqok everything seems fine now (choqok also found my accounts that I had already configured).
Comment 7 Dawit Alemayehu 2009-12-12 00:58:12 UTC
Simply put this is not a kio_http problem but rather a design issue in the KIO::Scheduler, which is respobsible for scheduling all jobs. If you had used ftp, sftp or any other remote protocol to retreive the resource the same way, you would encounter the same issue. 

The way KIO::Scheduler is designed applications that want to queue jobs so that there isn't so many of them running at the same time, will have to implicitly call  KIO::Scheduler::scheduleJob(...) after a call to KIO::get/post/... Unfortunately scheduleJob(...) itself has issues that are pending to be resolved with a patch that is currently up for a review. See http://reviewboard.kde.org/r/2323/ for all the gory details associated with this topic.

In other words the problem is not limited to kio_http. Therefore, I have changed the component to 'general'.
Comment 8 Dario Andres 2010-02-03 02:46:15 UTC
I wonder why this wasn't reassigned to the right people.
BTW, there is ungoing work to implement a new KIO scheduler. (4.5)
Comment 9 Andreas Hartmetz 2010-02-03 16:42:00 UTC
The new scheduler is in kdelibs now. It always limits the number of ioslaves (currently 5 per server and 30 total per application), so choqok cannot create extremely many ioslaves anymore - in the given scenario probably all requests go to the same host, so only 5 ioslaves could be created.
Aside:
When debugging a problem in the new scheduler I looked into choqok's code, and there are a few things that could be improved regarding its use of KIO. I'll talk about it with Mehrdad when I catch him on IRC.
Comment 10 Dawit Alemayehu 2011-05-01 19:23:16 UTC
Based on comment #9, let us mark this as resolved in KDE 4.5 or higher. If the problem still persists for you, feel free to reopen this ticket.