Bug 57248 - make KGets handling of many downloads smarter
Summary: make KGets handling of many downloads smarter
Status: CONFIRMED
Alias: None
Product: kget
Classification: Applications
Component: general (show other bugs)
Version: unspecified
Platform: Gentoo Packages Linux
: NOR wishlist
Target Milestone: ---
Assignee: KGet authors
URL:
Keywords:
: 63461 83967 (view as bug list)
Depends on:
Blocks:
 
Reported: 2003-04-15 07:50 UTC by wilbert
Modified: 2010-08-27 13:29 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
amarok 1.3.1 with 3 queued tracks (190.54 KB, image/jpeg)
2005-09-21 03:14 UTC, Damir Perisa
Details
amarok 1.3.1: info about queued tracks in status line (167.38 KB, image/jpeg)
2005-09-21 03:16 UTC, Damir Perisa
Details
amarok 1.3.1: queue/dequeue tracks (221.10 KB, image/jpeg)
2005-09-21 03:23 UTC, Damir Perisa
Details

Note You need to log in before you can comment on or make changes to this bug.
Description wilbert 2003-04-15 07:50:10 UTC
Version:           v0.8.3 (using KDE 3.1.1)
Installed from:    Gentoo
Compiler:          gcc version 3.2.1
OS:          Linux (i686) release 2.4.20

I once dragged the full contents (around 200 files) of a ftp directory to KGet from Konqueror.

KGet tried to connect 200 times to the same ftp server, and I had to click 200 times the OK button of the "Could not connect to ftp server" error dialog. :-)

It would be nice if KGet supported a maximum number of connections _per_server_, so that it (e.g.) never would open more than 2 connections to the same FTP server. (Like Downloader for X does.)

Also, it would be nice if KGet would automatically try to reconnect after some time when connections could not be made, instead of dequeuing them. Now I have to manually queue the download operations, 2 by 2, to not upset the ftp server.

Finally, it would be nice if KGet could somehow determine the bandwidth it is using, and have some download speed control possible (like slow, normal, max speed).

Other than that: thanks for a really great program, that fits very well into KDE!
Comment 1 Carsten Pfeiffer 2003-04-15 12:15:34 UTC
Subject: Re:  New: make KGets handling of many downloads smarter

On Tuesday 15 April 2003 07:50, you wrote:

> I once dragged the full contents (around 200 files) of a ftp directory to
> KGet from Konqueror.
>
> KGet tried to connect 200 times to the same ftp server, and I had to click
> 200 times the OK button of the "Could not connect to ftp server" error
> dialog. :-)

Oups :} Well, KGet actually has a connection-limit "Maximum number of opened 
connections" in the configuration dialog, the "Limits" page. Maybe your ftp 
server didn't let you in at all? Then the first transfer displays the error 
dialog, the next transfer is started, shows the error dialog again and so on. 
That could be improved somehow ;)

> It would be nice if KGet supported a maximum number of connections
> _per_server_, so that it (e.g.) never would open more than 2 connections to
> the same FTP server. (Like Downloader for X does.)

Hmm, would that help here? Why is one max limit for all servers not 
sufficient?

> Also, it would be nice if KGet would automatically try to reconnect after
> some time when connections could not be made, instead of dequeuing them.
> Now I have to manually queue the download operations, 2 by 2, to not upset
> the ftp server.

Indeed.

> Finally, it would be nice if KGet could somehow determine the bandwidth it
> is using, and have some download speed control possible (like slow, normal,
> max speed).

Yes, that's unfortunately not easily possible. I'll see if I find a solution 
for that when I'm having more free time.

> Other than that: thanks for a really great program, that fits very well
> into KDE!

I'm glad you enjoy it.

Cheers
Carsten Pfeiffer
-----BEGIN PGP SIGNATURE-----

iQEVAwUBPpvbUaWgYMJuwmZtAQGQngf+LTjp38yHMKArypyXvFy0hc9cRRzGr0Ri
bVxVW/h1Vtt/PPkKVxgRtiVXREAJZ4taD8WhUGJuCLnRcxC3CHdahR+Qzmci7AKR
aWksMZ8Jw5NnTMfqqBqbATrRLJA8wVL/lYPZEBRAdpfxOUCV3GRQHmHZJCkb494C
oRKShkyUCfAikQjyWOJdrcXuVCSxR43ggqr8N4/RKvP+V6FzxGeZJcLxRhOcO+fJ
YBZZIpkA5yr1dX7INtqz2QDjr0FZlg1DOh3WPRrhY8qA4PoHWL5+Mmsxd4LUrke1
xnSBigRiJHoWNunk+g4+gXIfno9032EvFV+u2Chp6eobAvTZgL5WFw==
=RFVr
-----END PGP SIGNATURE-----

Comment 2 wilbert 2003-04-15 13:37:23 UTC
Subject: Re:  make KGets handling of many downloads smarter

On Tuesday 15 April 2003 12:15, Carsten Pfeiffer wrote:
>
> On Tuesday 15 April 2003 07:50, you wrote:
> > I once dragged the full contents (around 200 files) of a ftp
> > directory to KGet from Konqueror.
> >
> > KGet tried to connect 200 times to the same ftp server, and I had to
> > click 200 times the OK button of the "Could not connect to ftp
> > server" error dialog. :-)
>
> Oups :} Well, KGet actually has a connection-limit "Maximum number of
> opened connections" in the configuration dialog, the "Limits" page.
> Maybe your ftp server didn't let you in at all? Then the first transfer
> displays the error dialog, the next transfer is started, shows the
> error dialog again and so on. That could be improved somehow ;)

the ftp server in question (ftp.hornet.org) only accepted 2 connections 
per ip-address.

> > It would be nice if KGet supported a maximum number of connections
> > _per_server_, so that it (e.g.) never would open more than 2
> > connections to the same FTP server. (Like Downloader for X does.)
>
> Hmm, would that help here? Why is one max limit for all servers not
> sufficient?

Well, many ftp servers do not accept more than a few connections per 
ip-adres, while one could easily have some other long-going downloads 
(iso's etc.) from other servers.

So I think both options make a lot of sense: max connections per server, 
and total max connections.

> > Also, it would be nice if KGet would automatically try to reconnect
> > after some time when connections could not be made, instead of
> > dequeuing them. Now I have to manually queue the download operations,
> > 2 by 2, to not upset the ftp server.
>
> Indeed.

best regards,
Wilbert

Comment 3 w7aggag 2003-12-20 02:56:07 UTC
Please add the feature of multi source download for a particular file

So let
Comment 4 Thibaut Cousin 2003-12-20 08:57:00 UTC
I'm not sure I agree with that last proposal. Overloading several servers at the same time for only one download is a very bad habit that many webmasters hate, and it doesn't help with the internet's global charge...

I suggest we stay within the bounds of internet politeness; KGet could use multiple sources only in a source becomes unavailable: then KGet could try the second source, or the third, and so on.

Thanks for your attention! 
Comment 5 Alexander Wigen 2004-09-03 15:18:59 UTC
I would also like to see a "Max downloads per host" selector in kget's limit section, not just for ftp connections though.
Comment 6 Iges Seem 2005-03-25 20:56:30 UTC
My wish is a bit similar mentioned above, has also todo with downloading multiple files from a ftp with connection limit. Just to clarify a bit:

my ISP provides an ftp server to swap files, but only one connection per ip is allowed, and the queue mode does not work properly with this. It seems that a successful connection is neseccary to queue a transfer, but the connection cannot be established because a transfer is already active and ftp has one connection per ip limit. So the transfers following the first one in list are delayed.

Would it be possible to change the queue mode so it would not require a connection check to queue a transfer or provide a "next in list" mode - meaning that if the current transfer has ended (download completed) simply the next transfer in list is activated.

Best regards.
Comment 7 Iges Seem 2005-03-26 11:46:29 UTC
yay, its already possible ;) Thanks for connection limit, damn need more sleep...
Comment 8 Kemal Nurveren 2005-05-11 10:37:27 UTC
Yes, I agree with the reporter who has sent this wish.
Comment 9 Urs Wolfer 2005-09-20 15:35:07 UTC
*** Bug 63461 has been marked as a duplicate of this bug. ***
Comment 10 Urs Wolfer 2005-09-20 16:49:43 UTC
*** Bug 83967 has been marked as a duplicate of this bug. ***
Comment 11 Dan 2005-09-21 02:35:29 UTC
It's been almost 2.5 years since this bug was opened, and it looks like a lot of people would like it.  While KGet is great most of the time, for cases where you have download limits on an FTP site, or are connected to a very slow site, and also want to download files from other unrelated sites simultaneously, this proposed feature is absolutely necessary.  Please read the text for bug 63461 (marked as a dupe of this bug) for a good description of why this feature is needed.

The other features listed in this report would also be nice.

Also, please check out the Downloader for X program previously mentioned.  I used to use this before KGet came along, and its features were excellent.  Unfortunately, it doesn't integrate well at all with KDE; emulating some of the advanced features and operation of that program in KGet would be ideal.
Comment 12 Damir Perisa 2005-09-21 02:48:49 UTC
> I had to click 200 times the OK button of the "Could not connect to ftp server" error dialog

i was also once in such a situation, when my modem disconnected and i was trying to download around 40 small text files from a friend. the error dialogue is horribly primitive for this modern time of passive notification boxes (e.g. kmail can be configured to notify new incomming emails in a passive box that appears and disappears after 2 seconds). if something happened and the user do not need to take a decision (e.g. [continure] or [cancel] transfer), then there is no point in having active alert windows that need to be "OK"'ed. (we are not MS windows, eh? :D )
Comment 13 Damir Perisa 2005-09-21 03:12:03 UTC
some more thoughs about many downloads behaviour...

let's assume we are having a list of 15 downloads in kget:

- 3 iso (knoppix, archie, and another iso) - 3 x ~610MB - all from different servers

- 10 big pdf files from university - 10 x 40MB - server accepting only 1 connection/client

- 2 spontanous downloads of some source-tarballs from a project on the internet - 2 x 4MB

all this items are in kget in the order as described (isos first)

the behaviour atm is that kget will can be limited by total open connections only. this means either 1 connection running :-( or all connections running :-(  ... or you start/stop downloads by hand :-((

what can be done to make this behaviour more intelligent? 
- we need a max open connections per server limit (this we all agree, right?)

is this enough? probably yes ... but from another application i know another very usefull feature that may make kget more attractive. you know amarok? the new multimedia player for kde --- with one feature that is really cool: "queue track"

how this applies to kget?

you have a list of items. if an item is queued, it will be played/downloaded as next, no matter where it is in the list (imagine huge lists of 200 items!). this way you can have your limits set but if you want 5 files from the middle of the list to be downloaded immediately after the current one, you do not need to move them up in list (you cannot move more than one item at once atm anyway) or start them manually or cancel the downloads in between... no! simply do queue the 5 items you want to be downloaded after the current downloads.  --- and you can dequeue them if you don't need them any more that fast

i would really like to see such an interface in download list behaviour as it exists in amarok play list behaviour 

i will attach 3 screenshoots for people who do not know amarok (you really miss a lot if you use another player for kde, anyway ;-) )
Comment 14 Damir Perisa 2005-09-21 03:14:57 UTC
Created attachment 12637 [details]
amarok 1.3.1 with 3 queued tracks

the tracks will be played in following order:

- current track finishes
- Hotel California
- Bonjour Hellville
- Samb-Adagio
- Denkmal
- Wenn du Schläfst
 ...
Comment 15 Damir Perisa 2005-09-21 03:16:36 UTC
Created attachment 12638 [details]
amarok 1.3.1: info about queued tracks in status line

small icon with number indicates, how many tracks/downloads are queued ...
right click opens list and option to dequeue all of them
Comment 16 Damir Perisa 2005-09-21 03:23:40 UTC
Created attachment 12639 [details]
amarok 1.3.1: queue/dequeue tracks

tracks can be queued and dequeued from right-click-menu at each track

and they are marked with this small flag with the number of place in the queue

yet another spontanous idea: i know that kget can quit or shutdown if finished
with whole list of downloads ... amarok's "Stop Playing After Track" analogue
feature "Go to Offline Mode after Active Downloads" can also be usefull, isn't
it?
Comment 17 Urs Wolfer 2005-09-21 15:43:52 UTC
Thanks a lot for your comments and ideas :) .

These wishes are on the TODO for the almost rewritten KGet2.

http://websvn.kde.org/branches/work/make_kget_cool/
Comment 18 Marek 2006-05-11 20:14:23 UTC
In addition, I wish there was an option for KGet to silently keep a log or aknowledge that the connection failed, and after a while silently retry. I really-really hate that you have to answer a popup every time something happens. Very annoying
Comment 19 Thibaut Cousin 2008-01-12 21:54:56 UTC
What's the status of this bugreport now that KDE4 is out?
Comment 20 Lukas Appelhans 2008-01-13 14:56:55 UTC
Hi Thibaut!
It is still on our TODO-list :)

Lukas
Comment 21 Matthias Fuchs 2010-08-27 13:29:02 UTC
Is this still a problem?

I downloaded 20 files from a ftp that only supports few connections. It took a while but it finished all downloads without a single user intervention.
The scheduler was set to download everything at the same time, if I set it to two downloads it even works better.

Then I downloaded 600 small files from a mirror that supports few connections only as well and here with the scheduler set to 2 downloads it also worked very well. So it would be great if you could retry it with the scheduler activated and set to few downloads.

The reason I am asking is that I hope to avoid adding a feature to set the connections per server, since that would be hard to implement nicely.