Bug 403054

I can confirm that the bug also occurs in OpenSUSE Tumbleweed.

Same bug with ktorrent 5.1.1 on Arch Linux.

Same issue running KTorrent 5.1.1

Manjaro
KDE Plasma 5.14.5
KDE Framework 5.54.5
Qt Version 5.12.0

https://dev.maxmind.com/geoip/geoip2/geolite2/
https://support.maxmind.com/geolite-legacy-discontinuation-notice/

On January 2, 2019, we discontinued our GeoLite Legacy databases. This discontinuation was first announced on January 2, 2018.

GeoLite Legacy databases are no longer available for download. Attempting to download a GeoLite Legacy database file will result in the error: “Database edition not found”.

Free geolocation data is still available through our GeoLite2 databases.

Same issue running ktorrent 5.1.0

Comment: Further to comment by GD #4 above, MaxMind states "...the [revised] GeoLite2 databases are free. However, you will need to update your GeoLite Legacy integrations to work with GeoLite2 databases."

https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz

openSUSE Leap 15
KDE Plasma Version: 5.12.6
KDE Frameworks Version: 5.45.0
Qt Version 5.9.4

Question: Is anyone @ KDE working to resolve this issue
by modifying ktorrent to access the new database instead
of the discontinued one ?

KTorrent has no maintainer currently, so it would require a volunteer to change the code.

(In reply to Christoph Feck from comment #7)
> KTorrent has no maintainer currently, so it would require a volunteer to
> change the code.

OK... here's the change that needs to be made in the source code....

In this file.... ktorrent-master/plugins/infowidget/geoipmanager.cpp

namespace kt
{
   //legacy database discontinued.... https://bugs.kde.org/show_bug.cgi?id=403054
  //    QUrl GeoIPManager::geoip_url = QUrl(QStringLiteral("http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz"));
  //new database is here.....    
  QUrl GeoIPManager::geoip_url = QUrl(QStringLiteral("https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz"));

The format of the database has changed, so only adjusting the URL won't work. See comment #5.

(In reply to Christoph Feck from comment #9)
> The format of the database has changed, so only adjusting the URL won't
> work. See comment #5.

Well then....

Does anyone know what needs to be changed to be able to use the new database ?

First thing would be something for totally disabling geolocalization inside the application just to avoid that annoying popup at every startup.

KTottent is a good torrent client, but unfortunately it's no longer maintained, I ended up using transmission-qt.

We just fixed this in OpenMandriva.
Patch here: https://github.com/OpenMandrivaAssociation/ktorrent/blob/master/ktorrent-5.1.1-geoip.patch

https://phabricator.kde.org/D21145

*** Bug 407507 has been marked as a duplicate of this bug. ***

*** Bug 396484 has been marked as a duplicate of this bug. ***

*** Bug 414563 has been marked as a duplicate of this bug. ***

(In reply to Bernhard Rosenkränzer from comment #13)
> https://phabricator.kde.org/D21145

by the way: there's a ktorrent 5.1.2 rpm for openSUSE with that patch in my OBS project, seems to work just fine.
https://build.opensuse.org/package/show/home:lemmy04/ktorrent

(In reply to Mathias Homann from comment #17)
> (In reply to Bernhard Rosenkränzer from comment #13)
> > https://phabricator.kde.org/D21145
> 
> by the way: there's a ktorrent 5.1.2 rpm for openSUSE with that patch in my
> OBS project, seems to work just fine.
> https://build.opensuse.org/package/show/home:lemmy04/ktorrent

Not anymore, it shows an error dialog with "Unknown host geolite.maxmind.com" now.

https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

(In reply to Bernhard Rosenkränzer from comment #12)
> We just fixed this in OpenMandriva.
> Patch here:
> https://github.com/OpenMandrivaAssociation/ktorrent/blob/master/ktorrent-5.1.
> 1-geoip.patch

Hm... I'm building ktorrent 5.1.2 with that patch, and I'm still getting:
- the popup about the old geoip library at startup
- no Geolocation info about peers

Any ideas?

(In reply to Mathias Homann from comment #19)
> (In reply to Bernhard Rosenkränzer from comment #12)
> > We just fixed this in OpenMandriva.
> > Patch here:
> > https://github.com/OpenMandrivaAssociation/ktorrent/blob/master/ktorrent-5.1.
> > 1-geoip.patch
> 
> Hm... I'm building ktorrent 5.1.2 with that patch, and I'm still getting:
> - the popup about the old geoip library at startup
> - no Geolocation info about peers
> 
> Any ideas?
See comment#18.

Quoting from that page:
Starting December 30, 2019, downloads will no longer be served from our public GeoLite2 page, from geolite.maxmind.com/download/geoip/database/*, or from any other public URL.

Given they're essentially making an automated download impossible these days, the only way I could see to keep updating this database is opening a QWebEngineView to make the user sign up (which will require the user to confirm an email - something we can't easily automate), then make the user store the login information and use that to access the db files -- which would end up being too complicated for most users as well as potentially resulting in unmaintainable code (if they keep changing download locations).

As far as I'm concerned, the ways forward could be:

1. Fork the last version of the database that was fully open and keep publishing it in a fixed place hosted in a country not affected by insane laws. Probably the best way to go, but I doubt there will be any volunteers to keep the db up to date? Hosting it wouldn't be an issue, we could put it on OpenMandriva's ABF server in Switzerland.

2. Drop the automated db downloads, instead just use a db file in a fixed location in the filesystem and require the user (or distro) to keep the db up to date. Disable the country lookup feature if the file isn't there.

3. Determine the country by other means (e.g. whois lookup of the IP -- this will result in some incorrect answers, but then again the geoIP DB isn't guaranteed to be 100% accurate either)

4. Combine approaches 1. and 3. - generate a new db from whois information with some way to submit overrides and use that in place of the no-longer-public db

Thoughts?

Actually there still seems to be a way to download/update the database via a different URL that contains the license key (from https://dev.maxmind.com/geoip/geoipupdate/#Direct_Downloads):
In order to download the databases from a script or program, please use the “direct download URL” on our GeoIP download page and make the following changes:

Replace /geoip_download_by_token with /geoip_download
Replace token=XXXX with license_key=YOUR_LICENSE_KEY. You will need to replace the YOUR_LICENSE_KEY placeholder with an active license key associated with your MaxMind account.
If you wish to always download the latest database, remove the date parameter from the URL.
If you are using wget or curl from a shell script, please be sure to quote the URL.
This download will return a compressed file in gzip format, which can be uncompressed using a program like 7-Zip or gzip.

So it might be possible to leverage this to provide automatic download/update (the user could provide the license key in the settings e.g.).

Possible caveat:
Download Limits

MaxMind reserves the right to limit the number of downloads made within a limited period of time.

Another option I see is just disable the automatic download (and error message), and let the user do it manually (registrating, downloading the database and copy it to ~/.local/share/ktorrent/). That's similar to your second suggestion, but would not require any change, won't require the user to have root permissions, and is probably better on a multi-user system (as the license key is bound to a certain user AIUI).
Or maybe a combination of the two, look the database up in a system-wide location and in ~/.local/share/ktorrent/ (or any other user specific location).
I'm pretty sure that shipping a central database is not an option for every distribution out there (for whatever reason), especially if it would only be for ktorrent.

Your #3 would probably work to, but would be more difficult to implement I suppose.

As for #1, would that even be legal? I'm not sure about that.
Also, it would certainly get outdated.

Thanks, confirmed that the download via the new URL + license_key works. That may be another option, but I doubt a lot of users will go through the process of signing up for the key and entering it in the settings.

Download limits are rather unlikely to become a problem, since ktorrent doesn't try to update the GeoIP database a lot.

#1 would be legal, since the database used to be under the
Creative Commons Attribution-ShareAlike 4.0 International license.
https://creativecommons.org/licenses/by-sa/4.0/
For all practical matters, that license is like the GPL.

#3 is fairly easy to do. This oneliner should be fairly accurate:
whois $IP_ADDR |grep -i '^Country:' |awk '{ print $2; }' |head -n1

Of course one down side of it is that it would mean firing off a whois request for every new IP seen, causing more network traffic than just looking it up in a local DB (hence idea #4).

(In reply to Bernhard Rosenkränzer from comment #23)
> #1 would be legal, since the database used to be under the
> Creative Commons Attribution-ShareAlike 4.0 International license.
> https://creativecommons.org/licenses/by-sa/4.0/
> For all practical matters, that license is like the GPL.
Hm, in that case, maybe it could even be shipped with ktorrent itself?

Forking and hosting it somewhere would probably be not much better (regarding updates and/or longetivity) I suppose.

(In reply to Wolfgang Bauer from comment #24)
> Hm, in that case, maybe it could even be shipped with ktorrent itself?
And then we could allow the user to update it to the latest "official" version in the settings if they provide a license key, maybe even provide an option for auto-update.

Kaffeine does something similar: they ship a default channel file (that gets updated on every release) and offer a button in the settings to download the latest version from some online provider.

Just a thought though.

*** Bug 416709 has been marked as a duplicate of this bug. ***

Created attachment 130468 [details]
Screen-cap of GeoIP.dat.gz now available.

Well I'll be damned... the file is available again.

http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz

I can't test it to see if it is compatible with our 'buggy' versions
of ktorrent because I now have a patched build from Slackware
which has an old GeoIP data compiled in.

+--------------------------+
Sun Mar 17 20:40:15 UTC 2019

kde/ktorrent-4.3.1-x86_64-4.txz:  Rebuilt.
  Embed a copy of the GeoIP database since the download link no longer works.
_______________________________________________________________________________

Nevermind.

Their 'fix' for the bug was to put a 'zero byte' on their server :(

root@glennmcc-i5:/# wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
--2020-07-28 21:32:08--  http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
Resolving geolite.maxmind.com (geolite.maxmind.com)... 23.217.138.110, 23.202.231.169
Connecting to geolite.maxmind.com (geolite.maxmind.com)|23.217.138.110|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0
Saving to: 'GeoIP.dat.gz'

GeoIP.dat.gz            [ <=>                ]       0  --.-KB/s    in 0s      

2020-07-28 21:32:09 (0.00 B/s) - 'GeoIP.dat.gz' saved [0/0]

Perhaps usable ?

https://mailfud.org/geoip-legacy/

Here you can find regularly updated GeoIP legacy databases.
Conversion is done from Maxmind GeoLite2 databases, using https://github.com/sherpya/geolite2legacy.

I think relying on yet another external ressource is the wrong approach.
These days it is easy enough to have a system-wide libgeoip set up that has up to date databases, why not simply link against that and be done with this mess?

If you don't have a proper fix, can you at least get rid of the annoying pop-up?

*** Bug 425579 has been marked as a duplicate of this bug. ***

Can confirm, I will look into fixing that.

A possibly relevant merge request was started @ https://invent.kde.org/network/ktorrent/-/merge_requests/37

>If you don't have a proper fix, can you at least get rid of the annoying pop-up?

IMHO that could be done for the 21.04 branch, thoughts?

(In reply to Alexander Lohnau from comment #37)
> >If you don't have a proper fix, can you at least get rid of the annoying pop-up?
> 
> IMHO that could be done for the 21.04 branch, thoughts?

Hmm, probably a bit too large change for bugfix release. And as you have said we change dependencies, which is not ideal from the point of view of dependency freeze.

(In reply to Alexander Lohnau from comment #37)
> >If you don't have a proper fix, can you at least get rid of the annoying pop-up?
> 
> IMHO that could be done for the 21.04 branch, thoughts?

Oh, I think just silencing the popup is fine.

Git commit ca128f13009c527339e28e305218ba01ea206f18 by Alexander Lohnau.
Committed on 21/05/2021 at 20:21.
Pushed by stikonas into branch 'release/21.04'.

Do not attempt to download non-existing GeoIp data

This skips the download and consequently silences the warning, a proper fix is in master implemented.

M  +2    -0    plugins/infowidget/geoipmanager.cpp

https://invent.kde.org/network/ktorrent/commit/ca128f13009c527339e28e305218ba01ea206f18

Git commit 6c1ae6b2530c5f7c72611a95aea92275520a5aee by Alexander Lohnau.
Committed on 21/05/2021 at 19:53.
Pushed by stikonas into branch 'master'.

Fix broken GeoIp plugin

Instead of the broken custom logic use only the system provided lib.
FIXED-IN: 21.08

M  +20   -22   plugins/infowidget/CMakeLists.txt
D  +0    -968  plugins/infowidget/GeoIP.c
D  +0    -148  plugins/infowidget/GeoIP.h
M  +0    -93   plugins/infowidget/geoipmanager.cpp
M  +0    -34   plugins/infowidget/geoipmanager.h
M  +0    -1    plugins/infowidget/infowidgetplugin.cpp
M  +24   -10   plugins/infowidget/peerviewmodel.cpp
M  +9    -2    plugins/infowidget/peerviewmodel.h

https://invent.kde.org/network/ktorrent/commit/6c1ae6b2530c5f7c72611a95aea92275520a5aee

FYI,

Slackware "fixed" this problem over 2yrs ago via....
________________________________________________________________________________
Sun Mar 17 20:40:15 UTC 2019

kde/ktorrent-4.3.1-x86_64-4.txz:  Rebuilt.
  Embed a copy of the GeoIP database since the download link no longer works.
_________________________________________________________________________________

you do realize that a two year old copy of the GeoIP informations is about as useful as not having any at all?

ktorrent shows simply in which country an IP is located but does not break
it down any further to show a precise location within that country.

Have there been drastic changes in the past 2yrs as to in which countries IPs are  located ?

(In reply to Glenn McCorkle from comment #44)
> ktorrent shows simply in which country an IP is located but does not break
> it down any further to show a precise location within that country.
> 
> Have there been drastic changes in the past 2yrs as to in which countries
> IPs are  located ?

In addition... there's not really any useful purpose to the GeoIP location
other than curiosity of wondering in what country a particular peer is located.

Do you know of some actual useful purpose for-which that info can be used ?

Can't speak for any other ktorrent users, but as the original reporter of this 'bug'....
I was simply interested in getting rid of that annoying pop-up. ;-)

One more point on top of that... I and I'm sure many, many ktorrent users
use anonymous proxies for doing our bittorrent downloads which of-course
makes the GeoIP location info a 100% moot-point ;-)

I have opened an MR here that ports KTorrent to use a free database provided by DB-IP.com: https://invent.kde.org/network/ktorrent/-/merge_requests/87