Bug 403054 - GeoIP.dat.gz
Summary: GeoIP.dat.gz
Status: RESOLVED FIXED
Alias: None
Product: ktorrent
Classification: Applications
Component: general (show other bugs)
Version: unspecified
Platform: Slackware Linux
: HI normal
Target Milestone: ---
Assignee: Alexander Lohnau
URL:
Keywords:
: 396484 407507 414563 416709 425579 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-01-09 23:34 UTC by Glenn McCorkle
Modified: 2023-11-14 18:33 UTC (History)
15 users (show)

See Also:
Latest Commit:
Version Fixed In: 21.08


Attachments
screen capture of ktorrent message (11.43 KB, image/png)
2019-01-09 23:34 UTC, Glenn McCorkle
Details
Screen-cap of GeoIP.dat.gz now available. (99.46 KB, image/png)
2020-07-29 01:19 UTC, Glenn McCorkle
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Glenn McCorkle 2019-01-09 23:34:46 UTC
Created attachment 117375 [details]
screen capture of ktorrent message

SUMMARY

File or folder not found

STEPS TO REPRODUCE
1. start ktorrent

OBSERVED RESULT

The file or folder http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz does not exist

EXPECTED RESULT


SOFTWARE/OS VERSIONS
Windows: 
MacOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

ADDITIONAL INFORMATION
Comment 1 Federico Kereki 2019-02-03 02:39:27 UTC
I can confirm that the bug also occurs in OpenSUSE Tumbleweed.
Comment 2 Patrick Silva 2019-02-03 18:10:05 UTC
Same bug with ktorrent 5.1.1 on Arch Linux.
Comment 3 Kevin Chapman 2019-02-06 23:33:24 UTC
Same issue running KTorrent 5.1.1

Manjaro
KDE Plasma 5.14.5
KDE Framework 5.54.5
Qt Version 5.12.0
Comment 4 Giusy Digital 2019-02-07 14:10:50 UTC
https://dev.maxmind.com/geoip/geoip2/geolite2/
https://support.maxmind.com/geolite-legacy-discontinuation-notice/

On January 2, 2019, we discontinued our GeoLite Legacy databases. This discontinuation was first announced on January 2, 2018.

GeoLite Legacy databases are no longer available for download. Attempting to download a GeoLite Legacy database file will result in the error: “Database edition not found”.

Free geolocation data is still available through our GeoLite2 databases.
Comment 5 Tallowwood 2019-03-06 01:28:12 UTC
Same issue running ktorrent 5.1.0

Comment: Further to comment by GD #4 above, MaxMind states "...the [revised] GeoLite2 databases are free. However, you will need to update your GeoLite Legacy integrations to work with GeoLite2 databases."

https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz

openSUSE Leap 15
KDE Plasma Version: 5.12.6
KDE Frameworks Version: 5.45.0
Qt Version 5.9.4
Comment 6 Glenn McCorkle 2019-03-16 00:36:27 UTC
Question: Is anyone @ KDE working to resolve this issue
by modifying ktorrent to access the new database instead
of the discontinued one ?
Comment 7 Christoph Feck 2019-03-31 08:25:11 UTC
KTorrent has no maintainer currently, so it would require a volunteer to change the code.
Comment 8 Glenn McCorkle 2019-04-08 23:45:32 UTC
(In reply to Christoph Feck from comment #7)
> KTorrent has no maintainer currently, so it would require a volunteer to
> change the code.

OK... here's the change that needs to be made in the source code....

In this file.... ktorrent-master/plugins/infowidget/geoipmanager.cpp

namespace kt
{
   //legacy database discontinued.... https://bugs.kde.org/show_bug.cgi?id=403054
  //    QUrl GeoIPManager::geoip_url = QUrl(QStringLiteral("http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz"));
  //new database is here.....    
  QUrl GeoIPManager::geoip_url = QUrl(QStringLiteral("https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz"));
Comment 9 Christoph Feck 2019-04-09 07:09:06 UTC
The format of the database has changed, so only adjusting the URL won't work. See comment #5.
Comment 10 Glenn McCorkle 2019-04-09 23:08:53 UTC
(In reply to Christoph Feck from comment #9)
> The format of the database has changed, so only adjusting the URL won't
> work. See comment #5.

Well then....

Does anyone know what needs to be changed to be able to use the new database ?
Comment 11 Giusy Digital 2019-04-10 05:14:04 UTC
First thing would be something for totally disabling geolocalization inside the application just to avoid that annoying popup at every startup.

KTottent is a good torrent client, but unfortunately it's no longer maintained, I ended up using transmission-qt.
Comment 12 Bernhard Rosenkränzer 2019-05-11 20:46:19 UTC
We just fixed this in OpenMandriva.
Patch here: https://github.com/OpenMandrivaAssociation/ktorrent/blob/master/ktorrent-5.1.1-geoip.patch
Comment 13 Bernhard Rosenkränzer 2019-05-11 20:56:15 UTC
https://phabricator.kde.org/D21145
Comment 14 Christoph Feck 2019-05-13 19:47:15 UTC
*** Bug 407507 has been marked as a duplicate of this bug. ***
Comment 15 Bart C 2019-05-27 05:27:46 UTC
*** Bug 396484 has been marked as a duplicate of this bug. ***
Comment 16 Mathias Homann 2019-11-27 09:06:16 UTC
*** Bug 414563 has been marked as a duplicate of this bug. ***
Comment 17 Mathias Homann 2019-11-27 14:52:04 UTC
(In reply to Bernhard Rosenkränzer from comment #13)
> https://phabricator.kde.org/D21145

by the way: there's a ktorrent 5.1.2 rpm for openSUSE with that patch in my OBS project, seems to work just fine.
https://build.opensuse.org/package/show/home:lemmy04/ktorrent
Comment 18 Wolfgang Bauer 2020-01-03 10:29:14 UTC
(In reply to Mathias Homann from comment #17)
> (In reply to Bernhard Rosenkränzer from comment #13)
> > https://phabricator.kde.org/D21145
> 
> by the way: there's a ktorrent 5.1.2 rpm for openSUSE with that patch in my
> OBS project, seems to work just fine.
> https://build.opensuse.org/package/show/home:lemmy04/ktorrent

Not anymore, it shows an error dialog with "Unknown host geolite.maxmind.com" now.

https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
Comment 19 Mathias Homann 2020-01-03 13:53:03 UTC
(In reply to Bernhard Rosenkränzer from comment #12)
> We just fixed this in OpenMandriva.
> Patch here:
> https://github.com/OpenMandrivaAssociation/ktorrent/blob/master/ktorrent-5.1.
> 1-geoip.patch

Hm... I'm building ktorrent 5.1.2 with that patch, and I'm still getting:
- the popup about the old geoip library at startup
- no Geolocation info about peers

Any ideas?
Comment 20 Wolfgang Bauer 2020-01-03 14:47:27 UTC
(In reply to Mathias Homann from comment #19)
> (In reply to Bernhard Rosenkränzer from comment #12)
> > We just fixed this in OpenMandriva.
> > Patch here:
> > https://github.com/OpenMandrivaAssociation/ktorrent/blob/master/ktorrent-5.1.
> > 1-geoip.patch
> 
> Hm... I'm building ktorrent 5.1.2 with that patch, and I'm still getting:
> - the popup about the old geoip library at startup
> - no Geolocation info about peers
> 
> Any ideas?
See comment#18.

Quoting from that page:
Starting December 30, 2019, downloads will no longer be served from our public GeoLite2 page, from geolite.maxmind.com/download/geoip/database/*, or from any other public URL.
Comment 21 Bernhard Rosenkränzer 2020-01-03 15:38:35 UTC
Given they're essentially making an automated download impossible these days, the only way I could see to keep updating this database is opening a QWebEngineView to make the user sign up (which will require the user to confirm an email - something we can't easily automate), then make the user store the login information and use that to access the db files -- which would end up being too complicated for most users as well as potentially resulting in unmaintainable code (if they keep changing download locations).

As far as I'm concerned, the ways forward could be:

1. Fork the last version of the database that was fully open and keep publishing it in a fixed place hosted in a country not affected by insane laws. Probably the best way to go, but I doubt there will be any volunteers to keep the db up to date? Hosting it wouldn't be an issue, we could put it on OpenMandriva's ABF server in Switzerland.

2. Drop the automated db downloads, instead just use a db file in a fixed location in the filesystem and require the user (or distro) to keep the db up to date. Disable the country lookup feature if the file isn't there.

3. Determine the country by other means (e.g. whois lookup of the IP -- this will result in some incorrect answers, but then again the geoIP DB isn't guaranteed to be 100% accurate either)

4. Combine approaches 1. and 3. - generate a new db from whois information with some way to submit overrides and use that in place of the no-longer-public db

Thoughts?
Comment 22 Wolfgang Bauer 2020-01-03 15:57:38 UTC
Actually there still seems to be a way to download/update the database via a different URL that contains the license key (from https://dev.maxmind.com/geoip/geoipupdate/#Direct_Downloads):
In order to download the databases from a script or program, please use the “direct download URL” on our GeoIP download page and make the following changes:

Replace /geoip_download_by_token with /geoip_download
Replace token=XXXX with license_key=YOUR_LICENSE_KEY. You will need to replace the YOUR_LICENSE_KEY placeholder with an active license key associated with your MaxMind account.
If you wish to always download the latest database, remove the date parameter from the URL.
If you are using wget or curl from a shell script, please be sure to quote the URL.
This download will return a compressed file in gzip format, which can be uncompressed using a program like 7-Zip or gzip.

So it might be possible to leverage this to provide automatic download/update (the user could provide the license key in the settings e.g.).

Possible caveat:
Download Limits

MaxMind reserves the right to limit the number of downloads made within a limited period of time.

Another option I see is just disable the automatic download (and error message), and let the user do it manually (registrating, downloading the database and copy it to ~/.local/share/ktorrent/). That's similar to your second suggestion, but would not require any change, won't require the user to have root permissions, and is probably better on a multi-user system (as the license key is bound to a certain user AIUI).
Or maybe a combination of the two, look the database up in a system-wide location and in ~/.local/share/ktorrent/ (or any other user specific location).
I'm pretty sure that shipping a central database is not an option for every distribution out there (for whatever reason), especially if it would only be for ktorrent.

Your #3 would probably work to, but would be more difficult to implement I suppose.

As for #1, would that even be legal? I'm not sure about that.
Also, it would certainly get outdated.
Comment 23 Bernhard Rosenkränzer 2020-01-03 16:28:16 UTC
Thanks, confirmed that the download via the new URL + license_key works. That may be another option, but I doubt a lot of users will go through the process of signing up for the key and entering it in the settings.

Download limits are rather unlikely to become a problem, since ktorrent doesn't try to update the GeoIP database a lot.

#1 would be legal, since the database used to be under the
Creative Commons Attribution-ShareAlike 4.0 International license.
https://creativecommons.org/licenses/by-sa/4.0/
For all practical matters, that license is like the GPL.

#3 is fairly easy to do. This oneliner should be fairly accurate:
whois $IP_ADDR |grep -i '^Country:' |awk '{ print $2; }' |head -n1

Of course one down side of it is that it would mean firing off a whois request for every new IP seen, causing more network traffic than just looking it up in a local DB (hence idea #4).
Comment 24 Wolfgang Bauer 2020-01-03 17:12:40 UTC
(In reply to Bernhard Rosenkränzer from comment #23)
> #1 would be legal, since the database used to be under the
> Creative Commons Attribution-ShareAlike 4.0 International license.
> https://creativecommons.org/licenses/by-sa/4.0/
> For all practical matters, that license is like the GPL.
Hm, in that case, maybe it could even be shipped with ktorrent itself?

Forking and hosting it somewhere would probably be not much better (regarding updates and/or longetivity) I suppose.
Comment 25 Wolfgang Bauer 2020-01-03 17:36:58 UTC
(In reply to Wolfgang Bauer from comment #24)
> Hm, in that case, maybe it could even be shipped with ktorrent itself?
And then we could allow the user to update it to the latest "official" version in the settings if they provide a license key, maybe even provide an option for auto-update.

Kaffeine does something similar: they ship a default channel file (that gets updated on every release) and offer a button in the settings to download the latest version from some online provider.

Just a thought though.
Comment 26 Patrick Silva 2020-01-25 07:07:24 UTC
*** Bug 416709 has been marked as a duplicate of this bug. ***
Comment 27 Glenn McCorkle 2020-07-29 01:19:38 UTC
Created attachment 130468 [details]
Screen-cap of GeoIP.dat.gz now available.
Comment 28 Glenn McCorkle 2020-07-29 01:21:32 UTC
Well I'll be damned... the file is available again.

http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz

I can't test it to see if it is compatible with our 'buggy' versions
of ktorrent because I now have a patched build from Slackware
which has an old GeoIP data compiled in.

+--------------------------+
Sun Mar 17 20:40:15 UTC 2019

kde/ktorrent-4.3.1-x86_64-4.txz:  Rebuilt.
  Embed a copy of the GeoIP database since the download link no longer works.
_______________________________________________________________________________
Comment 29 Glenn McCorkle 2020-07-29 01:31:34 UTC
Nevermind.

Their 'fix' for the bug was to put a 'zero byte' on their server :(
Comment 30 Glenn McCorkle 2020-07-29 01:32:39 UTC
root@glennmcc-i5:/# wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
--2020-07-28 21:32:08--  http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
Resolving geolite.maxmind.com (geolite.maxmind.com)... 23.217.138.110, 23.202.231.169
Connecting to geolite.maxmind.com (geolite.maxmind.com)|23.217.138.110|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0
Saving to: 'GeoIP.dat.gz'

GeoIP.dat.gz            [ <=>                ]       0  --.-KB/s    in 0s      

2020-07-28 21:32:09 (0.00 B/s) - 'GeoIP.dat.gz' saved [0/0]
Comment 31 Glenn McCorkle 2020-07-29 03:01:36 UTC
Perhaps usable ?

https://mailfud.org/geoip-legacy/

Here you can find regularly updated GeoIP legacy databases.
Conversion is done from Maxmind GeoLite2 databases, using https://github.com/sherpya/geolite2legacy.
Comment 32 Mathias Homann 2020-07-29 08:25:58 UTC
I think relying on yet another external ressource is the wrong approach.
These days it is easy enough to have a system-wide libgeoip set up that has up to date databases, why not simply link against that and be done with this mess?
Comment 33 Linus Kardell 2020-08-03 11:11:55 UTC
If you don't have a proper fix, can you at least get rid of the annoying pop-up?
Comment 34 Christoph Feck 2020-09-10 09:36:29 UTC
*** Bug 425579 has been marked as a duplicate of this bug. ***
Comment 35 Alexander Lohnau 2021-05-21 07:52:10 UTC
Can confirm, I will look into fixing that.
Comment 36 Bug Janitor Service 2021-05-21 19:36:04 UTC
A possibly relevant merge request was started @ https://invent.kde.org/network/ktorrent/-/merge_requests/37
Comment 37 Alexander Lohnau 2021-05-21 19:36:42 UTC
>If you don't have a proper fix, can you at least get rid of the annoying pop-up?

IMHO that could be done for the 21.04 branch, thoughts?
Comment 38 Andrius Štikonas 2021-05-21 20:01:07 UTC
(In reply to Alexander Lohnau from comment #37)
> >If you don't have a proper fix, can you at least get rid of the annoying pop-up?
> 
> IMHO that could be done for the 21.04 branch, thoughts?

Hmm, probably a bit too large change for bugfix release. And as you have said we change dependencies, which is not ideal from the point of view of dependency freeze.
Comment 39 Andrius Štikonas 2021-05-21 20:07:49 UTC
(In reply to Alexander Lohnau from comment #37)
> >If you don't have a proper fix, can you at least get rid of the annoying pop-up?
> 
> IMHO that could be done for the 21.04 branch, thoughts?

Oh, I think just silencing the popup is fine.
Comment 40 Alexander Lohnau 2021-05-21 21:40:35 UTC
Git commit ca128f13009c527339e28e305218ba01ea206f18 by Alexander Lohnau.
Committed on 21/05/2021 at 20:21.
Pushed by stikonas into branch 'release/21.04'.

Do not attempt to download non-existing GeoIp data

This skips the download and consequently silences the warning, a proper fix is in master implemented.

M  +2    -0    plugins/infowidget/geoipmanager.cpp

https://invent.kde.org/network/ktorrent/commit/ca128f13009c527339e28e305218ba01ea206f18
Comment 41 Alexander Lohnau 2021-05-21 21:41:10 UTC
Git commit 6c1ae6b2530c5f7c72611a95aea92275520a5aee by Alexander Lohnau.
Committed on 21/05/2021 at 19:53.
Pushed by stikonas into branch 'master'.

Fix broken GeoIp plugin

Instead of the broken custom logic use only the system provided lib.
FIXED-IN: 21.08

M  +20   -22   plugins/infowidget/CMakeLists.txt
D  +0    -968  plugins/infowidget/GeoIP.c
D  +0    -148  plugins/infowidget/GeoIP.h
M  +0    -93   plugins/infowidget/geoipmanager.cpp
M  +0    -34   plugins/infowidget/geoipmanager.h
M  +0    -1    plugins/infowidget/infowidgetplugin.cpp
M  +24   -10   plugins/infowidget/peerviewmodel.cpp
M  +9    -2    plugins/infowidget/peerviewmodel.h

https://invent.kde.org/network/ktorrent/commit/6c1ae6b2530c5f7c72611a95aea92275520a5aee
Comment 42 Glenn McCorkle 2021-05-22 17:15:22 UTC
FYI,

Slackware "fixed" this problem over 2yrs ago via....
________________________________________________________________________________
Sun Mar 17 20:40:15 UTC 2019

kde/ktorrent-4.3.1-x86_64-4.txz:  Rebuilt.
  Embed a copy of the GeoIP database since the download link no longer works.
_________________________________________________________________________________
Comment 43 Mathias Homann 2021-05-22 18:14:44 UTC
you do realize that a two year old copy of the GeoIP informations is about as useful as not having any at all?
Comment 44 Glenn McCorkle 2021-05-22 23:06:59 UTC
ktorrent shows simply in which country an IP is located but does not break
it down any further to show a precise location within that country.

Have there been drastic changes in the past 2yrs as to in which countries IPs are  located ?
Comment 45 Glenn McCorkle 2021-05-22 23:27:51 UTC
(In reply to Glenn McCorkle from comment #44)
> ktorrent shows simply in which country an IP is located but does not break
> it down any further to show a precise location within that country.
> 
> Have there been drastic changes in the past 2yrs as to in which countries
> IPs are  located ?

In addition... there's not really any useful purpose to the GeoIP location
other than curiosity of wondering in what country a particular peer is located.

Do you know of some actual useful purpose for-which that info can be used ?

Can't speak for any other ktorrent users, but as the original reporter of this 'bug'....
I was simply interested in getting rid of that annoying pop-up. ;-)
Comment 46 Glenn McCorkle 2021-05-22 23:32:03 UTC
One more point on top of that... I and I'm sure many, many ktorrent users
use anonymous proxies for doing our bittorrent downloads which of-course
makes the GeoIP location info a 100% moot-point ;-)
Comment 47 Jack Hill 2023-11-14 17:35:01 UTC
I have opened an MR here that ports KTorrent to use a free database provided by DB-IP.com: https://invent.kde.org/network/ktorrent/-/merge_requests/87