8000 Freegeoip database is deprecated, 2 new databases, 3 databases fixes … · pythonthings/ip2geotools@3f8c8fc · GitHub
[go: up one dir, main page]
Skip to content

Commit 3f8c8fc

Browse files
author
Tomas Caha
committed
Freegeoip database is deprecated, 2 new databases, 3 databases fixes because of their changes
1 parent 4deeab2 commit 3f8c8fc

File tree

9 files changed

+180
-70
lines changed

9 files changed

+180
-70
lines changed

‎CHANGELOG.rst

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,23 @@
1+
0.1.3 - 27-Nov-2018
2+
-------------------
3+
4+
* New ``ip2geotools.databases.commercial.Ipdata`` requested by Jonathan Kosgei from ipdata.co
5+
* New ``ip2geotools.databases.noncommercial.Ipstack`` as a replacement for ``ip2geotools.databases.noncommercial.Freegeoip``
6+
* Fix ``ip2geotools.databases.commercial.DbIpWeb`` because of new webpage layout
7+
* Fix ``ip2geotools.databases.commercial.Ip2LocationWeb`` because of new webpage layout
8+
* Fix ``ip2geotools.databases.commercial.NeustarWeb`` because of new URL
9+
* Default free api key in ``ip2geotools.databases.noncommercial.DbIpCity``
10+
* ``ip2geotools.databases.noncommercial.Freegeoip`` is deprecated!
11+
112
0.1.2 - 30-Nov-2017
2-
-----------------
13+
-------------------
314

415
* Fix ``ip2geotools.databases.commercial.DbIpWeb`` because of new webpage layout
516
* "IP address not found" error can be recognized in ``ip2geotools.databases.commercial.SkyhookContextAcceleratorIp``
617
* Custom exceptions can be formatted in ``ip2geotools.errors``
718

819
0.1.1 - 01-Nov-2017
9-
-----------------
20+
-------------------
1021

1122
* Fix installation from PyPi using ``pip``
1223

‎README.rst

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -21,26 +21,26 @@ Basic usage
2121

2222
.. code-block:: pycon
2323
24-
>>> from ip2geotools.databases.noncommercial import Freegeoip
25-
>>> response = Freegeoip.get('147.229.2.90')
24+
>>> from ip2geotools.databases.noncommercial import DbIpCity
25+
>>> response = DbIpCity.get('147.229.2.90', api_key='free')
2626
>>> response.ip_address
2727
'147.229.2.90'
2828
>>> response.city
29-
'Brno'
29+
'Brno (Brno střed)'
3030
>>> response.region
3131
'South Moravian'
3232
>>> response.country
3333
'CZ'
3434
>>> response.latitude
35-
49.2
35+
49.1926824
3636
>>> response.longitude
37-
16.6333
37+
16.6182105
3838
>>> response.to_json()
39-
'{"ip_address": "147.229.2.90", "city": "Brno", "region": "South Moravian", "country": "CZ", "latitude": 49.2, "longitude": 16.6333}'
39+
'{"ip_address": "147.229.2.90", "city": "Brno (Brno st\u0159ed)", "region": "South Moravian", "country": "CZ", "latitude": 49.1926824, "longitude": 16.6182105}'
4040
>>> response.to_xml()
41-
'<?xml version="1.0" encoding="UTF-8" ?><ip_location><ip_address>147.229.2.90</ip_address><city>Brno</city><region>South Moravian</region><country>CZ</country><latitude>49.2</latitude><longitude>16.6333</longitude></ip_location>'
41+
'<?xml version="1.0" encoding="UTF-8" ?><ip_location><ip_address>147.229.2.90</ip_address><city>Brno (Brno střed)</city><region>South Moravian</region><country>CZ</country><latitude>49.1926824</latitude><longitude>16.6182105</longitude></ip_location>'
4242
>>> response.to_csv(',')
43-
'147.229.2.90,Brno,South Moravian,CZ,49.2,16.6333'
43+
'147.229.2.90,Brno (Brno střed),South Moravian,CZ,49.1926824,16.6182105'
4444
4545
Command-line usage
4646
------------------
@@ -49,7 +49,7 @@ When installed, you can invoke ``ip2geotools`` from the command-line:
4949

5050
.. code:: bash
5151
52-
ip2geotools [-h] -d {dbipcity,hostip,freegeoip,maxmindgeolite2city,ip2location,dbipweb,maxmindgeoip2city,ip2locationweb,neustarweb,geobytescitydetails,skyhookcontextacceleratorip,ipinfo,eurek}
52+
ip2geotools [-h] -d {dbipcity,hostip,freegeoip,ipstack,maxmindgeolite2city,ip2location,dbipweb,maxmindgeoip2city,ip2locationweb,neustarweb,geobytescitydetails,skyhookcontextacceleratorip,ipinfo,eurek,ipdata}
5353
[--api_key API_KEY] [--db_path DB_PATH] [-u USERNAME]
5454
[-p PASSWORD] [-f {json,xml,csv-space,csv-tab,inline}] [-v]
5555
IP_ADDRESS
@@ -80,8 +80,8 @@ Examples:
8080

8181
.. code:: bash
8282
83-
$ ip2geotools 147.229.2.90 -d freegeoip -f json
84-
{"ip_address": "147.229.2.90", "city": "Brno", "region": "South Moravian", "country": "CZ", "latitude": 49.2, "longitude": 16.6333}
83+
$ ip2geotools 147.229.2.90 -d dbipcity -f json
84+
{"ip_address": "147.229.2.90", "city": "Brno (Brno st\u0159ed)", "region": "South Moravian", "country": "CZ", "latitude": 49.1926824, "longitude": 16.6182105}
8585
8686
Models
8787
------
@@ -138,9 +138,10 @@ Following classes access many different noncommercial and commercial geolocation
138138

139139
* ``DbIpCity``: https://db-ip.com/api/
140140
* ``HostIP``: http://hostip.info/
141-
* ``Freegeoip``: http://freegeoip.net/
141+
* ``Freegeoip``: http://freegeoip.net/ **Database is deprecated!**
142+
* ``Ipstack``: https://ipstack.com/
142143
* ``MaxMindGeoLite2City``: https://dev.maxmind.com/geoip/geoip2/geolite2/
143-
* ``Ip2Location``: http://lite.ip2location.com/database/ip-country-region-city-latitude-longitude
144+
* ``Ip2Location``: https://lite.ip2location.com/database/ip-country-region-city-latitude-longitude
144145

145146
``ip2geotools.databases.commercial``
146147
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -152,6 +153,7 @@ Following classes access many different noncommercial and commercial geolocation
152153
* ``SkyhookContextAcceleratorIp``: http://www.skyhookwireless.com/
153154
* ``IpInfo``: https://ipinfo.io/
154155
* ``Eurek``: https://www.eurekapi.com/
156+
* ``Ipdata``: https://ipdata.co/
155157

156158
Requirements
157159
------------
@@ -172,4 +174,4 @@ License
172174
Author
173175
------
174176

175-
``ip2geotools`` was written by Tomas Caha <tomas-net at seznam dot cz> for master\'s thesis at `FEEC <http://www.feec.vutbr.cz/>`_ `BUT <https://www.vutbr.cz/>`_ 2017/2018.
177+
``ip2geotools`` was written by Tomas Caha <tomas-net at seznam dot cz> for master\'s thesis at `FEEC <http://www.feec.vutbr.cz/>`_ `BUT <https://www.vutbr.cz/>`_ 2018/2019.

‎ip2geotools/__init__.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@
44
__title__ = 'ip2geotools'
55
__description__ = 'Simple tool for getting geolocation information on ' + \
66
'given IP address from various geolocation databases.'
7-
__version__ = '0.1.2'
7+
__version__ = '0.1.3'
88
__author__ = 'Tomas Caha'
99
__author_email__ = 'tomas-net@seznam.cz'
1010
__url__ = 'https://github.com/tomas-net/ip2geotools'
1111
__license__ = 'MIT License'
12-
__copyright__ = 'Copyright (c) 2017 Tomas Caha'
12+
__copyright__ = 'Copyright (c) 2018 Tomas Caha'

‎ip2geotools/cli.py

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
from ip2geotools.databases.noncommercial import DbIpCity, \
2222
HostIP, \
2323
Freegeoip, \
24+
Ipstack, \
2425
MaxMindGeoLite2City, \
2526
Ip2Location
2627
from ip2geotools.databases.commercial import DbIpWeb, \
@@ -30,7 +31,8 @@
3031
GeobytesCityDetails, \
3132
SkyhookContextAcceleratorIp, \
3233
IpInfo, \
33-
Eurek
34+
Eurek, \
35+
Ipdata
3436
from ip2geotools.models import IpLocation
3537
from ip2geotools.errors import LocationError
3638

@@ -58,11 +60,11 @@ def execute(self):
5860
description='{0} version {1}'.format(self.prog_name, ip2geotools.__version__) + \
5961
'\n\n{0}'.format(ip2geotools.__description__),
6062
epilog=('\n\nexample:' + \
61-
'\n get information on 147.229.2.90 from freegeoip.net in JSON format' + \
62-
'\n {prog_name} 147.229.2.90 -d freegeoip -f json' + \
63+
'\n get information on 147.229.2.90 from DB-IP API in JSON format' + \
64+
'\n {prog_name} 147.229.2.90 -d dbipcity -f json' + \
6365
'\n\nauthor:' + \
6466
'\n {prog_name} was written by {author} <{author_email}> ' + \
65-
'for master\'s thesis at FEEC BUT 2017/2018').format(
67+
'for master\'s thesis at FEEC BUT 2018/2019').format(
6668
prog_name=self.prog_name,
6769
author=ip2geotools.__author__,
6870
author_email=ip2geotools.__author_email__),
@@ -84,6 +86,7 @@ def execute(self):
8486
'dbipcity',
8587
'hostip',
8688
'freegeoip',
89+
'ipstack',
8790
'maxmindgeolite2city',
8891
'ip2location',
8992

@@ -96,6 +99,7 @@ def execute(self):
9699
'skyhookcontextacceleratorip',
97100
'ipinfo',
98101
'eurek',
102+
'ipdata',
99103
])
100104

101105
parser.add_argument('--api_key',
@@ -141,12 +145,18 @@ def execute(self):
141145
try:
142146
# noncommercial databases
143147
if arguments.database == 'dbipcity':
144-
ip_location = DbIpCity.get(arguments.IP_ADDRESS,
145-
api_key=arguments.api_key)
148+
if arguments.api_key:
149+
ip_location = DbIpCity.get(arguments.IP_ADDRESS,
150+
api_key=arguments.api_key)
151+
else:
152+
ip_location = DbIpCity.get(arguments.IP_ADDRESS)
146153
elif arguments.database == 'hostip':
147154
ip_location = HostIP.get(arguments.IP_ADDRESS)
148155
elif arguments.database == 'freegeoip':
149156
ip_location = Freegeoip.get(arguments.IP_ADDRESS)
157+
elif arguments.database == 'ipstack':
158+
ip_location = Ipstack.get(arguments.IP_ADDRESS,
159+
api_key=arguments.api_key)
150160
elif arguments.database == 'maxmindgeolite2city':
151161
ip_location = MaxMindGeoLite2City.get(arguments.IP_ADDRESS,
152162
db_path=arguments.db_path)
@@ -174,6 +184,12 @@ def execute(self):
174184
elif arguments.database == 'eurek':
175185
ip_location = Eurek.get(arguments.IP_ADDRESS,
176186
api_key=arguments.api_key)
187+
elif arguments.database == 'ipdata':
188+
if arguments.api_key:
189+
ip_location = Ipdata.get(arguments.IP_ADDRESS,
190+
api_key=arguments.api_key)
191+
else:
192+
ip_location = Ipdata.get(arguments.IP_ADDRESS)
177193

178194
# print formatted output
179195
if arguments.format == 'json':

‎ip2geotools/databases/commercial.py

Lines changed: 81 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -53,9 +53,8 @@ def get(ip_address, api_key=None, db_path=None, username=None, password=None):
5353
content = request.content.decode('utf-8')
5454
pq = pyquery.PyQuery(content)
5555
parsed_ip = pq('html > body div.container > h1') \
56+
.remove('span') \
5657
.text() \
57-
.lower() \
58-
.replace('ip address', '') \
5958
.strip()
6059
parsed_country = pq('html > body > div.container table tr:contains("Country") td') \
6160
.text() \
@@ -248,13 +247,18 @@ def get(ip_address, api_key=None, db_path=None, username=None, password=None):
248247
parsed_ip = pq('html > body > div#main.container table:first tr:contains("IP Address") td:nth-child(2)') \
249248
.text() \
250249
.strip()
251-
parsed_country = pq('html > body > div#main.container table:first tr:contains("Location") td:nth-child(2) img') \
250+
parsed_country = pq('html > body > div#main.container table:first tr:contains("Country") td:nth-child(2) img') \
252251
.attr('src') \
253252
.strip() \
254253
.replace('/images/flags/', '') \
255254
.replace('.png', '') \
256255
.upper()
257-
parsed_location = pq('html > body > div#main.container table:first tr:contains("Location") td:nth-child(2)') \
256+
parsed_region = pq('html > body > div#main.container table:first tr:contains("Region") td:nth-child(2)') \
257+
.eq(0) \
258+
.text() \
259+
.strip()
260+
parsed_city = pq('html > body > div#main.container table:first tr:contains("City") td:nth-child(2)') \
261+
.eq(0) \
258262
.text() \
259263
.strip()
260264
parsed_coords = pq('html > body > div#main.container table:first tr:contains("Latitude & Longitude of City") td:nth-child(2)') \
@@ -273,8 +277,8 @@ def get(ip_address, api_key=None, db_path=None, username=None, password=None):
273277
# format data
274278
try:
275279
ip_location.country = parsed_country
276-
ip_location.region = parsed_location.split(',')[1].strip()
277-
ip_location.city = parsed_location.split(',')[2].strip()
280+
ip_location.region = parsed_region
281+
ip_location.city = parsed_city
278282

279283
parsed_coords = parsed_coords.split('(')[0].split(',')
280284
ip_location.latitude = float(parsed_coords[0].strip())
@@ -292,15 +296,15 @@ def get(ip_address, api_key=None, db_path=None, username=None, password=None):
292296
class NeustarWeb(IGeoIpDatabase):
293297
"""
294298
Class for accessing geolocation data provided by searching directly
295-
on https://www.neustar.biz/resources/tools/ip-geolocation-lookup-tool/.
299+
on https://www.security.neustar/resources/tools/ip-geolocation-lookup-tool/.
296300
297301
"""
298302

299303
@staticmethod
300304
def get(ip_address, api_key=None, db_path=None, username=None, password=None):
301305
# process request
302306
try:
303-
request = requests.post('https://www.neustar.biz/resources/tools/ip-geolocation-lookup-tool',
307+
request = requests.post('https://www.security.neustar/resources/tools/ip-geolocation-lookup-tool',
304308
headers={'User-Agent': 'Mozilla/5.0'},
305309
data=[('ip', ip_address)],
306310
timeout=62)
@@ -676,3 +680,72 @@ def get(ip_address, api_key=None, db_path=None, username=None, password=None):
676680
ip_location.longitude = None
677681

678682
return ip_location
683+
684+
685+
class Ipdata(IGeoIpDatabase):
686+
"""
687+
Class for accessing geolocation data provided by https://ipdata.co/.
688+
689+
"""
690+
691+
@staticmethod
692+
def get(ip_address, api_key='test', db_path=None, username=None, password=None):
693+
# process request
694+
try:
695+
request = requests.get('https://api.ipdata.co/' + quote(ip_address)
696+
+ '?api-key=' + quote(api_key),
697+
timeout=62)
698+
except:
699+
raise ServiceError()
700+
701+
# check for HTTP errors
702+
if request.status_code != 200 and request.status_code != 400:
703+
if request.status_code == 401:
704+
raise PermissionRequiredError()
705+
elif request.status_code == 403:
706+
raise LimitExceededError()
707+
else:
708+
raise ServiceError()
709+
710+
# parse content
711+
try:
712+
content = request.content.decode('utf-8')
713+
content = json.loads(content)
714+
except:
715+
raise InvalidResponseError()
716+
717+
# check for errors
718+
if content.get('message'):
719+
if 'private IP address' in content['message']:
720+
raise IpAddressNotFoundError(ip_address)
721+
else:
722+
raise InvalidRequestError()
723+
724+
# prepare return value
725+
ip_location = IpLocation(ip_address)
726+
727+
# format data
728+
if content['country_code'] == '':
729+
ip_location.country = None
730+
else:
731+
ip_location.country = content['country_code']
732+
733+
if content['region'] == '':
734+
ip_location.region = None
735+
else:
736+
ip_location.region = content['region']
737+
738+
if content['city'] == '':
739+
ip_location.city = None
740+
else:
741+
ip_location.city = content['city']
742+
743+
if content['latitude'] != '-' and content['longitude'] != '-':
744+
ip_location.latitude = float(content['latitude'])
745+
ip_location.longitude = float(content['longitude'])
746+
else:
747+
ip_location.latitude = None
748+
ip_location.longitude = None
749+
750+
return ip_location
751+

‎ip2geotools/databases/interfaces.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,4 @@ def get(ip_address, api_key, db_path, username, password):
2727
"""
2828

2929
raise NotImplementedError
30+

0 commit comments

Comments
 (0)
0