8000 ENH: Create psm3.py by mikofski · Pull Request #694 · pvlib/pvlib-python · GitHub
[go: up one dir, main page]
Skip to content

ENH: Create psm3.py #694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
May 3, 2019
Merged

ENH: Create psm3.py #694

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e0859f7
Create psm3.py
mikofski Apr 25, 2019
6535a04
fix well known text for geometric POINT
mikofski Apr 26, 2019
f7003ac
TST: add test for getting psm3
mikofski Apr 26, 2019
15f93d3
update docs, closes #592
mikofski Apr 26, 2019
4c8c073
stickler, default interval
mikofski Apr 26, 2019
2b9cff5
add psm3 to api
mikofski Apr 26, 2019
c52d5b7
use astype, fix typos, change param names to match API
mikofski Apr 27, 2019
2809897
use pandas, better variable names, comments, set all dtypes
mikofski Apr 27, 2019
6cf57a4
stickler fixes
mikofski Apr 27, 2019
05cf879
api_key as param
mikofski Apr 27, 2019
13c196a
TST: decorate test_get_psm3 with needs_pandas_0_22
mikofski Apr 27, 2019
02c143f
api_key and email are required according to NREL
mikofski May 2, 2019
64d7cb9
Merge branch 'patch-1' of github.com:mikofski/pvlib-python into psm3
mikofski May 2, 2019
f9df42d
stickler: remove unused JSON import
mikofski May 2, 2019
b6f79cf
remove non-ASCII quotation marks around DEMO_KEY in warning
mikofski May 2, 2019
8884c7f
remove "required" for api_key param in get_psm3
wholmgren May 2, 2019
1935215
rewording for "email" param in get_psm3
wholmgren May 2, 2019
24c1f3b
clarify WKT formating and errors
mikofski May 2, 2019
4059db5
Merge branch 'patch-1' of github.com:mikofski/pvlib-python into psm3
mikofski May 2, 2019
f12c36b
DOC: defaults in argtype for get_psm3
mikofski May 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
clarify WKT formating and errors 
* in raise docstring, change "returns" to "raises", and clarify that
 either r.json()['errors'] or r.content is the exception arg depending
 on the type of HTTP error, and give examples
* add warning in docstring that data are limited to NSRDB locations
* comment on WKT formatting and add TODO to consider making a
 format_WKT() function in tools.py one day
* add response kwarg to HTTPError for better user exception handling
* in test_psm3 add comments to clarify the tests for errors and add two
 more test for bad "names" and interval args
  • Loading branch information
@mikofski
mikofski committed May 2, 2019
commit 24c1f3bdadb79e33646690d5b3ad29e273bea944
17 changes: 14 additions & 3 deletions pvlib/iotools/psm3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,10 @@ def get_psm3(latitude, longitude, api_key, email, names='tmy', interval=60,
Raises
------
requests.HTTPError
if the request return status is not ok then the ``'errors'`` from the
JSON response will be returned as an exception
if the request response status is not ok, then the ``'errors'`` field
from the JSON response or any error message in the content will be
raised as an exception, for example if the `api_key` was rejected or if
the coordinates were not found in the NSRDB

Notes
-----
Expand Down Expand Up @@ -103,6 +105,9 @@ def get_psm3(latitude, longitude, api_key, email, names='tmy', interval=60,

The second item is a dataframe with the timeseries data downloaded.

.. warning:: PSM3 is limited to data found in the NSRDB, please consult the
references below for locations with available data

See Also
--------
pvlib.iotools.read_tmy2, pvlib.iotools.read_tmy3
Expand All @@ -116,8 +121,14 @@ def get_psm3(latitude, longitude, api_key, email, names='tmy', interval=60,
<https://nsrdb.nrel.gov/>`_

"""
# The well know text (WKT) representation of geometry notation is strict.
# A POINT object is a string with longitude first, then the latitude, with
# four decimals each, and exactly one space between them.
longitude = ('%9.4f' % longitude).strip()
latitude = ('%8.4f' % latitude).strip()
# TODO: make format_WKT(object_type, *args) in tools.py

# required query-string parameters for request to PSM3 API
params = {
'api_key': api_key,
'full_name': full_name,
Expand All @@ -141,7 +152,7 @@ def get_psm3(latitude, longitude, api_key, email, names='tmy', interval=60,
errors = response.json()['errors']
except JSONDecodeError:
errors = response.content.decode('utf-8')
raise requests.HTTPError(errors)
raise requests.HTTPError(errors, response=response)
# the CSV is in the response content as a UTF-8 bytestring
# to use pandas we need to create a file buffer from the response
fbuf = io.StringIO(response.content.decode('utf-8'))
Expand Down
13 changes: 11 additions & 2 deletions pvlib/test/test_psm3.py
View file
Open in desktop 8000
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,13 @@
'Temperature Units', 'Pressure Units', 'Wind Direction Units',
'Wind Speed', 'Surface Albedo Units', 'Version']
PVLIB_EMAIL = 'pvlib-admin@googlegroups.com'
DEMO_KEY = 'DEMO_KEY'


@needs_pandas_0_22
def test_get_psm3():
"""test get_psm3"""
header, data = psm3.get_psm3(LATITUDE, LONGITUDE, 'DEMO_KEY', PVLIB_EMAIL)
header, data = psm3.get_psm3(LATITUDE, LONGITUDE, DEMO_KEY, PVLIB_EMAIL)
expected = pd.read_csv(TEST_DATA)
# check datevec columns
assert np.allclose(data.Year, expected.Year)
Expand All @@ -52,6 +53,14 @@ def test_get_psm3():
assert (data.index.tzinfo.zone == 'Etc/GMT%+d' % -header['Time Zone'])
# check errors
with pytest.raises(HTTPError):
# HTTP 403 forbidden because api_key is rejected
psm3.get_psm3(LATITUDE, LONGITUDE, api_key='BAD', email=PVLIB_EMAIL)
with pytest.raises(HTTPError):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does this one fail?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • api_key="BAD" fails because the response is HTTP/1.1 403 Forbidden and there's a long error message in the content, no JSON, that says something like, "please go to NREL developer network and register for a key"
  • Bristol-UK fails because PSM3 only works in the US, maybe South America, and I think some parts of India perhaps around Chennai? I can confirm - perhaps this should be a warning in the docstring?

perhaps I should add some more comments here to explain these tests - like I did here - for future devs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BAD was self explanatory. I don't think we need to test that it fails outside the US. Not our problem.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PSM3 has data in a lat/long rectangle around the lower 48 states, so there's coverage in northern Mexico for example. But it has a coastline filter and excludes data in the Gulf of Mexico. I think we should test for failure at lat/long where PSM3 does not have data.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does PSM3 return a sensible error if lat/lon outside of its area?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

from get_psm3 import get_psm3

import api_keys

MY_KEY = api_keys.get_api_key('nsrdb')

for lat, lon in zip([25, 25, 25], [-125, -124.5, -124]):
    try:
        meta, data = get_psm3(lat, lon, names='tmy', interval=60,
                              api_key=MY_KEY)
    except Exception as e:
        print("{}, {}: ".format(lat, lon), e)

Returns

25, -125:  ["The 'wkt' parameter did not match any known data points."]
25, -124.5:  Expecting value: line 1 column 1 (char 0)
25, -124:  Expecting value: line 1 column 1 (char 0)

The first line is what I expect, the next 2 are json decoding errors that puzzle me.

Copy link
Member
@cwhanse cwhanse May 2, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha, I suspect the 2nd and 3rd messages are empty responses because of the once-per-2second throttle, so raise requests.HTTPError(response.json()['errors']) is throwing the exception about Expecting value:...

Confirmed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

["The 'wkt' parameter did not match any known data points."]

This could be confusing to pvlib users because we're asking the to specify lat/lon instead of API. So I'm ok with catching this in the function and raising a more descriptive error about lat/lon.

psm3.get_psm3(51, -5, 'DEMO_KEY', PVLIB_EMAIL)
# coordinates were not found in the NSRDB
psm3.get_psm3(51, -5, DEMO_KEY, PVLIB_EMAIL)
with pytest.raises(HTTPError):
# names is not one of the available options
psm3.get_psm3(LATITUDE, LONGITUDE, DEMO_KEY, PVLIB_EMAIL, names='bad')
with pytest.raises(HTTPError):
# intervals can only be 30 or 60 minutes
psm3.get_psm3(LATITUDE, LONGITUDE, DEMO_KEY, PVLIB_EMAIL, interval=15)
0