8000 Add retrieval function for ERA5 reanalysis data by AdamRJensen · Pull Request #1264 · pvlib/pvlib-python · GitHub
[go: up one dir, main page]

Skip to content

Add retrieval function for ERA5 reanalysis data #1264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 44 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
8e8661c
Create era5.py
AdamRJensen Jun 16, 2021 8000
25662a4
Update era5.py
AdamRJensen Jun 17, 2021
f6bc57e
Update era5.py
AdamRJensen Jun 17, 2021
57eecb5
Update era5.py
AdamRJensen Jul 27, 2021
c135481
Fix stickler
AdamRJensen Jul 27, 2021
f4e9242
Add xarray and cdsapi to requirements and setup.py
AdamRJensen Aug 8, 2021
af3067e
Remove grib support and update variable_map
AdamRJensen Aug 8, 2021
b2c400c
Fix stickler
AdamRJensen Aug 8, 2021
bb62d7f
Add era5 to api.rst, __init__, and whatsnewfile
AdamRJensen Aug 9, 2021
9f6b15f
Fix error in __init__.py
AdamRJensen Aug 9, 2021
9a22c04
Change parse to read in __init__.py
AdamRJensen Aug 9, 2021
21bedd1
Update get_era5 documentation
AdamRJensen Aug 9, 2021
6c76efd
Merge remote-tracking branch 'upstream/master' into era5
AdamRJensen Aug 9, 2021
3422ddf
Improve docs and use open_mfdataset in read_era5
AdamRJensen Aug 11, 2021
545bafa
Coverage for ERA5 incl. test file
AdamRJensen Aug 11, 2021
8ed6df5
Export CDSAPI_KEY in conda_linux.yml
AdamRJensen Aug 11, 2021
041732f
Fix stickler
AdamRJensen Aug 11, 2021
93bd4ec
Install xarray with pip in requirements-py36
AdamRJensen Aug 11, 2021
519b060
Include xarray in TEST_REQUIRE in setup.py
AdamRJensen Aug 11, 2021
b732c76
Add UID to era5 tests
AdamRJensen Aug 11, 2021
4c911ac
Add requires_xarray to conftest.py
AdamRJensen Aug 11, 2021
f1b83a2
Renaming get_era5 inputs
AdamRJensen Aug 11, 2021
7a13531
Updated CDSAPI_KEY usage
AdamRJensen Aug 11, 2021
87b6a50
Extend test coverage
AdamRJensen Aug 11, 2021
bff7299
Add dask as optional dependency
AdamRJensen Aug 16, 2021
5de52bc
Update documentation
AdamRJensen Aug 16, 2021
0721f15
More documentation updates
AdamRJensen Aug 16, 2021
2e90c67
Extend parsed metadata
AdamRJensen Aug 16, 2021
9f0a624
Coverage for output_format parameter
AdamRJensen Aug 16, 2021
7921666
Fix stickler
AdamRJensen Aug 16, 2021
03e4a5f
Update description of test_get_cams_bad_request
AdamRJensen Aug 16, 2021
5e1db86
Implement changes from review by kanderso-nrel
AdamRJensen Aug 16, 2021
dd443e4
Remove cds_client input parameter
AdamRJensen Aug 17, 2021
c25780e
Localize dataframe and add helper functions to pvlib.tools
AdamRJensen Aug 17, 2021
5a00ad5
Reformat imports of non-standard packages
AdamRJensen Aug 19, 2021
46f38c1
Set quit=True in cds_client and add depdencies in whatsnew
AdamRJensen Aug 20, 2021
cde8a4a
Rename variables before extracting metadata
AdamRJensen Aug 20, 2021
3f96eec
Change file_location to file_url and remove lat/lon offset
AdamRJensen Aug 27, 2021
6cd44b1
Merge branch 'master' into era5
AdamRJensen Aug 27, 2021
5db2fef
Remove has_tables from conftest.py
AdamRJensen Aug 27, 2021
f6443fc
Merge branch 'era5' of https://github.com/AdamRJensen/pvlib-python in…
AdamRJensen Aug 27, 2021
6534c66
Revert "Merge branch 'era5' of https://github.com/AdamRJensen/pvlib-p…
AdamRJensen Aug 27, 2021
3ae5327
Revert back to 3f96eec
AdamRJensen Aug 27, 2021
faa5a45
Merge remote-tracking branch 'upstream/master' into era5
AdamRJensen Sep 13, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix stickler
  • Loading branch information
AdamRJensen committed Jul 27, 2021
commit c135481687b37368587def23271c410b6d106d3f
30 changes: 13 additions & 17 deletions pvlib/iotools/era5.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,7 @@ def Client(*a, **kw):
'mean_surface_downward_short_wave_radiation_flux_clear_sky',
'mean_surface_direct_short_wave_radiation_flux',
'mean_surface_direct_short_wave_radiation_flux_clear_sky',
'mean_top_downward_short_wave_radiation_flux'
]
'mean_top_downward_short_wave_radiation_flux']


ERA5_VARIABLE_MAP = {
Expand All @@ -52,14 +51,12 @@ def Client(*a, **kw):
'mean_surface_downward_short_wave_radiation_flux_clear_sky': 'ghi_clear',
'mean_surface_direct_short_wave_radiation_flux': 'bhi',
'mean_surface_direct_short_wave_radiation_flux_clear_sky': 'bhi_clear',
'mean_top_downward_short_wave_radiation_flux': 'ghi_extra',
}
'mean_top_downward_short_wave_radiation_flux': 'ghi_extra'}


ERA5_EXTENSIONS = {
'netcdf': '.nc',
'grib': '.grib',
}
'grib': '.grib'}

ERA5_HOURS = [
'00:00', '01:00', '02:00', '03:00', '04:00', '05:00', '06:00', '07:00',
Expand Down Expand Up @@ -95,7 +92,7 @@ def get_era5(latitude, longitude, start, end,

Also, access requires user registration [3]_ and creating a local file with
your personal API key, see [4]_ for instructions.

Important
---------
Retrieving data from the CDS data store is significanlty slower than
Expand All @@ -104,7 +101,7 @@ def get_era5(latitude, longitude, start, end,

You can check the status of your api at
https://cds.climate.copernicus.eu/cdsapp#!/yourrequests

The status of all queued requests can be seen at
https://cds.climate.copernicus.eu/live/queue

Expand Down Expand Up @@ -185,26 +182,25 @@ def get_era5(latitude, longitude, start, end,
'date': start.strftime('%Y-%m-%d') + '/' + end.strftime('%Y-%m-%d'),
'time': ERA5_HOURS,
'grid': grid,
'area': area,
}
'area': area}

# Retrieve path to the file
file_location = cds_client.retrieve(dataset_name, params)

if file_format not in list(ERA5_EXTENSIONS):
raise ValueError(f"File format must be in {list(ERA5_EXTENSIONS)}")

# Load file into memory
with requests.get(file_location.location) as res:
if file_format=='netcdf': # open netcdf files using xarray
if file_format == 'netcdf': # open netcdf files using xarray
ds = xr.open_dataset(res.content)

# Save the file locally if local_path has been specified
if local_path is not None:
with open(local_path, 'wb') as f:
f.write(res.content)

if (file_format=='grib') & (local_path is not None):
if (file_format == 'grib') & (local_path is not None):
warnings.warn('Parsing of grib files is not supported.')
ds = xr.open_dataset(local_path, engine='cfgrib')
elif (file_format == 'grib') & (local_path is None):
Expand All @@ -222,12 +218,12 @@ def read_era5(filename, map_variables=True):
The ERA5 time stamp convention is to label data periods by the right edge,
e.g., the time stamp 12:00 for hourly data refers to the period from 11.00
to 12:00.

"""
# load into memory
#with urlopen(filename) as f:
# ds = xr.open_dataset(f.read())
# with urlopen(filename) as f:
# ds = xr.open_dataset(f.read())

ds = xr.open_dataset(filename) #, chunks=chunks)
ds = xr.open_dataset(filename) # chunks=chunks)

return ds
0