8000 Add encoding parameter to read_tmy3 by AdamRJensen · Pull Request #1737 · pvlib/pvlib-python · GitHub
[go: up one dir, main page]

Skip to content

Add encoding parameter to read_tmy3 #1737

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
May 26, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
This reverts commit d868bb9.
Revert to f638a4d
  • Loading branch information
AdamRJensen committed May 26, 2023
commit 1ce2d5ccf167df67c98f6ba590eb1ad37300da3a
8 changes: 5 additions & 3 deletions docs/examples/adr-pvarray/plot_simulate_system.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,12 @@
PVLIB_DIR = pvlib.__path__[0]
DATA_FILE = os.path.join(PVLIB_DIR, 'data', '723170TYA.CSV')

tmy, metadata = iotools.read_tmy3(DATA_FILE, coerce_year=1990)
tmy, metadata = iotools.read_tmy3(DATA_FILE, coerce_year=1990,
map_variables=True)

df = pd.DataFrame({'ghi': tmy['GHI'], 'dhi': tmy['DHI'], 'dni': tmy['DNI'],
'temp_air': tmy['DryBulb'], 'wind_speed': tmy['Wspd'],
df = pd.DataFrame({'ghi': tmy['ghi'], 'dhi': tmy['dhi'], 'dni': tmy['dni'],
'temp_air': tmy['temp_air'],
'wind_speed': tmy['wind_speed'],
})

# %%
Expand Down
31 changes: 16 additions & 15 deletions docs/examples/irradiance-decomposition/plot_diffuse_fraction.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@
# of data measured from 1990 to 2010. Therefore we change the timestamps to a
# common year, 1990.
DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'
greensboro, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
greensboro, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
map_variables=True)

# Many of the diffuse fraction estimation methods require the "true" zenith, so
# we calculate the solar positions for the 1990 at Greensboro, NC.
Expand All @@ -36,8 +37,8 @@
solpos = get_solarposition(
greensboro.index.shift(freq="-30T"), latitude=metadata['latitude'],
longitude=metadata['longitude'], altitude=metadata['altitude'],
pressure=greensboro.Pressure*100, # convert from millibar to Pa
temperature=greensboro.DryBulb)
pressure=greensboro.pressure*100, # convert from millibar to Pa
temperature=greensboro.temp_air)
solpos.index = greensboro.index # reset index to end of the hour

# %%
Expand All @@ -56,10 +57,10 @@
# an exponential relation with airmass.

out_disc = irradiance.disc(
greensboro.GHI, solpos.zenith, greensboro.index, greensboro.Pressure*100)
greensboro.ghi, solpos.zenith, greensboro.index, greensboro.pressure*100)
# use "complete sum" AKA "closure" equations: DHI = GHI - DNI * cos(zenith)
df_disc = irradiance.complete_irradiance(
solar_zenith=solpos.apparent_zenith, ghi=greensboro.GHI, dni=out_disc.dni,
solar_zenith=solpos.apparent_zenith, ghi=greensboro.ghi, dni=out_disc.dni,
dhi=None)
out_disc = out_disc.rename(columns={'dni': 'dni_disc'})
out_disc['dhi_disc'] = df_disc.dhi
Expand All @@ -72,11 +73,11 @@
# developed by Richard Perez and Pierre Ineichen in 1992.

dni_dirint = irradiance.dirint(
greensboro.GHI, solpos.zenith, greensboro.index, greensboro.Pressure*100,
temp_dew=greensboro.DewPoint)
greensboro.ghi, solpos.zenith, greensboro.index, greensboro.pressure*100,
temp_dew=greensboro.temp_dew)
# use "complete sum" AKA "closure" equation: DHI = GHI - DNI * cos(zenith)
df_dirint = irradiance.complete_irradiance(
solar_zenith=solpos.apparent_zenith, ghi=greensboro.GHI, dni=dni_dirint,
solar_zenith=solpos.apparent_zenith, ghi=greensboro.ghi, dni=dni_dirint,
dhi=None)
out_dirint = pd.DataFrame(
{'dni_dirint': dni_dirint, 'dhi_dirint': df_dirint.dhi},
Expand All @@ -91,7 +92,7 @@
# splits kt into 3 regions: linear for kt <= 0.22, a 4th order polynomial
# between 0.22 < kt <= 0.8, and a horizontal line for kt > 0.8.

out_erbs = irradiance.erbs(greensboro.GHI, solpos.zenith, greensboro.index)
out_erbs = irradiance.erbs(greensboro.ghi, solpos.zenith, greensboro.index)
out_erbs = out_erbs.rename(columns={'dni': 'dni_erbs', 'dhi': 'dhi_erbs'})

# %%
Expand All @@ -102,7 +103,7 @@
# exponential correlation that is continuously differentiable and bounded
# between zero and one.

out_boland = irradiance.boland(greensboro.GHI, solpos.zenith, greensboro.index)
out_boland = irradiance.boland(greensboro.ghi, solpos.zenith, greensboro.index)
out_boland = out_boland.rename(
columns={'dni': 'dni_boland', 'dhi': 'dhi_boland'})

< 57AE /span>
Expand All @@ -118,20 +119,20 @@
# file together to make plotting easier.

dni_renames = {
'DNI': 'TMY3', 'dni_disc': 'DISC', 'dni_dirint': 'DIRINT',
'dni': 'TMY3', 'dni_disc': 'DISC', 'dni_dirint': 'DIRINT',
'dni_erbs': 'Erbs', 'dni_boland': 'Boland'}
dni = [
greensboro.DNI, out_disc.dni_disc, out_dirint.dni_dirint,
greensboro.dni, out_disc.dni_disc, out_dirint.dni_dirint,
out_erbs.dni_erbs, out_boland.dni_boland]
dni = pd.concat(dni, axis=1).rename(columns=dni_renames)
dhi_renames = {
'DHI': 'TMY3', 'dhi_disc': 'DISC', 'dhi_dirint': 'DIRINT',
'dhi': 'TMY3', 'dhi_disc': 'DISC', 'dhi_dirint': 'DIRINT',
'dhi_erbs': 'Erbs', 'dhi_boland': 'Boland'}
dhi = [
greensboro.DHI, out_disc.dhi_disc, out_dirint.dhi_dirint,
greensboro.dhi, out_disc.dhi_disc, out_dirint.dhi_dirint,
out_erbs.dhi_erbs, out_boland.dhi_boland]
dhi = pd.concat(dhi, axis=1).rename(columns=dhi_renames)
ghi_kt = pd.concat([greensboro.GHI/1000.0, out_erbs.kt], axis=1)
ghi_kt = pd.concat([greensboro.ghi/1000.0, out_erbs.kt], axis=1)

# %%
# Winter
Expand Down
7 changes: 4 additions & 3 deletions docs/examples/irradiance-transposition/plot_seasonal_tilt.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,13 @@ def get_orientation(self, solar_zenith, solar_azimuth):
# like we expect:

DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'
tmy, metadata = iotools.read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
tmy, metadata = iotools.read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
map_variables=True)
# shift from TMY3 right-labeled index to left-labeled index:
tmy.index = tmy.index - pd.Timedelta(hours=1)
weather = pd.DataFrame({
'ghi': tmy['GHI'], 'dhi': tmy['DHI'], 'dni': tmy['DNI'],
'temp_air': tmy['DryBulb'], 'wind_speed': tmy['Wspd'],
'ghi': tmy['ghi'], 'dhi': tmy['dhi'], 'dni': tmy['dni'],
'temp_air': tmy['temp_air'], 'wind_speed': tmy['wind_speed'],
})
loc = location.Location.from_tmy(metadata)
solpos = loc.get_solarposition(weather.index)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@
DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'

# get TMY3 dataset
tmy, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
tmy, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
map_variables=True)
# TMY3 datasets are right-labeled (AKA "end of interval") which means the last
# interval of Dec 31, 23:00 to Jan 1 00:00 is labeled Jan 1 00:00. When rolling
# up hourly irradiance to monthly insolation, a spurious January value is
Expand Down Expand Up @@ -60,9 +61,9 @@ def calculate_poa(tmy, solar_position, surface_tilt, surface_azimuth):
poa = irradiance.get_total_irradiance(
surface_tilt=surface_tilt,
surface_azimuth=surface_azimuth,
dni=tmy['DNI'],
ghi=tmy['GHI'],
dhi=tmy['DHI'],
dni=tmy['dni'],
ghi=tmy['ghi'],
dhi=tmy['dhi'],
solar_zenith=solar_position['apparent_zenith'],
solar_azimuth=solar_position['azimuth'],
model='isotropic')
Expand Down Expand Up @@ -97,7 +98,7 @@ def calculate_poa(tmy, solar_position, surface_tilt, surface_azimuth):
df_monthly['SAT-0.4'] = poa_irradiance.resample('m').sum()

# calculate the percent difference from GHI
ghi_monthly = tmy['GHI'].resample('m').sum()
ghi_monthly = tmy['ghi'].resample('m').sum()
df_monthly = 100 * (df_monthly.divide(ghi_monthly, axis=0) - 1)

df_monthly.plot()
Expand Down
5 changes: 3 additions & 2 deletions docs/examples/soiling/plot_greensboro_kimber_soiling.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,10 @@
DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'

# get TMY3 data with rain
greensboro, _ = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
greensboro, _ = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
map_variables=True)
# get the rain data
greensboro_rain = greensboro.Lprecipdepth
greensboro_rain = greensboro['Lprecip depth (mm)']
# calculate soiling with no wash dates and cleaning threshold of 25-mm of rain
THRESHOLD = 25.0
soiling_no_wash = kimber(greensboro_rain, cleaning_threshold=THRESHOLD)
Expand Down
1 change: 1 addition & 0 deletions docs/sphinx/source/reference/iotools.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ of sources and file formats relevant to solar energy modeling.
iotools.read_pvgis_tmy
iotools.get_pvgis_hourly
iotools.read_pvgis_hourly
iotools.get_pvgis_horizon
iotools.get_bsrn
iotools.read_bsrn
iotools.parse_bsrn
Expand Down
11 changes: 6 additions & 5 deletions docs/sphinx/source/user_guide/clearsky.rst
Original file line number Diff line number Diff line change
Expand Up @@ -216,25 +216,26 @@ wavelengths [Bir80]_, and is implemented in

In [1]: tmy_file = os.path.join(pvlib_data, '703165TY.csv') # TMY file

In [1]: tmy_data, tmy_header = read_tmy3(tmy_file, coerce_year=1999) # read TMY data
In [1]: tmy_data, tmy_header = read_tmy3(tmy_file, coerce_year=1999, map_variables=True)

In [1]: tl_historic = clearsky.lookup_linke_turbidity(time=tmy_data.index,
...: latitude=tmy_header['latitude'], longitude=tmy_header['longitude'])

In [1]: solpos = solarposition.get_solarposition(time=tmy_data.index,
...: latitude=tmy_header['latitude'], longitude=tmy_header['longitude'],
...: altitude=tmy_header['altitude'], pressure=tmy_data['Pressure']*mbars,
...: temperature=tmy_data['DryBulb'])
...: altitude=tmy_header['altitude'], pressure=tmy_data['pressure']*mbars,
...: temperature=tmy_data['temp_air'])

In [1]: am_rel = atmosphere.get_relative_airmass(solpos.apparent_zenith)

In [1]: am_abs = atmosphere.get_absolute_airmass(am_rel, tmy_data['Pressure']*mbars)
In [1]: am_abs = atmosphere.get_absolute_airmass(am_rel, tmy_data['pressure']*mbars)

In [1]: airmass = pd.concat([am_rel, am_abs], axis=1).rename(
...: columns={0: 'airmass_relative', 1: 'airmass_absolute'})

In [1]: tl_calculated = atmosphere.kasten96_lt(
...: airmass.airmass_absolute, tmy_data['Pwat'], tmy_data['AOD'])
...: airmass.airmass_absolute, tmy_data['precipitable_water'],
...: tmy_data['AOD (unitless)'])

In [1]: tl = pd.concat([tl_historic, tl_calculated], axis=1).rename(
...: columns={0:'Historic', 1:'Calculated'})
Expand Down
4 changes: 2 additions & 2 deletions docs/sphinx/source/user_guide/timetimezones.rst
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ Let's first examine how pvlib handles time when it imports a TMY3 file.
# some gymnastics to find the example file
pvlib_abspath = os.path.dirname(os.path.abspath(inspect.getfile(pvlib)))
file_abspath = os.path.join(pvlib_abspath, 'data', '703165TY.csv')
tmy3_data, tmy3_metadata = pvlib.iotools.read_tmy3(file_abspath)
tmy3_data, tmy3_metadata = pvlib.iotools.read_tmy3(file_abspath, map_variables=True)

tmy3_metadata

Expand All @@ -287,7 +287,7 @@ print just a few of the rows and columns of the large dataframe.

tmy3_data.index.tz

tmy3_data.loc[tmy3_data.index[0:3], ['GHI', 'DNI', 'AOD']]
tmy3_data.loc[tmy3_data.index[0:3], ['ghi', 'dni', 'AOD (unitless)']]

The :py:func:`~pvlib.iotools.read_tmy2` function also returns a DataFrame
with a localized DatetimeIndex.
Expand Down
47 changes: 47 additions & 0 deletions docs/sphinx/source/whatsnew/v0.10.0.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
.. _whatsnew_01000:


v0.10.0
-------


Breaking changes
~~~~~~~~~~~~~~~~
* Reorder arguments of :py:func:`pvlib.pvsystem.PVSystem.i_from_v`,
:py:func:`pvlib.pvsystem.i_from_v`, :py:func:`pvlib.pvsystem.v_from_i`,
:py:func:`pvlib.singlediode._lambertw_i_from_v`, and
:py:func:`pvlib.singlediode._lambertw_v_from_i` to match
:py:func:`pvlib.pvsystem.singlediode`.
(:issue:`1718`, :pull:`1719`)


Deprecations
~~~~~~~~~~~~


Enhancements
~~~~~~~~~~~~


Bug fixes
~~~~~~~~~


Testing
~~~~~~~


Documentation
~~~~~~~~~~~~~

Benchmarking
~~~~~~~~~~~~~


Requirements
~~~~~~~~~~~~


Contributors
~~~~~~~~~~~~
* Taos Transue (:ghuser:`reepoi`)
35 changes: 34 additions & 1 deletion docs/sphinx/source/whatsnew/v0.9.6.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,37 @@ v0.9.6 (Anticipated June 2023)
------------------------------


Breaking Changes
~~~~~~~~~~~~~~~~
* Modified the ``surface_azimuth`` parameter in :py:func:`pvlib.iotools.get_pvgis_hourly` to conform to the
pvlib azimuth convention (counterclockwise from north). Previously 0 degrees represented south.
(:issue:`1724`, :pull:`1739`)
* For consistency with the rest of pvlib, the ``tilt`` parameter is renamed
to ``surface_tilt`` in :py:func:`pvlib.soiling.hsu`. (:issue:`1717`, :pull:`1738`)

Deprecations
~~~~~~~~~~~~

* The ``recolumn`` parameter in :py:func:`pvlib.iotools.read_tmy3`, which maps
TMY3 column names to nonstandard alternatives, is now deprecated.
We encourage using ``map_variables`` (which produces standard pvlib names) instead.
(:issue:`1517`, :pull:`1623`)

Enhancements
~~~~~~~~~~~~
* Add optional encoding parameter to :py:func:`pvlib.iotools.read_tmy3`.
(:issue:`1732`, :pull:`1737`)

* Added function to retrieve horizon data from PVGIS
:py:func:`pvlib.iotools.get_pvgis_horizon`. (:issue:`1290`, :pull:`1395`)
* Added ``map_variables`` argument to the :py:func:`pvlib.iotools.read_tmy3` in
order to offer the option of mapping column names to standard pvlib names.
(:issue:`1517`, :pull:`1623`)
* Update the URL used in the :py:func:`pvlib.iotools.get_cams` function. The new URL supports load-balancing
and redirects to the fastest server. (:issue:`1688`, :pull:`1740`)
* :py:func:`pvlib.iotools.get_psm3` now has a ``url`` parameter to give the user
the option of controlling what NSRDB endpoint is used. (:pull:`1736`)
* :py:func:`pvlib.iotools.get_psm3` now uses the new NSRDB 3.2.2 endpoint for
hourly and half-hourly single-year datasets. (:issue:`1591`, :pull:`1736`)

Bug fixes
~~~~~~~~~
Expand Down Expand Up @@ -42,7 +64,18 @@ Contributors
~~~~~~~~~~~~
* Lakshya Garg (:ghuser:`Lakshyadevelops`)
* Adam R. Jensen (:ghuser:`adamrjensen`)
* Ben Pierce (:ghuser:`bgpierc`)
* Joseph Palakapilly (:ghuser:`JPalakapillyKWH`)
* Cliff Hansen (:ghuser:`cwhanse`)
* Anton Driesse (:ghuser:`adriesse`)
* Will Holmgren (:ghuser:`wholmgren`)
* Mark Mikofski (:ghuser:`mikofski`)
* Karel De Brabandere (:ghuser:`kdebrab`)
* Josh Stein (:ghuser:`jsstein`)
* Kevin Anderson (:ghuser:`kandersolar`)
* Siddharth Kaul (:ghuser:`k10blogger`)
* Kshitiz Gupta (:ghuser:`kshitiz305`)
* Stefan de Lange (:ghuser:`langestefan`)
* Andy Lam (:ghuser:`@andylam598`)
* :ghuser:`ooprathamm`
* Kevin Anderson (:ghuser:`kandersolar`)
49 changes: 49 additions & 0 deletions pvlib/data/test_read_pvgis_horizon.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
horizon_azimuth,horizon_elevation
0,9.9
7.5,13
15,14.5
22.5,15.7
30,14.9
37.5,15.3
45,15.7
52.5,15.7
60,13
67.5,11.5
75,11.1
82.5,11.5
90,10.3
97.5,11.5
105,10.3
112.5,9.5
120,10.7
127.5,11.8
135,11.8
142.5,8.8
150,8.4
157.5,7.3
165,5.7
172.5,5.7
180,4.6
187.5,3.4
195,0.8
202.5,0
210,0
217.5,0
225,0
232.5,0
240,0
247.5,0
255,0
262.5,0
270,0
277.5,0
285,0
292.5,0
300,0
307.5,0
315,1.1
322.5,1.9
330,3.8
337.5,5
345,6.5
352.5,9.2
Loading
0