8000 Add map_variables and -99999 nan value to read_crn by AdamRJensen · Pull Request #1368 · pvlib/pvlib-python · GitHub
[go: up one dir, main page]
Skip to content

Add map_variables and -99999 nan value to read_crn #1368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Feb 17, 2022
Merged

Add map_variables and -99999 nan value to read_crn #1368

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
9f8e256
Add map_variables argument to read_crn
AdamRJensen Jan 5, 2022
a10fe28
Add test coverage for map_variables
AdamRJensen Jan 6, 2022
456cb71
Update whatsnew
AdamRJensen Jan 7, 2022
6bc6950
Remove unnecessary tz_localize
AdamRJensen Jan 7, 2022
106030a
Replace nans with .replace instead of .where
AdamRJensen Jan 7, 2022
1fcaa97
Extend documentation
AdamRJensen Jan 7, 2022
02fbfbb
Simply test coverage of map_variables
AdamRJensen Jan 7, 2022
427fa38
Use assert_index_equal instead of assert
AdamRJensen Jan 7, 2022
ebe5510
Add import of assert_index_equal
AdamRJensen Jan 7, 2022
a409915
Add -99999 and -999999 to nan values
AdamRJensen Jan 7, 2022
81ead2d
Add -99999 nan bug to whatsnew
AdamRJensen Jan 8, 2022
ee56796
Add -99999 to test file
AdamRJensen Jan 10, 2022
b041b8e
Update doc and whatsnew
AdamRJensen Jan 10, 2022
a753377
Remove -999999 from list of nans
AdamRJensen Jan 10, 2022
485e2a9
Merge branch 'master' into patch-2 10000
AdamRJensen Jan 10, 2022
a66f71d
Minor doc update
AdamRJensen Jan 11, 2022
457fd5b
Add dictionary of nan values
AdamRJensen Jan 12, 2022
999738b
Change CRN_VARIABLE_MAP back to VARIABLE_MAP
AdamRJensen Jan 12, 2022
667f243
Remove numpy import
AdamRJensen Jan 12, 2022
a07acbc
Reformat setting of dtypes
AdamRJensen Jan 12, 2022
538fd75
Merge branch 'master' into patch-2
AdamRJensen Feb 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Reformat setting of dtypes
  • Loading branch information
@AdamRJensen
AdamRJensen committed Jan 12, 2022
commit a07acbc6ce5bb1a4b3ea51532f58a4981be634d4
21 changes: 10 additions & 11 deletions pvlib/iotools/crn.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@
import pandas as pd


HEADERS = (
'WBANNO UTC_DATE UTC_TIME LST_DATE LST_TIME CRX_VN LONGITUDE LATITUDE '
'AIR_TEMPERATURE PRECIPITATION SOLAR_RADIATION SR_FLAG '
'SURFACE_TEMPERATURE ST_TYPE ST_FLAG RELATIVE_HUMIDITY RH_FLAG '
'SOIL_MOISTURE_5 SOIL_TEMPERATURE_5 WETNESS WET_FLAG WIND_1_5 WIND_FLAG'
)
HEADERS = [
'WBANNO', 'UTC_DATE', 'UTC_TIME', 'LST_DATE', 'LST_TIME', 'CRX_VN',
'LONGITUDE', 'LATITUDE', 'AIR_TEMPERATURE', 'PRECIPITATION',
'SOLAR_RADIATION', 'SR_FLAG', 'SURFACE_TEMPERATURE', 'ST_TYPE', 'ST_FLAG',
'RELATIVE_HUMIDITY', 'RH_FLAG', 'SOIL_MOISTURE_5', 'SOIL_TEMPERATURE_5',
'WETNESS', 'WET_FLAG', 'WIND_1_5', 'WIND_FLAG']

VARIABLE_MAP = {
'LONGITUDE': 'longitude',
Expand Down Expand Up @@ -107,13 +107,12 @@ def read_crn(filename, map_variables=True):
"""

# read in data
data = pd.read_fwf(filename, header=None, names=HEADERS.split(' '),
widths=WIDTHS, na_values=NAN_DICT)
data = pd.read_fwf(filename, header=None, names=HEADERS, widths=WIDTHS,
na_values=NAN_DICT)
# Remove rows with all nans
data = data.dropna(axis=0, how='all')
# loop here because dtype kwarg not supported in read_fwf until 0.20
for (col, _dtype) in zip(data.columns, DTYPES):
data[col] = data[col].astype(_dtype)
# set dtypes here because dtype kwarg not supported in read_fwf until 0.20
data = data.astype(dict(zip(HEADERS, DTYPES)))
Comment on lines +114 to +115
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pvlib now requires pandas > 0.22 so perhaps we can remove this line in favor of the dtype kwarg

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not believe this is possible as there are rows with all nans - and you cannot set the column dtype to int if the column contains nans. In the current version, the dtypes are set after the all nan rows are removed.

I get the following error when i set dtype=dict(zip(HEADERS, DTYPES)) with read_fwf:

ValueError: Unable to convert column WBANNO to type int64


# set index
# UTC_TIME does not have leading 0s, so must zfill(4) to comply
Expand Down
0