8000 BUG: fix + test open_mfdataset fails on variable attributes with list… · pydata/xarray@8931c69 · GitHub
[go: up one dir, main page]
Skip to content

Commit 8931c69

Browse files
HasanAhmadQ7max-sixty
HasanAhmadQ7
authored and
max-sixty
committed
BUG: fix + test open_mfdataset fails on variable attributes with list… (#3181)
* BUG: fix + test open_mfdataset fails on variable attributes with list type Using open_mfdataset on a series of netcdf files having variable attributes with type list will fail with the following exception, when these attributes have different values from one file to another: solves: #3034 * DOC: add #3034 bug fix to whats new * Update xarray/core/utils.py Co-Authored-By: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> * Update xarray/core/utils.py Co-Authored-By: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> * BUG: fix + test open_mfdataset fails on variable attributes with list type Using open_mfdataset on a series of netcdf files having variable attributes with type list will fail with the following exception, when these attributes have different values from one file to another: solves: #3034 * DOC: add #3034 bug fix to whats new * Update xarray/core/utils.py Co-Authored-By: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> * Update xarray/core/utils.py Co-Authored-By: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> * DOC: update whats' new
1 parent 01a9baa commit 8931c69

File tree

3 files changed

+44
-5
lines changed

3 files changed

+44
-5
lines changed

‎doc/whats-new.rst

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,12 @@ Bug fixes
6161
By `Tom Nicholas <http://github.com/TomNicholas>`_.
6262
- Fixed crash when applying ``distributed.Client.compute()`` to a DataArray
6363
(:issue:`3171`). By `Guido Imperiale <https://github.com/crusaderky>`_.
64-
65-
64+
- Better error message when using groupby on an empty DataArray (:issue:`3037`).
65+
By `Hasan Ahmad <https://github.com/HasanAhmadQ7>`_.
66+
- Fix error that arises when using open_mfdataset on a series of netcdf files
67+
having differing values for a variable attribute of type list. (:issue:`3034`)
68+
By `Hasan Ahmad <https://github.com/HasanAhmadQ7>`_.
69+
6670
.. _whats-new.0.12.3:
6771

6872
v0.12.3 (10 July 2019)
@@ -103,8 +107,6 @@ Bug fixes
103107
- Fix HDF5 error that could arise when reading multiple groups from a file at
104108
once (:issue:`2954`).
105109
By `Stephan Hoyer <https://github.com/shoyer>`_.
106-
- Better error message when using groupby on an empty DataArray (:issue:`3037`).
107-
By `Hasan Ahmad <https://github.com/HasanAhmadQ7>`_.
108110

109111
.. _whats-new.0.12.2:
110112

‎xarray/core/utils.py

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,18 +129,31 @@ def maybe_wrap_array(original, new_array):
129129

130130
def equivalent(first: T, second: T) -> bool:
131131
"""Compare two objects for equivalence (identity or equality), using
132-
array_equiv if either object is an ndarray
132+
array_equiv if either object is an ndarray. If both objects are lists,
133+
equivalent is sequentially called on all the elements.
133134
"""
134135
# TODO: refactor to avoid circular import
135136
from . import duck_array_ops
136137
if isinstance(first, np.ndarray) or isinstance(second, np.ndarray):
137138
return duck_array_ops.array_equiv(first, second)
139+
elif isinstance(first, list) or isinstance(second, list):
140+
return list_equiv(first, second)
138141
else:
139142
return ((first is second) or
140143
(first == second) or
141144
(pd.isnull(first) and pd.isnull(second)))
142145

143146

147+
def list_equiv(first, second):
148+
equiv = True
149+
if len(first) != len(second):
150+
return False
151+
else:
152+
for f, s in zip(first, second):
153+
equiv = equiv and equivalent(f, s)
154+
return equiv
155+
156+
144157
def peek_at(iterable: Iterable[T]) -> Tuple[T, Iterator[T]]:
145158
"""Returns the first value from iterable, as well as a new iterator with
146159
the same content as the original iterable

‎xarray/tests/test_backends.py

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2361,6 +2361,30 @@ def test_open_mfdataset_manyfiles(readengine, nfiles, parallel, chunks,
23612361
assert_identical(original, actual)
23622362

23632363

2364+
@requires_netCDF4
2365+
def test_open_mfdataset_list_attr():
2366+
"""
2367+
Case when an attribute of type list differs across the multiple files
2368+
"""
2369+
from netCDF4 import Dataset
2370+
with create_tmp_files(2) as nfiles:
2371+
for i in range(2):
2372+
f = Dataset(nfiles[i], "w")
2373+
f.createDimension("x", 3)
2374+
vlvar = f.createVariable("test_var", np.int32, ("x"))
2375+
# here create an attribute as a list
2376+
vlvar.test_attr = ["string a {}".format(i),
2377+
"string b {}".format(i)]
2378+
vlvar[:] = np.arange(3)
2379+
f.close()
2380+
ds1 = open_dataset(nfiles[0])
2381+
ds2 = open_dataset(nfiles[1])
2382+
original = xr.concat([ds1, ds2], dim='x')
2383+
with xr.open_mfdataset([nfiles[0], nfiles[1]], combine='nested',
2384+
concat_dim='x') as actual:
2385+
assert_identical(actual, original)
2386+
2387+
23642388
@requires_scipy_or_netCDF4
23652389
@requires_dask
23662390
class TestOpenMFDatasetWithDataVarsAndCoordsKw:

0 commit comments

Comments
 (0)
0