ds.notnull() fails with AttributeError on pandas 0.21.0rc1 #1663

gerritholl · 2017-10-27T15:19:33Z

xarray.Dataset({"A": ("x", arange(5))}).notnull() fails with an 'AttributeError' when using numpy 1.13.3, xarray 0.9.6, and pandas 0.21.0rc1. The AttributeError is raised by pandas; see below.

$ cat mwe.py
#!/usr/bin/env python3.6

import numpy
print(numpy.__version__)
import xarray
print(xarray.__version__)
import pandas
print(pandas.__version__)

xarray.Dataset({"A": ("x", numpy.arange(5))}).notnull()
$ ./mwe.py
1.13.3
0.9.6
0.21.0rc1
Traceback (most recent call last):
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/xarray/core/ops.py", line 193, in func
    return getattr(self, name)(*args, **kwargs)
AttributeError: 'Variable' object has no attribute 'notna'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./mwe.py", line 10, in <module>
    xarray.Dataset({"A": ("x", numpy.arange(5))}).notnull()
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/xarray/core/dataset.py", line 2485, in func
    ds._variables[k] = f(self._variables[k], *args, **kwargs)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/xarray/core/ops.py", line 195, in func
    return f(self, *args, **kwargs)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/pandas/core/dtypes/missing.py", line 212, in notna
    res = isna(obj)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/pandas/core/dtypes/missing.py", line 45, in isna
    return _isna(obj)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/pandas/core/dtypes/missing.py", line 60, in _isna_new
    return obj._constructor(obj._data.isna(func=isna))
AttributeError: 'Variable' object has no attribute '_constructor'

The text was updated successfully, but these errors were encountered:

gerritholl · 2017-10-27T15:36:11Z

Just confirmed this is caused by a change in pandas somewhere between 0.20.3 and 0.21.0rc1. I don't know if that is a bug in pandas, or a deliberate change that xarray will somehow need to handle, in particular after 0.21.0 final is released.

gerritholl · 2017-10-27T16:44:01Z

The offending commit is pandas-dev/pandas@7930202

closes #15001

gerritholl · 2017-10-27T16:48:39Z

What I still don't know: is this a bug in xarray or a bug in pandas? Or neither?

gerritholl · 2017-10-27T17:10:54Z

I think we'd need to change PANDAS_UNARY_FUNCTIONS = ['isnull', 'notnull'] in ops.py; I'll have a try.

gfyoung · 2017-10-27T17:17:03Z

@gerritholl : Weird that that commit would have caused things to break, since the changes should have been backwards-compatible (all we did was alias).

gerritholl · 2017-10-27T17:20:58Z

I'm not sure if I understand correctly, but it appears xarray has a hardcoded list of names of pandas functions/methods that need to be treated in a particular way. I might be on the wrong track though.

In pandas commit pandas-dev/pandas@7930202 isna and notna were added as aliases for isnull and notnull. Those need to be added to PANDAS_UNARY_FUNCTIONS for xarray datasets notnull to work. Closes pydata#1663.

shoyer · 2017-10-27T19:15:07Z

We actually dealt with this back in #1549, but need to get a new xarray release out promptly so users can upgrade pandas.

gerritholl · 2017-10-27T21:30:08Z

Oh, I missed that. I should have tried with xarray master.

gerritholl changed the title ~~ds.notnull() fails with AttributeError:~~ ds.notnull() fails with AttributeError on pandas 0.21.0rc1 Oct 27, 2017

gerritholl referenced this issue in pandas-dev/pandas Oct 27, 2017

COMPAT: rename isnull -> isna, notnull -> notna (#16972)

7930202

closes #15001

gerritholl mentioned this issue Oct 27, 2017

COMPAT: rename isnull -> isna, notnull -> notna pandas-dev/pandas#16972

Merged

gerritholl mentioned this issue Oct 27, 2017

BUG: Added new names for pandas isna/notna unary functions #1664

Closed

1 task

shoyer mentioned this issue Oct 27, 2017

v0.10 Release #1535

Closed

13 tasks

jhamman closed this as completed Nov 1, 2017

jhamman mentioned this issue Nov 13, 2017

Error accessing isnull() and notnull() on Dataset #1715

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

ds.notnull() fails with AttributeError on pandas 0.21.0rc1 #1663

ds.notnull() fails with AttributeError on pandas 0.21.0rc1 #1663

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!