8000 Register class a subclass of NumPy array · Issue #4072 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

8000 Register class a subclass of NumPy array #4072

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hgrecco opened this issue Nov 24, 2013 · 13 comments
Closed

Register class a subclass of NumPy array #4072

hgrecco opened this issue Nov 24, 2013 · 13 comments

Comments

@hgrecco
Copy link
hgrecco commented Nov 24, 2013

Some packages use isinstance(x, numpy.ndarray) to check if a given object can be used as an ndarray. This fails (of course) for object from classes derived from object even if they implement all numpy methods and attributes. It would be good to be able to register a class as a ndarray subclass using abc

@njsmith
Copy link
Member
njsmith commented Nov 24, 2013

Lots of code also uses isinstance to check if an object actually is an
ndarray, and this is important (e.g. before calling C functions that depend
on ndarray's memory layout). I don't think we can change isinstance. And
how do you know what exactly the code that calls isinstance actually
depends on anyway? ndarray's python level attributes? Ufunc compatibility?
np.percentile working?

Most code should either call asarray or just start using the object, if it
wants to work with other array like types.
On 24 Nov 2013 08:59, "Hernan Grecco" notifications@github.com wrote:

Some packages use isinstance(x, numpy.ndarray) to check if a given object
can be used as an ndarray. This fails (of course) for object from classes
derived from object even if they implement all numpy methods and
attributes. It would be good to be able to register a class as a ndarray
subclass using abc


Reply to this email directly or view it on GitHubhttps://github.com//issues/4072
.

@charris
Copy link
Member
charris commented Nov 24, 2013

I don't think ndarray was designed for duck typing and

8000
@charris charris closed this as completed Nov 24, 2013
@hgrecco
Copy link
Author
hgrecco commented Nov 25, 2013

Thanks for taking the time to answer this.

For the purpose that I have described, there is no real difference between guarding with isinstance or using asarray: objects that are capable of the same things as ndarray are converted into ndarray.

I wrote a small toy code to demonstrate the problem:

import numpy as np

class S(np.ndarray):
    pass

class W(object):

    def __init__(self, value):
        self.value = value

    def __getattr__(self, name):
        return getattr(self.value, name)

    def __getitem__(self, key):
        return self.value.__getitem__(key)

a = np.arange(1, 10)
w = W(np.arange(1, 10))
s = S(a.shape)
s[:] = np.arange(1, 10)

func = np.diff

da = func(a)
dw = func(w)
ds = func(s)

print('Numpy version', np.__version__)
print('Array', np.asanyarray(a), da.shape, type(da))
print('Wrapping', np.asanyarray(a), dw.shape, type(dw))
print('Subclassing', np.asanyarray(a), ds.shape, type(ds))

The output is:

('Numpy version', '1.7.1')
('Array', array([1, 2, 3, 4, 5, 6, 7, 8, 9]), (8,), <type 'numpy.ndarray'>)
('Wrapping', array([1, 2, 3, 4, 5, 6, 7, 8, 9]), (8,), <type 'numpy.ndarray'>)
('Subclassing', array([1, 2, 3, 4, 5, 6, 7, 8, 9]), (8,), <class '__main__.S'>)

Notice that for Wrapping the output is an instance of ndarray instead of an instance of W. This is because asanyarray is called within diff and it returns a ndarray.

I also understand that I can roll my own diff to work with other classes. But this means that all the code that relies on numpy.diff (or any other function that calls asanyarray) will not work properly; and the only solution will be to monkey patch numpy.diff.

@hgrecco
Copy link
Author
hgrecco commented Nov 25, 2013

@charris registering using the mechanisms provided by the Abstract Base Class module is not the same as duck typing.

@seberg
Copy link
Member
seberg commented Nov 25, 2013

Not sure I understand, but if your wrapping class implements __array_wrap__ most of these things should just work.

@hgrecco
Copy link
Author
hgrecco commented Nov 25, 2013

Depends what do you mean by work. It does return the right values, but not the right class.

Just replace the last 4 lines of the code above by:

print('Numpy version', np.__version__)
print('Array', type(np.asanyarray(a)), da.shape, type(da))
print('Wrapping', type(np.asanyarray(w)), dw.shape, type(dw))
print('Subclassing', type(np.asanyarray(s)), ds.shape, type(ds))

The output is:

('Numpy version', '1.7.1')
('Array', <type 'numpy.ndarray'>, (8,), <type 'numpy.ndarray'>)
('Wrapping', <type 'numpy.ndarray'>, (8,), <type 'numpy.ndarray'>)
('Subclassing', <class '__main__.S'>, (8,), <class '__main__.S'>)

Maybe I am forgetting to define something? My interpretation is that because W is not a subclass of ndarray, a asanyarray(w) creates a new array with the content (buffer) of w.

@rkern
Copy link
Member
rkern commented Nov 26, 2013

I would not turn ndarray itself into an ABC. It's way too big of an interface. Typically, ndarray-like-but-not-actual-subclasses only implement a tiny fraction of the whole ndarray interface and usually none of the C struct implementation that most of these type-checks for ndarray are actually requesting. The best we could do is to come up with a set of smaller ABCs that represent useful subsets of the ndarray interface that could be usefully implemented by other types that do not subclass from ndarray.

None of this will solve your wrapper problem, though. You need to implement __array_wrap__ in your wrapper class, as @seberg suggests.

@hgrecco
Copy link
Author
hgrecco commented Nov 26, 2013

@rkern I am not sure that I follow. What I am suggesting is the ability to register an unrelated class as a subclass of ndarray using the mechanism provided by the abc module or similar. If you come up with smaller ABCs, then I can register a class and I am done.

Additionally, __array_wrap__ IS implemented in the code shown above (via __getattr__) and it does not work. You can even do it explicitly:

    def __array_wrap__(self, out_arr, context=None):
        return self.value.__array_wrap__(out_arr, context)

and you will get the same result.

@rkern
Copy link
Member

I am saying that registering a non-subclass as an implementation of ndarray is just a non-starter. The interface is far too big. ndarray is not a good ABC, and we will not try to make it one. Not least, the C implementation of ndarray is part of its interface that most of these strict isinstance(x, np.ndarray) checks are looking for. Defeating that check by being registering classes that will almost never implement that C interface is just looking for trouble.

You might be able to make good, small ABCs that are subsets of the Python-side ndarray interface, and I wish you luck in that.

As for implementing __array_wrap__, our suggestion is that you need to explicitly implement a correct __array_wrap__ for your desired semantics. In your case, to return an instance of your wrapper class rather than an ndarray.

@hgrecco
Copy link
Author
hgrecco commented Nov 26, 2013

@rkern Thanks for taking the time to answer the idea about registering

But there is one thing that I still do not get about __array_wrap__

Changing the code to:

    def __getattr__(self, name):
        print(name)
        return getattr(self.value, name)

__array_wrap__ does not seem to be used. Only __array_struct__.

@rkern
Copy link
Member
rkern commented Nov 26, 2013

Yes, sorry. __array_wrap__ doesn't work. np.asanyarray() allows only strict subclasses.

@hgrecco
Copy link
Author
hgrecco commented Nov 27, 2013

Then it seems that the only real option is to subclass.

@eric-wieser
Copy link
Member
eric-wieser commented Nov 17, 2016

__array_wrap__ IS implemented in the code shown above (via __getattr__)
[...]
__array_wrap__ does not seem to be used.

I think what you're seeing here is __array_wrap__ is looked up on the class __dict__ like other magic methods, not through the usual attribute lookup scheme on the instance. There's not much you can do about this - you're forced to either define each method manually, or create proxy methods automatically in the metaclass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants
0