NumPy ABI does not successfully maintain forward compatibility -- should it? #5888
Comments
|
I'm pretty sure that numpy itself also raises an error when running against an older version than compiled against: https://github.com/numpy/numpy/blob/master/numpy/core/code_generators/generate_numpy_api.py#L90 |
|
This is related to the desire to hide implementation details, which is discussed in http://docs.scipy.org/doc/numpy/reference/c-api.deprecations.html#background in the docs. And maybe in other places? |
|
Some more digging for discussions on the Cython issue: |
|
This has come up again now that pip has started to properly support build-requires (see PEP 518). The problem is that scipy (for example) then has to decide what version of numpy to declare a build-dependency on. If they depend on plain There's lots more discussion here: pypa/pip#4582 Anyway, it seems like what we really want is a way for scipy to tell numpy "I'm using the 1.8 ABI" at build time, and produce a binary that uses the 1.8 ABI, even when building against a newer numpy. That breaks us out of this loop, because scipy can build-require One way this might work:
We already have most of the code we'd need to implement this – every binary that uses the C ABI already gets a magic number compiled in saying which version of the ABI it expects, so if you have a scipy-built-against-numpy-1.8 and import it on numpy 1.13, then numpy 1.13 has a little table that says "ah, this is expecting the 1.8 ABI, which contained these entries, let me export just those ones". The problem is that right now this flexibility is only exposed at import time, not at build time. |
|
Thanks for bringing this up again @njsmith. I've thought about it for a bit just now and have the vague feeling some tricky issue could come up (also remembering the pain of the not working OS X 10.5 SDK), but can't put my finger on it. Implementation of what you propose seems relatively straightforward. |
@njsmith I assume you mean 1.12.x? |
|
Yes. |
|
Out of curiosity, where is the magic number table defined? |
I've just read through that, and agree with most the responses you got from Nathaniel, Thomas and Paul. In general, this isn't really a major issue, it's just something to be aware of at the moment: just build your wheels against the lowest NumPy version you want to support at runtime. That one requirement can be expressed in the current |
I don't think there is any guarantee that this will continue to work for every numpy version ever released in the future. For example, maybe some day numpy 2.0 will be released with a very different API which won't be compatible with packages built against numpy 1.x. So having some way to indicate that would still be useful. By the way, I never claimed that it's a big issue. But I've been bitten by this (not with numpy but with another Python package with a C API). |
Of course, there is no such guarantee. That's a different issue though - the only way to protect against future API or ABI changes is to use |
|
Well, we would definitely increment major version, so most software (with very little risk) should be able to say < 2.0 probably? Not sure that is good habit though. |
But why not? Why can't numpy have some kind of ABI guarantee? In fact, that's already the case in practice. Why not make it official? This will not make it impossible for numpy to change its ABI, it just means that the version needs to be increased to 2.0 if that happens. |
|
@jdemeyer I think we have that guarantee, except in some rare cases were we have a long deprecation beforehand. I suppose I misunderstood the thread maybe then. |
We will do exactly that, and make every effort to keep ABI compatibility for 1.x. Again that has very little to do with either forward compatibility or your pypa thread though. It seems like we're talking past each other here. If we release 2.0 with an incompatible ABI, then one should build against 2.0 as the lowest supported version. That can be done today, simply by putting |
|
We recently ran into this with pycuda + numpy as well. We had in our Reported this against pycuda here: inducer/pycuda#209 A possible workaround from the pycuda side would be for them to pin to the exact oldest version against which it can build (so 1.6 in this case) in its |
|
If I recall correctly, SciPy pins its build system to the oldest numpy it wishes to support, but tests against later numpy versions. |
|
Closing. This is effectively solved by https://pypi.org/project/oldest-supported-numpy/ and build requires. So we don't attempt this, and thats that. |
Not sure whether this is something we consider a bug, but flagging for potential discussion and so it doesn't get lost: I hadn't realized until today that NumPy in practice does not provide a forward compatible ABI, i.e., if you build with numpy 1.9 then try to run against 1.8, this may not work. Apparently packages that care about this like sklearn are actively working around this by carefully installing old versions of numpy before building wheels.
In particular, we have several times added extra fields to the
dtypestruct. In practice this is normally fine b/c no-one actually accesses these fields, but Cython in particular does struct size checking. For backwards compat -- build against 1.8 and then run against 1.9 -- the struct appears to get larger, and Cython merely issues a warning (which we suppress). For forward compat -- build against 1.9 and then run against 1.8 -- the struct appears to get smaller, and in this case Cython issues a hard error.We could work around this by simply exposing a truncated struct to user code, so that Cython sees a small struct when doing
sizeof, and the actual object is always larger then this, meaning that we always hit the warning path rather than the error path.I don't know if this is the only problem we would have to fix in order to achieve forward compatibility, e.g. I haven't checked the C API import code to see what import_multiarray or import_umath do when they find themselves running against an older version of numpy.
If we want to take ABI compatibility seriously I guess we should probably also start requiring C API users to explicitly state which version of the ABI they expect, and enforce that they don't get access to anything newer than that. This would at least give us the option then in the future to provide different versions of the same function to old-users and new-users.
The text was updated successfully, but these errors were encountered: