ENH: Provide PyArray_CastBuffer for numpy 1.x compatibility #27624
Comments
|
Ping @seberg - maybe we could change something about |
|
Removing the maxargs limit is possible, I am not sure how hard it will be. One of those things that is probably not terribly hard, but needs someone to work on concentrated for a while. I'll try to look at the code that uses this function. The removed API was 8000 very odd (most of the code was unused in NumPy!). I suppose a "fix" might be to go back to iterating all arrays indvidiually (with the cast included in the new iterator). I don't mind adding that function, although in this use-case I suspect even that individual iterator setup would probably perform better and be clearer. |
|
Thank you @seberg and @ngoldbaum for looking into this.
I've been searching in the code a little bit. In some cases it seems to that it can be changed by a malloc/free,
I initially tried that, but I couldn't find information in the docs about the casting. Furthermore, astronomical images in FITS format are always stored in big-endian byte order, so I have to bitswap the pointers before casting.
It would work for me, but I don't know how perform the bitswap and casting with the new iterators. Is it in the public API? |
There should be a few places where we have static arrays in functions. All of them would be nice to replace, but I suspect we do need to make sure to not litter the code with actual I suspect the So if you have some time, I would definite review all parts of this, and it would be nice to make progress, even if we don't finish it.
Yes, of course it can do all of that for you! Please don't hesitate to ask (or maybe join the NumPy slack even) if you need more pointers. The path to iterating the arrays individually, would be to just use Unless you know for sure, you do need to pass the order, and you need to know that all arrays have the same shape ahead of time, manually broadcast them (or use the advanced version, which allows passing a shape). The main downside of not using the multi-iter, is that advancing the iterators is just more work. Typically, there is the "External loop" optimization, which allows you to do the inner-most iteration in a simple C for-loop (with strides). Anyway, you still will gain performance, I think. Because buffered casting should offset the downsides (compared to the old code). |
|
@sergiopasra did you ever make progress here or should we discuss/look into it further? (The request to make it public is still good of course, even if I don't really think it would be a great solution here as such.) |
|
Hi, I think I have a workaround for my problem by using a list of Each of them has less than maxargs operands. I have to manage the iters in sync, but it seems to work nicely with my use case. I think I will not need the compatibility API, so this issue can be closed. Thank very much for your suggestions, they helped me a lot. |
|
@sergiopasra maybe not that helpful unfortunately, but NumPy 2.3 should remove the |
Proposed new feature or change:
PyArray_GetCastFuncwas removed in numpy 2.0. In the documentation of the change is it mentioned that:So I'm requesting the creation of such API
PyArray_CastBuffer()so that the functionality ofPyArray_GetCastFunccan be still be used in numpy 2Contex
I'm the maintainer of several processing pipelines for the 10 meter telescope GTC, at la Palma Observatory, Spain.
Sometimes we need to operate over ~100 2000x2000 arrays. I have implemented the operations using
PyArrayIterin C, and it works well, but our current code doesn't compile in numpy 2.0, in part becausePyArray_GetCastFunchas been removed.I have reimplemented our code using
NpyIter_MultiNewand it works well in numpy 2 and the code is cleaner, but then we can't use it because we routinely reach the limit of NPY_MAXARGS (#4398), we would need at least NPY_MAXARGS=256 to recover the functionality we have today.The text was updated successfully, but these errors were encountered: