8000 Increase NPY_MAXARGS to more than 32 · Issue #4398 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

Increase NPY_MAXARGS to more than 32 #4398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
FrancescAlted opened this issue Feb 28, 2014 · 18 comments
Open

Increase NPY_MAXARGS to more than 32 #4398

FrancescAlted opened this issue Feb 28, 2014 · 18 comments
Labels
01 - Enhancement 63 - C API Changes or additions to the C API. Mailing list should usually be notified.

Comments

@FrancescAlted
Copy link

It is quite frequent for some applications (numexpr, but others too) to hit the NPY_MAXARGS limit. You can find a report about this problem here:

PyTables/PyTables#286

Making this number larger (say 256) would alleviate the issue a lot.

PR #226 tries to tackle the problem, but probably just increasing the value would be enough. There has been a recent discussion in the numpy mailing list too:

http://mail.scipy.org/pipermail/numpy-discussion/2014-February/069266.html
@seberg
Copy link
Member
seberg commented Feb 28, 2014

gh-226 is of no use for pytables, which uses the new iterators... MapIter should only use MaxDims I think, not maxargs, so it should be fine (if it uses maxargs we can change it to use maxdims), I don't know about other public stuff like the old multiter changed in gh-226.

@charris
Copy link
Member
charris commented Mar 28, 2014

Where do we sit with this/

@charris
Copy link
Member
charris commented Mar 28, 2014

@seberg @FrancescAlted I have the sense this may have been fixed. Is that the case?

@seberg
Copy link
Member
seberg commented Mar 28, 2014

No, there was no change in ndite regarding this.

juliantaylor added a commit to juliantaylor/numpy that referenced this issue Jul 5, 2014
@juliantaylor
ENH: increase NPY_MAXARGS to 256
5bc8d74 
Allows larger numbers of arguments for nditer using functions. This can
be helpful for complex boolean expressions used in pytables.
Closes numpygh-4398
@juliantaylor juliantaylor mentioned this issue Jul 5, 2014
Closed
@153957 153957 mentioned this issue Aug 10, 2014
Open
@moorepants moorepants mentioned this issue Jul 1, 2015
Closed
@tobgu
Copy link
tobgu commented Feb 9, 2016

What's the status regarding this one? I'm hitting the limit quite frequently while building queries dynamically from pandas/numexpr. I read through #4840 but could not find a clear conclusion there. Raising the limit to 256 would be enough for me personally but removing the threshold completely would of course be the best.

@kmax12
Copy link
kmax12 commented Mar 1, 2016

the limit is an issue I hit frequently as well.

if it is not possible to remove the limit, are there suggested workarounds?

@richardotis richardotis mentioned this issue Jun 6, 2016
Merged
@stevebissett
Copy link

Did anything more come of this?

@hnykda
Copy link
hnykda commented Sep 2, 2017 •
edited
Loading

I have the same problem when using numpyexp in pandas. E.g. this raises too many inputs:

import pandas as pd
cols = ['a_'+str(x) for x in range(0, 50)]
df = pd.DataFrame(pd.np.random.rand(100, 50), columns=cols)
df.eval('+'.join(cols[:33]))

while df.eval('+'.join(cols[:31])) works.

Why is there such limitation?

Fortunately, it is possible to overcome this in pandas by passing engine='python' (which will be probably slower): df.eval('+'.join(cols[:33]), engine='python')

Numpy 1.12.1

@Rishav1
Copy link
Rishav1 commented Feb 7, 2019

The limit is way too low in my opinion as well. I use the following hack from it's userdocks sometimes.

np.choose(a,c) == np.array([c[a[I]][I[:1]] for I in ndi.ndindex(a.shape)])

@charris
Copy link
Member
charris commented Feb 7, 2019

Sounds like this needs an entry in NumPy 2.0 .

@seberg
Copy link
Member
seberg commented Feb 7, 2019

Good plan (I see you already did, although I guess it isn't technically backward incompatible). I wonder if now with C99 which makes not-fixed width stack arrays a bit more wieldy we could actually think about relaxing it in specific places (such as NpyIter).

@shoyer
Copy link
Member
shoyer commented Feb 8, 2019

There are still cases where using small fixed sized arrays are advantageous due to speed (they can be allocated on the stack), e.g., inside dispatch for __array_ufunc__ and __array_function__. But this is definitely the exception, not the normal. Certainly increasing NPY_MAXARGS to 256 seems like a simple win.

@seberg seberg added the 60 - Major release Issues that need or may be better addressed in a major release label May 2, 2019
@shoyer shoyer added the 63 - C API Changes or additions to the C API. Mailing list should usually be notified. label May 10, 2019
@vss888
Copy link
vss888 commented Aug 19, 2021

At least 9 years later, the issue is still there.

@mattip
Copy link
Member
mattip commented Aug 19, 2021

In trying to find the email list discussion, it seems like numpy-discussion.10968.n7.nabble.com no longer is available (maybe temprary?) and looking at the Feb 2014 mailing list archives (as hinted in the link at the top of the issue) I could not find the discussion. Nor could find a relevant discussion from the link at the top of the closed PR #226 from 2012.

@vincentsarago vincentsarago mentioned this issue May 24, 2023
Closed
@seberg seberg removed the 60 - Major release Issues that need or may be better addressed in a major release label Jun 29, 2023
@seberg
Copy link
Member
seberg commented Jun 29, 2023

I am removing the "major release" again. There is no real ABI issue with this (unlike NPY_MAXDIMS). We could bump it, but honestly, in most places its used it probably isn't even super hard to just replace it.

We could still consider just deleting it from the public API, nobody needs this anyway (there is one place where it is used, publicly, but its at the end of a struct).

@FrancescAlted FrancescAlted mentioned this issue Feb 15, 2024
Closed
@rgommers
Copy link
Member

gh-25149 increased NPY_MAXARGS to 64. Release note entry for it: https://numpy.org/devdocs/release/2.0.0-notes.html#npy-maxargs-not-constant-and-pyarraymultiiterobject-size-change.

@seberg can this issue be closed, or did you leave it open on purpose?

@seberg
Copy link
Member
seberg commented Jun 25, 2024

I had left it open, because there is an argument to remove the limit entirely or increase it more (at least for the main iterator).
But we could also close it for now and reopen or make a new comment when it proofs insufficient (I am sure there are some users who will like even more).

@LudovicArnould1 LudovicArnould1 mentioned this issue Aug 21, 2024
Closed
@sergiopasra sergiopasra mentioned this issue Oct 22, 2024
Closed
@sergiopasra sergiopasra mentioned this issue Oct 23, 2024
Closed
@seberg
Copy link
Member
seberg commented Dec 31, 2024

gh-28080 will remove any limitation from the iterator. Once that is done, one can look at removing the limitation from ufuncs as well (I am not sure if this issue is about ufuncs or just the iterator originally).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
01 - Enhancement 63 - C API Changes or additions to the C API. Mailing list should usually be notified.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ENH: increase NPY_MAXARGS to 256 juliantaylor/numpy
0