Increase maximum number of array dimensions? #5744

wmayner · 2015-04-03T23:14:45Z

At my lab, we're working with arrays with very many dimensions. We've run up against the hardcoded limit of 32 dimensions for np.arrays. What is the rationale for this limit, and is it possible to increase it? Thanks!

The text was updated successfully, but these errors were encountered:

charris · 2015-04-04T00:01:36Z

This has been discussed before, but is not completely trivial. I'd point you to the mailing list archives, but my browser currently crashes on http://dir.gmane.org/gmane.comp.python.numeric.general.

njsmith · 2015-04-05T05:28:16Z

Mostly the reason we have a finite limit is laziness -- there are a number
of places in the code that just statically allocate a MAX_DIM buffer
instead of writing slightly more complicated code using malloc/free. I
think we'd be happy to accept patches to remove the limit, and this could
be done incrementally, one place at a time. It shouldn't be difficult, just
a bit tedious.

Given the current situation where we do have a single static limit, I think
we hesitate a bit to just bump up that limit, because once we try handling
all the weird 0.1% long-tail use cases then we'll keep having to bump the
limit up arbitrarily far, and that impacts all the users who don't need
massively high dimensions.
On Apr 3, 2015 4:14 PM, "Will Mayner" notifications@github.com wrote:

At my lab, we're working with arrays with very many dimensions. We've run
up against the hardcoded limit of 32 dimensions for np.arrays. What is
the rationale for this limit, and is it possible to increase it? Thanks!

—
Reply to this email directly or view it on GitHub
#5744.

See the following references: - https://stackoverflow.com/questions/22747068/is-there-a-max-number-of-arguments-javascript-functions-can-accept - https://bugs.webkit.org/show_bug.cgi?id=80797 - numpy/numpy#5744 Note that the maximum number of function arguments can vary from engine to engine. Here, we choose something of a lowest common denominator which may not be valid everywhere.

wmayner · 2020-05-01T05:45:22Z

Hello again (5 years later)!

We're running into this problem again. It seems that in the meantime it's been noticed by others.

@njsmith, I could try to implement a dynamic solution as you described, but I don't have a lot of experience with C and I have no experience with NumPy internals. Looks like NPY_MAXDIMS appears in the source 234 times; the tedium isn't a problem, but it's possible that this would change behavior meaningfully somewhere. Is this still feasible? If it's still worth trying, it would be a huge help if you or another maintainer could describe the approach you had in mind in more detail.

@jcmgray, @mrocklin, @jrbourbeau, would this be useful for Dask beyond dask/dask#5595?

seberg · 2020-05-01T14:32:43Z

I think the biggest/toughest chunk for relaxing either MAX_ARGS or MAX_NDIM will be the NPY_ITER code. But as an experiment, it may be better to try with some other functions and see how it goes/what the impact is before diving into that directly (more as a trial balloon).

I do not know if the heap allocations would mean overhead, but I assume that can be made so low, that it just doesn't matter. Or should we consider again to simply bump up NPY_MAXDIMS slightly, because the use-case is different from one that adds many dimensions of size 1?

charris · 2020-05-01T15:26:32Z

I suspect that the original 32 came from 2*32 for 32 bit systems, so 64 might be the proper number for MAX_DIM these days, although actual memory address buses were about 48 bits last time I looked.

jrbourbeau · 2020-05-01T17:13:49Z

would this be useful for Dask beyond dask/dask#5595?

Yes, the maximum number of dimensions impacts dask arrays (at least those backed by numpy arrays) outside of tensordot. For example:

dask.array.empty example:

In [1]: import dask.array as da

In [2]: x = da.empty((1,) * 33)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
~/github/dask/dask/dask/array/utils.py in meta_from_array(x, ndim, dtype)
     90             if ndim > x.ndim:
---> 91                 meta = meta[(Ellipsis,) + tuple(None for _ in range(ndim - meta.ndim))]
     92                 meta = meta[tuple(slice(0, 0, None) for _ in range(meta.ndim))]

IndexError: invalid index to scalar variable.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-2-021c43ce84db> in <module>
----> 1 x = da.empty((1,) * 33)

~/github/dask/dask/dask/array/wrap.py in wrap_func_shape_as_first_arg(func, *args, **kwargs)
     70
     71     dsk = dict(zip(keys, vals))
---> 72     return Array(dsk, name, chunks, dtype=dtype)
     73
     74

~/github/dask/dask/dask/array/core.py in __new__(cls, dask, name, chunks, dtype, meta, shape)
   1053             raise ValueError(CHUNKS_NONE_ERROR_MESSAGE)
   1054
-> 1055         self._meta = meta_from_array(meta, ndim=self.ndim, dtype=dtype)
   1056
   1057         for plugin in config.get("array_plugins", ()):

~/github/dask/dask/dask/array/utils.py in meta_from_array(x, ndim, dtype)
     96                 meta = meta.reshape((0,) * ndim)
     97     except Exception:
---> 98         meta = np.empty((0,) * ndim, dtype=dtype or x.dtype)
     99
    100     if np.isscalar(meta):

ValueError: maximum supported dimension for an ndarray is 32, found 33

That said, it doesn't seems to be a limitation lots of users are running into (dask/dask#5595 is the only reported issue I'm aware of)

Just out of curiosity, @wmayner how many dimensions are you typically dealing with? Would the previous suggestion from @charris help?

wmayner · 2020-05-01T23:55:29Z

@jrbourbeau, we likely wouldn't need more than 64 dimensions for now, so if increasing to 64 is an option, @charris, that would be great.

In case it's helpful to motivate the change, I'll describe our use case:

We're representing the probabilities of state transitions over a single timestep in a discrete dynamical system. If the system has N elements with n_i states each, then there are S = n_0 * n_1 * ... * n_(N - 1) states of the system, and the joint distribution of states at t = 0 and states at t = 1 has S * S elements.

Our approach is to use separate dimensions to index the state of each element at t = 0 and t = 1, so we need 2 * N dimensions. This is useful because in our calculations we need to multiply different marginal distributions together; in many cases these distributions are over the states of distinct but overlapping sets of elements, so it isn't a straightforward outer product. But if each element has its own dimension, then we can take advantage of broadcasting semantics and just use the * operator to implement the product we need, which is efficient and conceptually simple. (If anyone has any suggestions for a more efficient way to do this, let me know!)

Increasing the limit to 64 would allow us to represent systems of up to 32 elements instead of 16, which would be a qualitatively significant improvement for our research questions.

@njsmith's point about hesitating to increase the limit for long-tail cases is well taken, though. But if the overhead is very low, then increasing to 64 would be a simple stopgap solution until we can make the allocations dynamically.

charris · 2020-05-02T00:58:39Z

@wmayner That sounds like something that could be more flexibly handled by some custom code.

seberg · 2020-05-02T01:30:16Z

I would like to know what @jcmgray reason was for opening the dask issue on the limited number of dimensions numpy/dask support and whether that is related to memory size (something like 2**48), etc.

jcmgray · 2020-05-02T03:58:26Z

Hi, yes I have run up against this limit quite a few times.

As some technical context, basically when simulating quantum systems its natural to represent each system as a dimension (rather than manually 'vectorizing' them into a joint dimension). This is particularly true for tensor networks (basically graphs with an array at each node, with ndim given by the number of incident edges), where the number of dimensions of each array changes pretty dynamically throughout manipulations.

In a very common case each dimension is of size 2, and I guess this is a key point: 2^32 * float32 e.g. is still only ~16GB so even for straight numpy arrays well within what would be tractable in a fairly moderate HPC setting.

In principle dask might be a great, simple way to handle these sizes of array out-of-core/in a distributed fashion, but because 'chunked' dimensions are still dimensions (even if size 1), the ndim limit comes into play.

The problem is particularly compounded for dask.array.tensordot (the key function for tensor networks!), for which the current implementation first computes the outer product, then sums over the shared dimensions. Meaning that even if for z = tensordot(x, y, axes=....) each array x, y and z all have ndim<32, the intermediate array can break the limit.

I should say, this isn't a blocker for me right now, and these borderline HPC cases are understandably I guess not numpy's focus. But it's certainly possibly to bump up against this particular limit quite easily in the quantum setting.

mattip · 2020-05-02T08:06:48Z

If I understand correctly, the shape of a ndarray as described would contain only zeros and ones, otherwise memory use would explode. This would be better served by a different data structure, and is not a sufficient reason to fundamentally change ndarray.

Any proposal to make the number of dimensions flexible would need to be carefully benchmarked and should come with a cache of vectors that could be reused rather than repeatedly calling malloc/free.

matthew-brett · 2020-05-02T10:21:35Z

What would be the cost of just bumping MAX_DIMS up to 64?

jcmgray · 2020-05-02T19:05:52Z

If I understand correctly, the shape of a ndarray as described would contain only zeros and ones, otherwise memory use would explode. This would be better served by a different data structure, and is not a sufficient reason to fundamentally change ndarray.

If this was in reference to my usecase - the arrays are generally filled with complex numbers, and yes, the memory does explode! To be clear, my view is just that a moderate increase in MAX_DIM, to e.g. 64, would be potentially useful.

cmatt23 · 2020-05-22T00:04:01Z

I've been running trade studies on how different models interact with each other. Each dimension in my results array is linked to a particular input parameter (whether that is a single value or a range of values). Since I am using multiprocessing pools to run the simulations in parallel, my problem with the 32 dimension limit arises when I return the data from the pool and reshape it to match the inputs (+ a couple dimensions for outputs).

So far my workaround has been to write different versions of the same functions to evaluate the effects of different combinations of inputs independently.I would also appreciate an update to this, even if just to 64 dimensions to start with. Thanks!

seberg · 2020-05-22T00:06:40Z

To be clear, the "64 to start with" is exactly the thing that makes this harder to just decide on for me at least... 64 is the absolute plausible limit at this time, I would personally prefer much less, e.g. 40 or 48 since both should be plenty in a 2**N scenario.

eric-wieser · 2020-05-22T05:51:07Z

Each dimension in my results array is linked to a particular input parameter (whether that is a single value or a range of values).

Have you considered using xarray, which lets you name your dimensions, meaning you do not need to insert padding dimensions of length 1?

jcmgray · 2020-10-28T19:57:47Z

As noted here, this does also currently limit the total number of indices involved in an numpy.einsum expression, which is a much easier threshold to breach:

import numpy as np
import string

d = 11
# total dims involved = 3 * d
# but max number of array dims = 2 * d

x = np.random.uniform(size=[2] * 2 * d)
y = np.random.uniform(size=[2] * 2 * d)

chars = string.ascii_lowercase + string.ascii_uppercase
eq = f"{chars[0 * d:2 * d]},{chars[1 * d:3 * d]}"
print(eq)
# abcdefghijklmnopqrstuv,lmnopqrstuvwxyzABCDEFG
#            ........... ...........
#                  contracted

z = np.einsum(eq, x, y)
# ValueError: too many subscripts in einsum

(n.b. this is just the simplest illustrative equation - I know one could use tensordot for this specific case).

The following if (ndim_iter >= NPY_MAXDIMS) { part of c_einsum is the relevant numpy code:

numpy/numpy/core/src/multiarray/einsum.c.src

Lines 947 to 964 in 4cba2d9

    
           /* 
        
            * Set up the labels for the iterator (output + combined labels). 
        
            * Can just share the output_labels memory, because iter_labels 
        
            * is output_labels with some more labels appended. 
        
            */ 
        
           iter_labels = output_labels; 
        
           ndim_iter = ndim_output; 
        
           for (label = min_label; label <= max_label; ++label) { 
        
               if (label_counts[label] > 0 && 
        
                       memchr(output_labels, label, ndim_output) == NULL) { 
        
                   if (ndim_iter >= NPY_MAXDIMS) { 
        
                       PyErr_SetString(PyExc_ValueError, 
        
                                   "too many subscripts in einsum"); 
        
                       goto fail; 
        
                   } 
        
                   iter_labels[ndim_iter++] = label; 
        
               } 
        
           }

(Possibly for this particular problem there are also other fixes other than increasing NPY_MAXDIMS.)

I do fairly regularly come across this limitation in real contractions! Currently I need to switch to e.g. ctf in these cases.

mofeing · 2021-07-19T09:10:55Z

As many in this post, I frequently bump into this issue. In the end, I forked and just increased the maximum number of dimensions to 64. I just had to update some tests but it works flawlessly.

As noted here, this does also currently limit the total number of indices involved in an numpy.einsum expression, which is a much easier threshold to breach:

import numpy as np
import string

d = 11
# total dims involved = 3 * d
# but max number of array dims = 2 * d

x = np.random.uniform(size=[2] * 2 * d)
y = np.random.uniform(size=[2] * 2 * d)

chars = string.ascii_lowercase + string.ascii_uppercase
eq = f"{chars[0 * d:2 * d]},{chars[1 * d:3 * d]}"
print(eq)
# abcdefghijklmnopqrstuv,lmnopqrstuvwxyzABCDEFG
#            ........... ...........
#                  contracted

z = np.einsum(eq, x, y)
# ValueError: too many subscripts in einsum

@jcmgray Your code gives me no problem now and I didn't need to change einsum.c.src

I want to do a PR but first wanted to ask if there is still any blocking issue.

isacdaavid · 2023-03-27T18:01:11Z

Is there any update regarding this? Willingness to accept @mofeing or someone else's patches?

Xarray is far from being a real alternative. Even though I can use a DataArray's named dimensions to do without singleton dimensions, the underlying data still is an ndarray, so switching to xarray simply pushes the limit from 32 dimensions in total to 32 nontrivial dimensions.

mattip · 2023-03-28T01:01:04Z

This might be something to consider for a NumPy 2.0 release

seberg · 2023-06-29T12:22:00Z

OK, there be problems and someone needs to step up to attack them. We can bump up NPY_MAXDIMS, but it comes with two problems:

If a downstream library uses it in their headers, things could get awkward, but this seems unlikely/rare enough that it is probably fine in practice (the library should hardcode its own limit). ( I am not actually worried about this)
We are ABI locked in to NPY_MAXDIMS in our legacy iterators. While not impossible its difficult to change this:
- All macros could account for the numpy version, but this seems tedious and potentially costly?
- There could be a new "legacy" iterator with (mostly) identical API. In theory, we could swap it in with the numpy2_compat idea. I am fine with a numpy2_compat, it just need some work.
- We could accept that at least downstream will have functions that only support 32dims for the time being but ship a new API anyway and use it internally.

@mattip doesn't like it, but this makes me wonder if it isn't a good first step to just delete (or rename) NPY_MAXDIMS and NPY_MAXARGS make them internal and rename them for their use in the public iterator ABI/API.

The purpose would be only to remove the chicken-egg problem by removing the promise that there is a MAXDIMS constant you can rely on (code that uses it will have to add a check). That way, a future person can create that new iterator and start using it within NumPy right away without less danger of downstream segfaulting for very large arrays.

Unless we don't care about downstream segfaulting for those, but even then :)

NoureldinYosri · 2023-06-29T17:40:23Z

@seberg from your comment it seems that there is no technical hard blockers inside numpy itself and the fear is of potentially breaking downstream libraries.

If pumping up the number of dimensions is too risky then maybe an alternative would be to create a different version of numpy that supports the 64 dimensions, kinda like how we can do pip install jax[cpu] and pip install jax[cuda], pip install jax[cuda11_pip] and pip install jax[cuda12_pip] to differntiate between jax on CPU and jax on GPU and which cuda version to use. so pip install numpy would install the usual numpy and pip install numpy[64] would install the 64-dimension version.

At code level the NPY_MAXDIMS and NPY_MAXARGS macros would be guarded by

#ifndef NPY_MAXDIMS
#define NPY_MAXDIMS 32
#endif
#ifndef NPY_MAXARGS
#define NPY_MAXARGS 32
#endif

This keeps the old behaviour of the code with NPY_MAXARGS defined and equal to 32 for downstream libraries. while for the 64 version we would just need to add an extra -D NPY_MAXARGS=64 -D NPY_MAXDIMS=64 option to the compilation command.

If this is okay and nothing is expected to break then I can send a PR with this solution.

Note: this is blocking Cirq from supporting simulations for circuits with more than 32 qubits quantumlib/Cirq#6031

seberg · 2023-06-29T18:00:36Z

technical hard blockers inside numpy itself and the fear is of potentially breaking downstream libraries

That is the technical hard blocker. A special install doesn't work, it is the opposite problem from cupy: you would need a scipy[64] if numpy[64]. You can do that if you compile everything yourself, but you won't manage to package it.
Even if it did, it sounds more complicated than writing a new iterator and putting it into a numpy2_compat library.

I am OK with just saying some of scipy can only do 32 dims, going ahead fast would also means some of NumPy may not (unless someone can promise to work on it).
I am also fine with just not doing anything right now except removing the promise that NumPy arrays have at most 32 dimensions (i.e. deleting NPY_MAXDIMS from the public API), but Matti isn't wrong that this would be more convincing if someone were already indicating that they will work on actually making it happen.

NoureldinYosri · 2023-06-29T18:16:18Z

I am OK with just saying some of scipy can only do 32 dims

I think this is okay for our use case (cirq simulation).

Is there a list of things within numpy that will break with 64 dimensions?

seberg · 2023-06-29T19:33:46Z

The public iterator API/ABI for the "old style" iterator https://github.com/numpy/numpy/blob/main/numpy/core/include/numpy/ndarraytypes.h#L1139 and the macros blow mostly (there is a multiiter, and a few other iterators although those matter less).

mattip · 2023-06-30T09:20:09Z

After sleeping on this, maybe it is OK to error out if using the old style iterator (i.e. PyArray_IterNew) with an array that is too big? I think we would also have to refactor some of SciPy to avoid this, it seems to be used quite extensively there. Maybe we could provide a migration guide for people still using the simpler iterator.

seberg · 2023-06-30T11:37:23Z

I don't think SciPy has a lot, maybe even just in ndimage, we have quite a lot within NumPy, pandas has a bit (but shouldn't care about high dimensions), etc...

mattip · 2023-06-30T13:10:05Z

I get quite a few hits when doing git grep PyArray_IterNew in scipy

NoureldinYosri · 2023-06-30T13:14:59Z

Are the "old style" iterator planned to be supported or dropped in future versions? If it will be dropped then it doesn't make sense to make it support 64 dimensions.

I think a first step would be to provide it through special install as I suggest earlier #5744 (comment) this will allow the people who need this (e.g. quantum computing community) to use and test the library and see if anything breaks. Eventually the change can be made permanent in numpy v2 as @mattip suggested in #5744 (comment).

This way we unblock Cirq and similar libraries allowing researchers to simulate quantum systems with more than 32 qubits, while giving time to other libraries to update their code to support the 64 dims if they need to.

What do you think?

mattip · 2023-06-30T13:39:32Z

I think a first step would be to provide it through special install

As @seberg stated previously, that is only the tip of the iceberg. You now need to rebuild special versions of all the other libraries that use NPY_MAXARGS: we already know that SciPy, and Pandas use it (or implicitly use it via PyArray_IterNew). Since this is all open source, I think you could do the experiment: change the define, build NumPy and let us know how it goes.

seberg · 2023-06-30T14:08:54Z

My suggestion is the following:

We remove NPY_MAXDIMS and renaming it to something that is a mouth-full like NPY_OLD_STYLE_ITER_MAXDIMS
- I can accept doing this even if we end up not increasing the limit. Just so I can tell those that want to bump it: It's on you, we need some work to be done before that can happen. (without bumping, there is the "needs major release" hurdle additionally)
Someone replaces the current legacy iterator within NumPy with a size agnostic one, which we make public (this is the work that needs to be done)
- If we find a enough motivation and dev cycles, we can backport that into a/the numpy2_compat package which would allow us to make all of downstream compatible for them! (still feels magical to me that this can be done, but it's possible :); there might be very niche failures possible, but they would have work arounds)
- If we do not find enough dev-cycles, SciPy, we will just accept that some code in downstream libs won't work for super high dimensional arrays. If that is a problem for "Cirq", etc. then you have to work on the previous point.

Even if it is possible to ship wheels in such a way (and I doubt it is), you would double the amount of wheels that a lot of downstream needs to build. I am pretty sure it's easier to do the full thing I said above (including that numpy2_compat library).

NoureldinYosri · 2023-06-30T15:37:28Z

I built the library with 64 instead of 32 and it works fine

>> np.ones((1,)*64, dtype=np.complex128)
array([[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[1.+0.j]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]])

Running the tests, most of them pass. Most of the few that fail seem to be asserting the number of dimensions <= 32 so they should be fine to modify.

Failed Tests

FAILED numpy/core/tests/test_conversion_utils.py::TestIntpConverter::test_too_many_dims - Failed: DID NOT RAISE <class 'ValueError'>
FAILED numpy/core/tests/test_indexing.py::TestIndexing::test_too_many_fancy_indices_special_case - AssertionError: IndexError not raised by __getitem__
FAILED numpy/core/tests/test_indexing.py::TestIndexing::test_too_many_advanced_indices[1-32-True] - Failed: DID NOT RAISE <class 'IndexError'>
FAILED numpy/core/tests/test_indexing.py::TestIndexing::test_too_many_advanced_indices[1-32-False] - Failed: DID NOT RAISE <class 'IndexError'>
FAILED numpy/core/tests/test_indexing.py::TestIndexing::test_too_many_advanced_indices[1-40-True] - Failed: DID NOT RAISE <class 'IndexError'>
FAILED numpy/core/tests/test_indexing.py::TestIndexing::test_too_many_advanced_indices[1-40-False] - Failed: DID NOT RAISE <class 'IndexError'>
FAILED numpy/core/tests/test_indexing.py::TestIndexing::test_too_many_advanced_indices[32-32-True] - Failed: DID NOT RAISE <class 'IndexError'>
FAILED numpy/core/tests/test_indexing.py::TestIndexing::test_too_many_advanced_indices[32-32-False] - Failed: DID NOT RAISE <class 'IndexError'>
FAILED numpy/core/tests/test_indexing.py::TestIndexing::test_too_many_advanced_indices[32-32-index2] - Failed: DID NOT RAISE <class 'IndexError'>
FAILED numpy/core/tests/test_indexing.py::TestIndexing::test_too_many_advanced_indices[32-40-True] - Failed: DID NOT RAISE <class 'IndexError'>
FAILED numpy/core/tests/test_indexing.py::TestIndexing::test_too_many_advanced_indices[32-40-False] - Failed: DID NOT RAISE <class 'IndexError'>
FAILED numpy/core/tests/test_multiarray.py::TestZeroRank::test_invalid_newaxis - AssertionError: IndexError not raised by subscript
FAILED numpy/core/tests/test_multiarray.py::TestScalarIndexing::test_invalid_newaxis - AssertionError: IndexError not raised by subscript
FAILED numpy/core/tests/test_multiarray.py::TestNewBufferProtocol::test_error_too_many_dims - AssertionError: RuntimeError not raised by array
FAILED numpy/core/tests/test_numeric.py::TestBroadcast::test_number_of_arguments - AssertionError: ValueError not raised by broadcast
FAILED numpy/core/tests/test_overrides.py::TestGetImplementingArgs::test_too_many_duck_arrays - AssertionError: TypeError not raised
FAILED numpy/core/tests/test_overrides.py::TestArrayFunctionImplementation::test_too_many_args - AssertionError: Regex pattern 'maximum number' does not match "no implementation found for 'numpy.core.tests.test_overrides.func' on types that implement __array_function__: ...
FAILED numpy/core/tests/test_regression.py::TestRegression::test_array_ndmin_overflow - AssertionError: ValueError not raised by <lambda>
FAILED numpy/core/tests/test_regression.py::TestRegression::test_frompyfunc_many_args - AssertionError: ValueError not raised by frompyfunc
FAILED numpy/core/tests/test_umath.py::test_outer_exceeds_maxdims - AssertionError: ValueError not raised

I think that having some downstream libraries support only 32 dimensions not the full 64 dimensions is fine as a starting point. this in addition to replacing NPY_MAXDIMS with NPY_OLD_STYLE_ITER_MAXDIMS as @seberg suggested should be fine at least to get the ball rolling.

seberg · 2023-06-30T16:07:07Z

It would be nice to at least start fixing NumPy internals which will also remain limited to 32 dimensions before we bump it though.

mattip · 2023-06-30T20:04:41Z

I built the library with 64 F986 instead of 32 and it works fine

Cool. Can the large arrays be used in the rest of your daily work or do you need to also rebuild everything else?

… wrapping it This is to avoid the numpy limit on the number of dimensions quantumlib/Cirq#6031 The 1D representation should only be used when the number of qubits is greater than the numpy limit on the number of dimensions (currently set to 32) numpy/numpy#5744. ```py3 _, state_vector, _ = s.simulate_into_1d_array(c) ``` fixes quantumlib/Cirq#6031

seberg · 2023-11-29T12:46:07Z

Closing, as it's bumped on main and hopefully nobody runs into huge issues. However, I did not bother to try to fix all paths, because it's annoying, since the iterators rely on a certain ABI and are used both inside NumPy and publically.

So if you care about this, check the PR and try to stuff the wholes that fail...

weinbe58 mentioned this issue Feb 15, 2019

reshaping for entropy calculation fails for system sizes N>31 QuSpin/QuSpin#176

Closed

Nucs mentioned this issue Jun 14, 2019

[Discussion] Ground Rules and Library Structure/Architecture SciSharp/NumSharp#284

Open

jcmgray mentioned this issue Nov 15, 2019

tensordot and maximum number dimensions dask/dask#5595

Closed

mofeing mentioned this issue Apr 14, 2021

numpy limits the maximum tensor rank to 32 indexes bsc-quantic/rosnet#1

Closed

23pointsNorth mentioned this issue Nov 24, 2021

High dimensional search space limit SimonBlanke/Gradient-Free-Optimizers#19

Closed

rht mentioned this issue Mar 11, 2023

cirq.sample_state_vector fails when the number of qubits > 32 quantumlib/Cirq#6031

Closed

jcmgray mentioned this issue Mar 31, 2023

ENH: speed up einsum with optimize using batched matmul #23513

Open

3 tasks

NoureldinYosri mentioned this issue Aug 16, 2023

Create read only simulation state vector quantumlib/Cirq#6248

Closed

mgunyho mentioned this issue Aug 18, 2023

Consistently report all dimensions in error messages if invalid dimensions are given pydata/xarray#8079

Merged

3 tasks

NoureldinYosri mentioned this issue Sep 21, 2023

Create a new method to return the final state vector array instead of wrapping it quantumlib/qsim#623

Merged

seberg mentioned this issue Nov 15, 2023

API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL #25149

Merged

seberg closed this as completed Nov 29, 2023

rgommers mentioned this issue Dec 18, 2023

API: Bump NPY_MAXDIMS to 64 and NPY_MAXARGS to 128 #24072

Closed

rht mentioned this issue Feb 12, 2024

Request for releasing a new version of cuQuantum Appliance NVIDIA/cuQuantum#98

Open

Uh oh!

Increase maximum number of array dimensions? #5744

Increase maximum number of array dimensions? #5744

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!