10000 Error reading blackrock nev file, IndexError: too many indices, in blackrockrawio.py - due to deprecation drop in Numpy 1.23.0 · Issue #1151 · NeuralEnsemble/python-neo · GitHub
[go: up one dir, main page]
Skip to content

Error reading blackrock nev file, IndexError: too many indices, in blackrockrawio.py - due to deprecation drop in Numpy 1.23.0 #1151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
histed opened this issue Aug 12, 2022 · 7 comments · Fixed by #1177
Closed

Error reading blackrock nev file, IndexError: too many indices, in blackrockrawio.py - due to deprecation drop in Numpy 1.23.0 #1151

histed opened this issue Aug 12, 2022 · 7 comments · Fixed by #1177

Comments

@histed
Copy link
Contributor
histed commented Aug 12, 2022 •
edited
Loading

To replicate:
brIO = neo.io.BlackrockIO('datafile002.nev')

Error is at:

                    mask = [ev_ids == i]
>                   curr_data = data[mask]
E                   IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

../../../../miniconda38/envs/mh38a/lib/python3.8/site-packages/neo/rawio/blackrockrawio.py:442: IndexError

Datafile:

datafile002.nev.gz
@histed
Copy link
Contributor Author
histed commented Aug 12, 2022

Versions:

(Pdb) neo.version
'0.10.0'
(Pdb) np.version
'1.23.1'
(Pdb) sys.version
'3.8.10 | packaged by conda-forge | (default, May 10 2021, 22:58:09) \n[Clang 11.1.0 ]'

@histed histed changed the title Error reading blackrock nev file Error reading blackrock nev file, IndexError: too many indices, in blackrockrawio.py Aug 12, 2022
@cboulay
Copy link
Contributor
cboulay commented Aug 17, 2022 •
edited
Loading

I'm encountering this too, using i140703-001-03.nev from https://gin.g-node.org/INT/multielectrode_grasp

The error is in _parse_header.

python-neo/neo/rawio/blackrockrawio.py

Lines 439 to 442 in 50abfbc

for k, (data, ev_ids) in self.nev_data.items():
for i in np.unique(ev_ids):
mask = [ev_ids == i]
curr_data = data[mask]

data is a structured array (fields for timestamp, packet_id, etc.) of shape (1317,)

mask is a len-1 list containing a boolean array.

If I simply remove the square brackets around mask and use the bool array directly then this fixes the problem: curr_data = data[ev_ids == i]

This is 4-5 year old code, so it's amazing I hadn't encountered this before. I wonder what's different about that file vs the files I was using previously.

@histed
Copy link
Contributor Author
histed commented Aug 17, 2022 via email

@cboulay
Copy link
Contributor
cboulay commented Aug 18, 2022

It's in a block of code that only gets touched when there are no .nsx files present. So you can trigger this, for example, by opening a nev file that has been run through a sorter and has an extra -03 on the filename so the matching .nsx files are missed.

The error has been there since at least before Python 3.8, because I tried checking out old commits and nothing before fe56cc3 will work with Python >= 3.8, and that particular commit still has the bug.

I'm not terribly interested in tracking it down more precisely than that.

If everyone is happy with my proposed solution above then I can make a PR.

@histed
Copy link
Contributor Author
histed commented Aug 19, 2022

Looks to me like the issue is a change in Numpy that changed how indexing works:
https://github.com/numpy/numpy/releases

Multidimensional indexing with non-tuple values is not allowed.
Previously, code such as arr[ind] where ind = [[0, 1], [0, 1]]
produced a FutureWarning and was interpreted as a multidimensional
index (i.e., arr[tuple(ind)]). Now this example is treated like an
array index over a single dimension (arr[array(ind)]).
Multidimensional indexing with anything but a tuple was deprecated
in NumPy 1.15.

(numpy/numpy#21029)

So what happened before Numpy 1.23.0 was:
mask = [bool_array] was interpreted as mask = tuple([bool_array]) which is equiv to mask = bool_array for this indexing case.

@histed histed changed the title Error reading blackrock nev file, IndexError: too many indices, in blackrockrawio.py Error reading blackrock nev file, IndexError: too many indices, in blackrockrawio.py - due to deprecation drop in Numpy 1.23.0 Aug 19, 2022
@histed
Copy link
Contributor Author
histed commented Aug 19, 2022

Indeed if I change the code in blackrockrawio.py to

             for i in np.unique(ev_ids):
                    mask = tuple([ev_ids == i])
                    curr_data = data[mask]

it works.

Looks like this change was added in 2018 in this commit: 780d4c4

I looked through blackrockrawio.py and it looks like this is the only line with 'mask' in it that has the bool array wrapped into a list.
So I vote for dropping the square brackets.

@cboulay I'm good with your solution - please go ahead and make the pull request.
Thanks!
Mark

@histed
Copy link
Contributor Author
histed commented Aug 19, 2022

fyi @muellerbjoern - proposed bugfix

This was referenced Oct 1, 2022
Closed
Merged
JuliaSprenger added a commit that referenced this issue Oct 4, 2022
@JuliaSprenger
Merge pull request #1177 from SachsLab/cboulay/1151
714c9c2 
BlackrockRawIO - Fix #1151 indexing bug when no nsx present
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@histed @cboulay
0