8000 regression (from 1.5.x) with return of operations on memmap'ed arrays (Trac #2215) · Issue #667 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

regression (from 1.5.x) with return of operations on memmap'ed arrays (Trac #2215) #667

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
numpy-gitbot opened this issue Oct 19, 2012 · 11 comments · Fixed by #7406
Closed

regression (from 1.5.x) with return of operations on memmap'ed arrays (Trac #2215) #667

numpy-gitbot opened this issue Oct 19, 2012 · 11 comments · Fixed by #7406
Labels
00 - Bug component: numpy._core

Comments

@numpy-gitbot
Copy link

Original ticket http://projects.scipy.org/numpy/ticket/2215 on 2012-09-14 by @yarikoptic, assigned to unknown.

As
http://projects.scipy.org/numpy/ticket/2179
stated: "Ideally one should be returning other ndarray objects, e.g. numpy scalars in the case of sum(), but this requires a larger rewrite. At least numpy version 1.5.1rc1 did this. The only time the _mmap should be kept (as I am aware of) is when slicing. "

This issue remains not fixed thus opening a separate issue for the above. Taking into account that np.asscalar() called on scalars
(e.g. which would be returned by np.sum on ndarrays) crashes, it becomes impossible to write code agnostic of ndarray/memmap container types.

it is also not clear why array-wide operations on mode='c' memmap'ed arrays which cannot really get reflected in the file storage, remain memmapped and do not result in normal ndarrays, but I guess that would be a separate issue...
@charris
Copy link
Member
charris commented Feb 18, 2014

@yarikoptic Could you add some specific examples of what you want to see?

@yarikoptic
Copy link
Contributor

oh boy -- I do not remember details NOW. is there a backup of those issues in their entirety somewhere to refresh what caused me to file the issue? (urls now get redirected here to github)

@charris
Copy link
Member
charris commented Feb 18, 2014

If there were any attachments, they would be here along with any comments. But I agree that this is ancient, about 1000 years in computer seconds ;) If you can't jog your memory, I'll close this because I don't know what it is talking about. Not that I've got much experience with memmapped files in the first place.

@mdickinson
Copy link
Contributor

I can't access the original ticket, but in NumPy 1.8.2, x.sum() gives a scalar if x is a regular ndarray, and a zero-dimensional memmap if x is a memmap instance. As a result, np.isscalar(x.sum()) passes in the first case and fails in the second (and as I recently discovered, scipy.ndimage.measurements.center_of_mass breaks on memmap instances because of this).

This discrepancy seems to fit with numpy-gitbot's original message for this issue.

@mdickinson
Copy link
Contributor

as I recently discovered

Hmm; that's overstating my involvement: this came from investigating someone else's Stack Overflow question. See here: http://stackoverflow.com/q/25369982/270986

@charris
Copy link
Member
charris commented Aug 18, 2014

The sum method is implemented with add.reduce, see numpy/core/_methods.py. If that is the problem, I would suspect that x.min and friends should also cause problems.

@mdickinson
Copy link
Contributor

Your suspicions seem to be accurate 😀.

Python 3.4.1 (default, May 21 2014, 01:39:38) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__version__
'1.8.2'
>>> x = np.memmap("my_memmap", dtype=float, mode='w+', shape=(3, 4))
>>> x[...] = np.arange(12).reshape((3, 4))
>>> x.sum()
memmap(66.0)
>>> np.array(x).sum()
66.0
>>> x.min()
memmap(0.0)
>>> np.array(x).min()
0.0

@charris
Copy link
Member
charris commented Aug 18, 2014

Hmm, by and large, the operations try to preserve subclasses, and memmap inherits from ndarray. The best fix for this might be in numpy.memmap, defined in numpy/core/memmap.py, but it isn't clear to me how. Just making memmap scalars numpy scalars might be a bit tricky. They should be an instances of ndarray in any case. Might be that the isscalar function could be fixed to recognize scalar instances of subclasses of ndarray.

@charris
Copy link
Member
charris commented Aug 19, 2014

For instance,

In [5]: isscalar(1)
Out[5]: True

In [6]: class Foo(int): pass

In [7]: isscalar(Foo(1))
Out[7]: False

If people are going to subclass ndarray and scalar types, we might want to modify isscalar.

@seberg
Copy link
Member
8000 seberg commented Aug 19, 2014

That would be a bit larger modification though. 0-d arrays (array scalars) are currently not considered scalars by isscalar. To me the subclass thing seems like broadening it is good though.

@Nodd
Copy link
Contributor
Nodd commented Aug 25, 2014

is it possible to use __array_wrap__ or something similar in memmap.py to return a ndarray if the result is not a view on a memmap ?

@ahaldane ahaldane mentioned this issue Mar 10, 2016
Closed
@lesteve lesteve mentioned this issue Mar 10, 2016
Merged
@zougloub zougloub mentioned this issue Apr 29, 2018
Merged
zougloub pushed a commit to spinalcordtoolbox/spinalcordtoolbox that referenced this issue Apr 30, 2018
@charleygros @zougloub
sct_deepseg_sc - Issue when input is .nii instead of .nii.gz (#1634, #…
75a7408 
…1659)

Cast np.memmap into np.array when calling `center_of_mass()`.

Upstream issue numpy/numpy#667 is fixed but we bundle an old numpy version.
jcohenadad pushed a commit to spinalcordtoolbox/spinalcordtoolbox that referenced this issue Dec 18, 2019
@charleygros @zougloub
sct_deepseg_sc - Issue when input is .nii instead of .nii.gz (#1634, #…
ead73aa 
…1659)

Cast np.memmap into np.array when calling `center_of_mass()`.

Upstream issue numpy/numpy#667 is fixed but we bundle an old numpy version.

Former-commit-id: 75a7408
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy._core
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ENH ufunc called on memmap return a ndarray lesteve/numpy
6 participants
@yarikoptic @seberg @charris @Nodd @mdickinson @numpy-gitbot
0