10000 genfromtxt won't unpack if dtype=None · Issue #4341 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

genfromtxt won't unpack if dtype=None #4341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Gabriel-p opened this issue Feb 21, 2014 · 6 comments · Fixed by #16650
Closed

genfromtxt won't unpack if dtype=None #4341

Gabriel-p opened this issue Feb 21, 2014 · 6 comments · Fixed by #16650
Labels
00 - Bug component: numpy.lib
Milestone
1.20.0 release

Comments

@Gabriel-p
Copy link
Contributor

Not sure if this is a bug or a feature but if I use:

f_data = np.genfromtxt('file.dat', dtype=None, unpack=True)

where file.dat is a standard data file with several columns (ie: http://pastebin.com/ihEW2dxS) the result is not transposed as one would expect having set unpack=True.

For example, the line f_data = np.loadtxt('file.dat', unpack=True) works as expected.
@charris
Copy link
Member
charris commented Feb 24, 2014

Looks like a bug, marking it so until informed otherwise.

@ddasilva
Copy link
Contributor
ddasilva commented Mar 3, 2014

I investigated this a little bit.

The result of @Gabriel-p's code is a shape (12,) flex array. The code calls output.T on it, but since it's one dimensional this is a no-op.

The biggest issue that dtype=None specifies that it will be flex array. Each column has a separate inferred type. If we were to truly transpose this result, each row would need to have a separate type. AFAIK, the only way to do that is to have a list or tuple of homogeneous arrays.

Since numCols << numRows in the text file, using a list or tuple shouldn't be a big deal. If implemented that way, it would unpack for x, y, z = np.genfromtxt(...) as specified in the docs, and solve @Gabriel-p's case.

However, genfromtxt is already over 600 lines. Any thoughts?

@hsgg
Copy link
hsgg commented May 16, 2014

Just got bitten by this, too. I made a test case, see

https://github.com/hsgg/numpy.git

@moi90
Copy link
moi90 commented Dec 22, 2017 •
edited
Loading

I also experience this. loadtxt is a bit more clever:

if unpack:
        if len(dtype_types) > 1:
            # For structured arrays, return an array for each field.
            return [X[field] for field in dtype.names]
        else:
            return X.T

Why doesn't genfromtxt do this?

@adehecq
Copy link
adehecq commented Jun 10, 2020

Any update on this issue? It seems like it hasn't changed since 2014.

@WarrenWeckesser
Copy link
Member
WarrenWeckesser commented Jun 10, 2020 •
edited
Loading

The suggestion @moi90 made a couple years ago is correct. genfromtxt should do the same thing as loadtxt.

I think it will be easy to fix, so I've tagged the issue as a "good first issue".

@WarrenWeckesser WarrenWeckesser added this to the 1.20.0 release milestone Jun 10, 2020
@AndrewEckart AndrewEckart mentioned this issue Jun 20, 2020
Merged
AndrewEckart added a commit to AndrewEckart/numpy that referenced this issue Jul 12, 2020
@AndrewEckart
ENH: Allow genfromtxt to unpack structured arrays
189ad50 
Previously, `genfromtxt` failed to transpose its output when
`unpack=True` and `dtype` was structured (or `dtype=None` and the
inferred `dtype` was structured). This patch resolves the issue by
returning a list of arrays, as in `loadtxt`. Includes tests and updates
the docstring to match `loadtxt`.

Fixes numpy#4341
@mzechmeister mzechmeister mentioned this issue Mar 14, 2021
Closed
@emirkmo emirkmo mentioned this issue Jan 19, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy.lib
Projects
None yet
Milestone
1.20.0 release
Development

Successfully merging a pull request may close this issue.

ENH: Allow genfromtxt to unpack structured arrays AndrewEckart/numpy
8 participants
@charris @hsgg @WarrenWeckesser @moi90 @ddasilva @mattip @adehecq @Gabriel-p
0