8000 align is not working when using only strings · Issue #5293 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

align is not working when using only strings #5293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pduch opened this issue Nov 19, 2014 · 10 comments
Closed

align is not working when using only strings #5293

pduch opened this issue Nov 19, 2014 · 10 comments

Comments

@pduch
Copy link
pduch commented Nov 19, 2014

import numpy as np
a = np.zeros(4, dtype=np.dtype('|S4', align=True))
print a.flags['ALIGNED'] # -- should be True

Works with 1.8 but not with 1.9
@njsmith
Copy link
Member
njsmith commented Nov 19, 2014

I'm not even sure what alignment is supposed to mean for strings -
characters are single bytes, so do they just need single byte alignment?

What are you hoping to learn by checking the alignment flag? Do you have
some code that requires a specific alignment?
On 19 Nov 2014 02:44, "pduch" notifications@github.com wrote:

import numpy as np
a = np.zeros(4, dtype=np.dtype('|S4', align=True))
print a.flags['ALIGNED'] # -- should be True

Works with 1.8 but not with 1.9

—
Reply to this email directly or view it on GitHub
#5293.

@pduch
Copy link
Author
pduch commented Nov 19, 2014

I need to call c code with an array of character. It has been working for
years but with 1.9 it is not.
The issue is that it is impossible to get the align flag to true, therefore
impossible to call c code.

2014-11-19 8:30 GMT-05:00 Nathaniel J. Smith notifications@github.com:

I'm not even sure what alignment is supposed to mean for strings -
characters are single bytes, so do they just need single byte alignment?

What are you hoping to learn by checking the alignment flag? Do you have
some code that requires a specific alignment?
On 19 Nov 2014 02:44, "pduch" notifications@github.com wrote:

import numpy as np
a = np.zeros(4, dtype=np.dtype('|S4', align=True))
print a.flags['ALIGNED'] # -- should be True

Works with 1.8 but not with 1.9

—
Reply to this email directly or view it on GitHub
#5293.

—
Reply to this email directly or view it on GitHub
#5293 (comment).

@juliantaylor
Copy link
Contributor

you are using a 32 bit platform and the array is 8 byte aligned?

the issue is that in the new indexing code the string is copied by the itemsize so it needs itemsize alignment. Though currently it uses 16 bytes for all string types which could probably be reduced to the itemsize which would allow S4 to work but not S16 or S9.

I think the new indexing is the only code that tries to copy based on the itemsize so possibly we could revert back to no alignment requirements for sizes not powers of 2 and itemsize alignment for sizes of power 2, that would still leave S16 as problematic as we need to check what is the maximum algnment the compiler needs to copy. GCC provides this information in a macro, other compilers likely not.

@juliantaylor
Copy link
Contributor

or we decouple the aligned flag of an array from the actual required alignment.
the aligned flag then only indicates the alignment of the type which is redundant information for C programs and useless inside of numpy. But it probably makes third party users happy.

@njsmith
Copy link
Member
njsmith commented Nov 19, 2014

Yeah, that's why I'm looking for information on how people are actually
using the aligned flag. :-)

@pduch: In general there shouldn't be any problem with calling C code with
an unaligned array. (After all, numpy is itself written in C, and it
handles unaligned arrays fine!) It should only be a problem if the specific
C code in question is written in such a way that it has specific alignment
requirements. Can you elaborate on what you're doing exactly?
On 19 Nov 2014 14:44, "Julian Taylor" notifications@github.com wrote:

or we decouple the aligned flag of an array from the actual required
alignment.
the aligned flag then only indicates the alignment of the type which is
redundant information for C programs and useless inside of numpy. But it
probably makes third party users happy.

—
Reply to this email directly or view it on GitHub
#5293 (comment).

@pduch
Copy link
Author
pduch commented Nov 19, 2014

Hi Jilian,

Here is the full version... :)
I am on Windows-x86_64 architecture using Python 2.7.2. I have not tested
it on our Linux box yet......

We are calling, from Python, an optimizer with a C api. It has been working
for years but since we upgraded to 1.9, we are getting the following error:
ArgumentError: .....: array must have flags ['C_CONTIGUOUS', 'ALIGNED']

We are using ctypes to call the C api. The parameter causing the problem is
a char array:

my_array = zeros(26, dtype='|S1')

When I check the flags on this, ALIGNED is always False. It was True with
previous versions.

Thanks,
Patrick

2014-11-19 9:44 GMT-05:00 Julian Taylor notifications@github.com:

or we decouple the aligned flag of an array from the actual required
alignment.
the aligned flag then only indicates the al 8000 ignment of the type which is
redundant information for C programs and useless inside of numpy. But it
probably makes third party users happy.

—
Reply to this email directly or view it on GitHub
#5293 (comment).

@juliantaylor
Copy link
Contributor

hm why should it be unaligned on win64, what is the alignment of the array? check with:

my_array.__array_interface__["data"]

@juliantaylor
Copy link
Contributor

nevermind there is a bug in the alignment check it also needs uses the copy alignment for the strides which is of course false for stride 1, that needs to be fixed

@pduch
Copy link
Author
pduch commented Nov 19, 2014

OK thanks a lot :)

Cheers,
P

2014-11-19 12:21 GMT-05:00 Julian Taylor notifications@github.com:

nevermind there is a bug in the alignment check it also needs uses the
copy alignment for the strides which is of course false for stride 1, that
needs to be fixed

—
Reply to this email directly or view it on GitHub
#5293 (comment).

charris pushed a commit to charris/numpy that referenced this issue Jan 22, 2015
@juliantaylor @charris
BUG: fix string arrays not being aligned
e0ef28b 
If itemsize is a power of two use that as the required alignment up to
the maximum provided by the platform. Power of two sizes may be accessed
via larger moves than bytes.
Non-power of two sizes are accessed bytewise and can thus always be
considered aligned.
Closes numpygh-5293
@pduch
Copy link
Author
pduch commented Apr 22, 2015

thanks guys

2015-04-21 22:25 GMT-04:00 Julian Taylor notifications@github.com:

Closed #5293 #5293 via 3351dbf
3351dbf
.

—
Reply to this email directly or view it on GitHub
#5293 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@juliantaylor @njsmith @pduch
0