BUG: Cannot convert ctypes struct into numpy array #10528

eric-wieser · 2018-02-06T02:04:56Z

This seems like a bug inherited from ctypes, so maybe we need a workaround:

Start off with a very simple struct:

class foo(ctypes.Structure):
    _fields_ = [('a', ctypes.c_uint8), ('b', ctypes.c_uint32)]
f = foo()

We'd expect this to insert padding, and it does:

>>> memoryview(f).itemsize  # ok
8

But that padding doesn't show up in the format string:

>>> memoryview(f).format
'T{<B:a:<I:b:}'

The docs for struct say that

No padding is added when using non-native size and alignment, e.g. with ‘<’, ‘>’ "

So this format has size 5, which contradicts itemsize.

Sure enough, we catch this in our PEP3118 parser:

>>> np.array(f)
RuntimeWarning: Item size computed from the PEP 3118 buffer format string does not match the actual item size.
Out[45]: array(<__main__.foo object at 0x7f2715469bf8>, dtype=object)

I suspect there's already an issue for this - perhaps we need a PEP3118 tag.

The fix would be to special case ctypes objects in the array constructor, and not call memoryview on them because it doesn't actually work.

Worse yet, numpy doesn't stand a chance with an unpadded struct:

class foo(ctypes.Structure):
    _fields_ = [('a', ctypes.c_uint8), ('b', ctypes.c_uint32)]
    _pack_ = 1
f = foo()

>>> memoryview(f).itemsize  # ok
5
>>> memoryview(f).format  # WAT
'B'

The text was updated successfully, but these errors were encountered:

eric-wieser · 2018-02-06T05:41:23Z

Bug filed at https://bugs.python.org/issue32780, patch at python/cpython#5561. cc @ahaldane, since you've found PEP3118 to be vague in the past.

…ype(ctype_obj)) Fixes numpygh-10528 Fixes numpygh-10978 Fixes numpygh-11150 A warning is thrown when ctypes misbehaves, in order to not hide issues that need fixing upstream

The summary of this diff is that it: * adds a `_ctypes_alloc_format_padding` function to append strings like `37x` to a format string to indicate 37 padding bytes * removes the branches that amount to "give up on producing a valid format string if the struct is packed" * combines the resulting adjacent `if (isStruct) {`s now that neither is `if (isStruct && !isPacked) {` * invokes `_ctypes_alloc_format_padding` to add padding between structure fields, and after the last structure field. The computation used for the total size is unchanged from ctypes already used. This patch does not affect any existing aligment computation; all it does is use subtraction to deduce the amount of paddnig introduced by the existing code. --- Without this fix, it would never include padding bytes - an assumption that was only valid in the case when `_pack_` was set - and this case was explicitly not implemented. This should allow conversion from ctypes structs to numpy structs Fixes numpy/numpy#10528

eric-wieser added component: numpy._core 00 - Bug labels Feb 6, 2018

eric-wieser mentioned this issue Feb 6, 2018

gh-76961: Fix the PEP3118 format string for ctypes.Structure python/cpython#5561

Merged

eric-wieser mentioned this issue Feb 6, 2018

Problems with numpy.array() from ctypes object with _pack_? #7752

Closed

eric-wieser mentioned this issue Jun 8, 2018

BUG: Work around past and present PEP3118 issues in ctypes #11277

Merged

charris closed this as completed in #11277 Jun 8, 2018

eric-wieser mentioned this issue Aug 1, 2018

BUG: Cannot convert ctypes structure with _pack_ into a numpy dtype #11651

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Cannot convert ctypes struct into numpy array #10528

BUG: Cannot convert ctypes struct into numpy array #10528

BUG: Cannot convert ctypes struct into numpy array #10528

BUG: Cannot convert ctypes struct into numpy array #10528

Comments