8000 gh-97588: Fix ctypes structs by matthiasgoergens · Pull Request #97702 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-97588: Fix ctypes structs #97702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 128 commits into from
May 29, 2024
Merged
Changes from 1 commit
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
7922585
gh-97588: Fix ctypes structs
matthiasgoergens Oct 1, 2022
2307932
📜🤖 Added by blurb_it.
blurb-it[bot] Oct 1, 2022
47f826c
Handling pack as well
matthiasgoergens Oct 1, 2022
5a32211
Handle basedict, too
matthiasgoergens Oct 1, 2022
8d79d8f
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 2, 2022
2d07375
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 3, 2022
8bc8535
Merge remote-tracking branch 'matthias/fix-bitfield-clean' into fix-b…
matthiasgoergens Oct 5, 2022
c3d162b
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Oct 5, 2022
e5ed9ac
Split windows and linux
matthiasgoergens Oct 5, 2022
b0f9819
Compiles
matthiasgoergens Oct 5, 2022
2dee0e3
Abstract out proto stuff
matthiasgoergens Oct 5, 2022
79ef347
Clean align
matthiasgoergens Oct 5, 2022
6170dad
Hypothesis tests pass
matthiasgoergens Oct 5, 2022
871ca1a
Formatting
matthiasgoergens Oct 5, 2022
c162144
Clean up
matthiasgoergens Oct 5, 2022
a57cf2c
Adapt tests for Windows
matthiasgoergens Oct 5, 2022
3f7c4cc
Fix order
matthiasgoergens Oct 5, 2022
e39a271
Fix alignment test
matthiasgoergens Oct 5, 2022
359ed58
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Oct 5, 2022
52ef8d2
Avoid casting
matthiasgoergens Oct 5, 2022
e8102c4
Fixup
matthiasgoergens Oct 5, 2022
600e144
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 5, 2022
9bad706
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 5, 2022
5121857
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Oct 6, 2022
6cd27ea
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Oct 8, 2022
6b6fa8a
Add ability to force msvc compatibility even when not doing any packing
matthiasgoergens Oct 8, 2022
ca9d580
More tests
matthiasgoergens Oct 8, 2022
1401ee4
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 9, 2022
235fa68
Big endian works
matthiasgoergens Oct 9, 2022
a534e18
Merge remote-tracking branch 'matthias/fix-bitfield-clean' into fix-b…
matthiasgoergens Oct 9, 2022
53db061
More tests
matthiasgoergens Oct 9, 2022
9c249c3
More asserts
matthiasgoergens Oct 9, 2022
3cf4747
Drop unneeded parameters
matthiasgoergens Oct 9, 2022
8beddb9
Drop unneeded parameters
matthiasgoergens Oct 9, 2022
9b64ff6
Drop unneeded parameters
matthiasgoergens Oct 9, 2022
4d0dab5
Clean up
matthiasgoergens Oct 9, 2022
bf98667
Explain
matthiasgoergens Oct 9, 2022
8223063
More cleanup
matthiasgoergens Oct 9, 2022
e80a09a
More cleanup
matthiasgoergens Oct 9, 2022
33a10fb
More cleanup
matthiasgoergens Oct 9, 2022
0f3c5a3
Move common assert
matthiasgoergens Oct 9, 2022
47eca3c
Break line
matthiasgoergens Oct 9, 2022
456afd0
Clean TODO
matthiasgoergens Oct 9, 2022
bc799ac
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 9, 2022
212cf13
Fix test for Windows
matthiasgoergens Oct 9, 2022
e7e5ae9
Merge remote-tracking branch 'matthias/fix-bitfield-clean' into fix-b…
matthiasgoergens Oct 9, 2022
45e26ec
Remove warning
matthiasgoergens Oct 9, 2022
ab42187
Port test from PR-19850
matthiasgoergens Oct 9, 2022
640c062
Fix assert
matthiasgoergens Oct 9, 2022
f3e04af
Revert "Port test from PR-19850"
matthiasgoergens Oct 9, 2022
bff34a1
Support gcc's packed structs
matthiasgoergens Oct 9, 2022
5d6f0f7
Remove gcc_packed experiment
matthiasgoergens Oct 9, 2022
d9c1fca
Fix test
matthiasgoergens Oct 10, 2022
cc9ea8a
Fix ms_struct
matthiasgoergens Oct 10, 2022
1554083
Fix ms_struct
matthiasgoergens Oct 10, 2022
a86eadc
Indenting
matthiasgoergens Oct 10, 2022
eefa1ad
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Oct 10, 2022
28c7fe7
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 10, 2022
f0a92b0
Fix miscounted parens
matthiasgoergens Oct 10, 2022
a3a390b
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 10, 2022
e997b5f
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Mar 14, 2023
5a7ea09
make regen-all
matthiasgoergens Mar 14, 2023
0f73919
clean up
matthiasgoergens Mar 14, 2023
cdfc3c6
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Mar 15, 2023
6226e55
make regen-all
matthiasgoergens Mar 15, 2023
4d48eba
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Mar 16, 2023
20b5582
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Mar 17, 2023
eb502d7
Explain where 999 comes from
matthiasgoergens Mar 17, 2023
0cf5f4e
fix typo
matthiasgoergens Mar 17, 2023
3457558
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Mar 17, 2023
fd3cd0b
Explain magic 999
matthiasgoergens Mar 17, 2023
2d8492e
Incorporate Sam's tests
matthiasgoergens Mar 27, 2023
a9f7f14
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens May 24, 2023
bee6c53
Tickle CI/CD
matthiasgoergens May 24, 2023
220e19e
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens May 24, 2023
60cebe2
Merge branch 'main' into fix-bitfield-clean
gpshead May 24, 2023
3424a7d
Merge commit '698a0da7d440856a90b45964e9082b5a55387b80' into fix-bitf…
matthiasgoergens Apr 14, 2024
2506eea
Merge commit '0841ca7932987f30a2a23d39f3e6e141622d6fea' into fix-bitf…
matthiasgoergens Apr 14, 2024
65654b4
Merge commit '9f7176d360b5898003d5ca78bab1822ad67867c4' into fix-bitf…
matthiasgoergens Apr 14, 2024
98767da
Fix compile
matthiasgoergens Apr 14, 2024
3ca703f
Merge commit '298bcdc185d1a9709271e61a4cc529d33483add4' into fix-bitf…
matthiasgoergens Apr 14, 2024
c40ef7d
Merge commit 'dcaf33a41d5d220523d71c9b35bc08f5b8405dac' into fix-bitf…
matthiasgoergens Apr 14, 2024
d840f01
Merge commit '7e2fef865899837c47e91ef0180fa59eb03e840b' into fix-bitf…
matthiasgoergens Apr 14, 2024
8b9f0eb
Merge commit 'ea94b3b149eeadf33c2f7c46f16dcda0adc7cf4e' into fix-bitf…
matthiasgoergens Apr 14, 2024
4e8c220
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Apr 14, 2024
0369d0d
Fix merge
matthiasgoergens Apr 14, 2024
d4dc2c0
Clean up
matthiasgoergens Apr 14, 2024
f75d7d6
Fix tests
matthiasgoergens Apr 14, 2024
cdc5cdc
Merge in the main branch
encukou Apr 22, 2024
0da36ad
Add Hypothesis test
encukou Apr 22, 2024
b6f7117
Merge in the main branch
encukou Apr 29, 2024
5e47d5f
dict -> info
encukou Apr 30, 2024
de22b39
Conditionalize ms_struct
encukou Apr 30, 2024
cdd1860
Use fixed-width types in tests
encukou Apr 30, 2024
dd84ac3
Use subTest
encukou Apr 30, 2024
8a6fb67
fixup! dict
encukou Apr 30, 2024
07ea42d
Turn `info->size != info->align` assertion into a TypeError
encukou Apr 30, 2024
3fa7f55
PEP 7
encukou Apr 30, 2024
eb2c4fb
Change _ms_struct_ to _layout_
encukou Apr 30, 2024
22cd86c
Merge in the main branch
encukou Apr 30, 2024
637b961
PEP 7
encukou Apr 30, 2024
5309b7e
TMP
encukou Apr 30, 2024
489c5c8
Add generated tests
encukou May 2, 2024
3594473
Merge in the main branch
encukou May 2, 2024
f780496
Remove ideas
encukou May 2, 2024
6ade022
Regen
encukou May 2, 2024
cfa6647
Do not set _layout_
encukou May 2, 2024
df162b0
We don't always get union alignment right
encukou May 2, 2024
836a5cc
Remove unused stuff
encukou May 2, 2024
b07ae33
Packing implies ms layout
encukou May 2, 2024
6dd7a8d
Docs & whitespace
encukou May 2, 2024
0cf1049
Use limited API again
encukou May 2, 2024
bc91549
Appease the linter
encukou May 3, 2024
70bbc26
Fix ReST typo
encukou May 3, 2024
6e23b3d
Use nicer error output for memory dumps
encukou May 3, 2024
1b90841
Regen global strings
encukou May 3, 2024
2c4874b
Allow alignment < size, as in int64_t on x86 (32-bit), GCC layout
encukou May 3, 2024
738323f
Add examples from MS docs
encukou May 4, 2024
af7487c
Use correct spelling for test skips
encukou May 5, 2024
5353352
Only use 'attribute((ms_struct))' on x86_64 & ppc; on GCC and clang
encukou May 5, 2024
fe76d45
Remove the `info->size == info->align` assertion for _layout_='ms'
encukou May 5, 2024
c4714f1
Fix silly mistake in the macro
encukou May 5, 2024
c79725d
Regen
encukou May 5, 2024
cace0c9
Include field name in error message
encukou May 5, 2024
61e92bf
Conditionally skip tests that assume sizeof(int64) == alignment(int64)
encukou May 5, 2024
ba61051
Use ms_struct only on Windows and x86/ppc GCC/clang
encukou May 5, 2024
8991444
Use proper PyArg_ParseTuple code for Py_ssize_t
encukou May 6, 2024
bc1225b
Merge in the main branch
encukou May 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Turn info->size != info->align assertion into a TypeError
  • Loading branch information
encukou committed Apr 30, 2024
commit 07ea42def08b306b5ffef41f7ef45760d1cd9818
31 changes: 25 additions & 6 deletions Modules/_ctypes/cfield.c
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ PyCField_FromDesc manages:
- *palign: the alignment requirements of the last field we placed.
*/

static void
static int
PyCField_FromDesc_gcc(Py_ssize_t bitsize, Py_ssize_t *pbitofs,
Py_ssize_t *psize, Py_ssize_t *poffset, Py_ssize_t *palign,
CFieldObject* self, StgInfo* info,
Expand All @@ -111,8 +111,13 @@ PyCField_FromDesc_gcc(Py_ssize_t bitsize, Py_ssize_t *pbitofs,
self->offset = round_down(*pbitofs, 8*info->size) / 8;
Py_ssize_t effective_bitsof = *pbitofs - 8 * self->offset;
self->size = BUILD_SIZE(bitsize, effective_bitsof);
assert(info->size == info->align);
assert(effective_bitsof <= info->size * 8);
if (info->size != info->align) {
PyErr_SetString(
PyExc_TypeError,
"bitfield's base type size differs from alignment");
return -1;
}
} else {
assert(bitsize == 0);
self->offset = round_down(*pbitofs, 8*info->align) / 8;
Expand All @@ -121,9 +126,11 @@ PyCField_FromDesc_gcc(Py_ssize_t bitsize, Py_ssize_t *pbitofs,

*pbitofs += bitsize;
*psize = round_up(*pbitofs, 8) / 8;

return 0;
}

static void
static int
PyCField_FromDesc_msvc(
Py_ssize_t *pfield_size, Py_ssize_t bitsize,
Py_ssize_t *pbitofs, Py_ssize_t *psize, Py_ssize_t *poffset,
Expand Down Expand Up @@ -159,7 +166,12 @@ PyCField_FromDesc_msvc(

self->offset = *poffset - (*pfield_size) / 8;
if(is_bitfield) {
assert(info->size == info->align);
if (info->size != info->align) {
PyErr_SetString(
PyExc_TypeError,
"bitfield's base type size differs from alignment");
return -1;
}
assert(0 <= (*pfield_size + *pbitofs));
assert((*pfield_size + *pbitofs) < info->size * 8);
self->size = BUILD_SIZE(bitsize, *pfield_size + *pbitofs);
Expand All @@ -170,6 +182,8 @@ PyCField_FromDesc_msvc(

*pbitofs += bitsize;
*psize = *poffset;

return 0;
}

PyObject *
Expand Down Expand Up @@ -251,22 +265,27 @@ PyCField_FromDesc(ctypes_state *st, PyObject *desc, Py_ssize_t index,
assert(bitsize <= info->size * 8);

// `pack` only makes sense in msvc compatibility mode.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.python.org/3/library/ctypes.html#structure-union-alignment-and-byte-order says

By default, Structure and Union fields are aligned in the same way the C compiler does it. It is possible to override this behavior by specifying a pack class attribute in the subclass definition. This must be set to a positive integer and specifies the maximum alignment for the fields. This is what #pragma pack(n) also does in MSVC.

GCC has a suspiciously similarly named __attribute__((packed)) / #pragma pack. But it does something different. To get GCC to give you the behaviour as described in the doc, you also need to enable __attribute__((ms_struct)) / #pragma ms_struct on. And that's what we are doing here.

Copy link
Contributor Author
@matthiasgoergens matthiasgoergens Oct 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: the behaviour described in the quoted docs that we are implementing here is rather different from what everyone in #19850 seems to have understood.

As far as I can tell in that PR everyone assume that _packed_ = 1 means GCC's __attribute__((packed)).

This misunderstanding might be an argument for changing the docs? However that would be a breaking change.

In general, this PR breaks bug-for-bug compatibility with the past, but does not change documented behaviour.

At the moment, we don't support GCC's version of packed structs. Neither in the existing code nor in this PR.

It's fairly easy to add this behaviour on top of my code; but I would suggest using a new attribute (like _gcc_packed_ or so), instead of changing the documented behaviour of the existing _pack_.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction: support for GCC's packing isn't actually as easy as I imagined.

The problem is that GCC's packing can make eg a uint8_t straddle two bytes. In order to support that, we'd need to revamp much more of ctypes.

Copy link
Contributor Author
@matthiasgoergens matthiasgoergens Oct 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's an example to show the problem:

#include <stdio.h>
#include <stdint.h>
#include <string.h>

typedef struct
__attribute__ ((packed))
{
  uint8_t a: 4;
  uint8_t b: 8;
  uint8_t c: 4;
} Foo;

typedef union {
  Foo foo;
  uint8_t x[sizeof(Foo)];
} U;

int main(int argc, char** argv) {
    printf("%lu\n", __alignof__(Foo));
    printf("%lu\n", sizeof(Foo));

    U u;
    memset(&u, 0, sizeof(U));
    // Set b to all 1s.
    u.foo.b = -1;
    // Because we are on a little endian machine, it makes more sense to print
    // the bytes in reverse order.
    // This prints "0f f0", showing that b straddles a byte.
    for(int i=sizeof(Foo)-1; i>=0; --i) {
        printf("%02x ", u.x[i]);
    }
    printf("\n");
    return 0;
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the right thing to do here would be to warn when _packed_ is used with the GCC semantics.

int result;
if (ms_struct || pack != 0) {
PyCField_FromDesc_msvc(
result = PyCField_FromDesc_msvc(
pfield_size, bitsize, pbitofs,
psize, poffset, palign,
pack,
self, info,
is_bitfield
);
} else {
PyCField_FromDesc_gcc(
result = PyCField_FromDesc_gcc(
bitsize, pbitofs,
psize, poffset, palign,
self, info,
is_bitfield
);
}
if (result < 0) {
Py_DECREF(self);
return NULL;
}
assert(!is_bitfield || (LOW_BIT(self->size) <= self->size * 8));
if(big_endian && is_bitfield) {
self->size = BUILD_SIZE(NUM_BITS(self->size), 8*info->size - LOW_BIT(self->size) - bitsize);
Expand Down
0