8000 gh-132983: Introduce `_zstd` bindings module by emmatyping · Pull Request #133027 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-132983: Introduce _zstd bindings module #133027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 59 commits into from
May 4, 2025
Merged

gh-132983: Introduce _zstd bindings module #133027

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
9814e3b
Add _zstd module
emmatyping Apr 26, 2025
fda87c8
Add _zstd to modules
emmatyping Apr 26, 2025
887e564
Fix path for compression.zstd module
emmatyping Apr 26, 2025
cdba656
Ignore _zstd module like _io
emmatyping Apr 26, 2025
6b67e9b
Expand module state macros to improve code quality
emmatyping Apr 26, 2025
a99a5d2
Remove backticks suggested in review
emmatyping Apr 27, 2025
02cd17a
Use critical sections to lock object state
emmatyping Apr 27, 2025
54eca74
Remove compress/decompress and mark module as not reliant on the GIL
emmatyping Apr 27, 2025
f605956
Lift critical section to avoid clang warning
emmatyping Apr 27, 2025
2eadc65
Respond to comments by picnixz
emmatyping Apr 27, 2025
8eac354
Call out pyzstd explicitly in license description
emmatyping Apr 27, 2025
26775be
Use a much more robust implementation...
emmatyping Apr 27, 2025
eae460f
Use PyList_GetItemRef for thread safety purposes
emmatyping Apr 27, 2025
2ab5e4a
Use a macro for the minimum supported version
emmatyping Apr 27, 2025
d5bf1c1
remove const from primivite types
emmatyping Apr 27, 2025
9e92b9f
Use PyMem_New in another spot
emmatyping Apr 27, 2025
47f815a
S 8000 implify error handling in _get_frame_size
emmatyping Apr 27, 2025
6a4f7b8
Another simplification of error handling in get_frame_info
emmatyping Apr 27, 2025
d7b3805
Rename _module_state to mod_state
emmatyping Apr 27, 2025
c225ea6
Rewrite comment explaining the context of the code
emmatyping Apr 28, 2025
6e8c61c
Add link to pyzstd
emmatyping Apr 28, 2025
e52ad06
Add TODO about refactoring dict training code
emmatyping Apr 28, 2025
2a1ad8b
Use PyModule_AddObjectRef over PyModule_AddObject
emmatyping Apr 28, 2025
94473b9
Check result of OutputBufferGrow
emmatyping Apr 28, 2025
e2b2515
Simplify return logic in `add_constant_to_type`
emmatyping Apr 29, 2025
cd2f085
Ignore return value of _zstd_clear()
emmatyping Apr 29, 2025
79e174f
Remove redundant comments
emmatyping Apr 29, 2025
ce6f79c
Remove __reduce__ from ZstdDict
emmatyping Apr 29, 2025
e15dd85
Use PyUnicode_FromFormat instead of a buffer
emmatyping Apr 29, 2025
685a3d1
Don't use C constants/types in error messages
emmatyping Apr 29, 2025
1b9f786
Make error messages easier to understand for Python users
emmatyping Apr 29, 2025
40c653c
Lower minimum required version 1.4.0
emmatyping Apr 30, 2025
428677d
Use casts and make slot function signatures correct
emmatyping Apr 30, 2025
0962bbb
Be consistent with CPython on const usage
emmatyping Apr 30, 2025
85efc18
Make else clauses in line with PEP 7
emmatyping Apr 30, 2025
cadf6e4
Fix over-indented blocks in argument clinic
emmatyping Apr 30, 2025
e45c22a
Merge branch 'main' into 3.14-zstd-c-code
emmatyping Apr 30, 2025
b9415be
Merge branch 'main' into 3.14-zstd-c-code
emmatyping Apr 30, 2025
6760545
Add critical section around ZSTD_DCtx_setParameter
emmatyping May 1, 2025
c082d8a
Add a TODO about refactoring critical sections
emmatyping May 1, 2025
e825285
Use Py_UNREACHABLE
emmatyping May 1, 2025
f02ff5a
Move bytes operations out of Py_BEGIN_ALLOW_THREADS
emmatyping May 2, 2025
0d69c8c
Add TODO about ensuring a lock is held
emmatyping May 2, 2025
58b0008
Remove asserts that may not be correct
emmatyping May 2, 2025
c786c3c
Add TODO to make ZstdDict and others GC objects
emmatyping May 2, 2025
a9d57fc
Make objects GC tracked
emmatyping May 2, 2025
cc9ab2a
Remove unused include
emmatyping May 2, 2025
1f65d19
Fix some memory issues
emmatyping May 2, 2025
552da2d
Fix refleaks on module and in ZstdDict
emmatyping May 2, 2025
c0648d8
Update configure to check for ZDICT_finalizeDictionary
emmatyping May 3, 2025
d149b2c
Merge branch 'main' into 3.14-zstd-c-code
emmatyping May 3, 2025
a5ea379
Properly check version in configure
emmatyping May 3, 2025
0d88d6d
exit(1) if check fails
emmatyping May 3, 2025
4a0a33e
Use AC_RUN_IFELSE
emmatyping May 3, 2025
30b3934
Use a define() to re-use version check
emmatyping May 3, 2025
90b9172
Merge branch 'main' into 3.14-zstd-c-code
emmatyping May 3, 2025
8fdc7f7
Actually properly set _zstd module status based on version
emmatyping May 3, 2025
10e2e80
Merge branch 'main' into 3.14-zstd-c-code
emmatyping May 3, 2025
d90b29d
Merge branch ' 10000 ;main' into 3.14-zstd-c-code
emmatyping May 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Actually properly set _zstd module status based on version
  • Loading branch information
@emmatyping
emmatyping committed May 3, 2025
commit 8fdc7f7328f7411c5791f2957d74c24bcbb54595
18 changes: 11 additions & 7 deletions configure
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 6 additions & 3 deletions configure.ac
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5415,7 +5415,8 @@ PKG_CHECK_MODULES([LIBLZMA], [liblzma], [have_liblzma=yes], [
])
])

define([TEST_ZSTD_VERSION],[
have_libzstd=no
AC_DEFUN([TEST_ZSTD_VERSION],[
AC_MSG_CHECKING([if libzstd is new enough])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There seem to be two problems with the new enough check:

  1. There should be AC_MSG_RESULT([yes]) and AC_MSG_RESULT([no]) calls to complete the messages:
checking for libzstd... yes
checking if libzstd is new enough... checking for hstrerror... yes


dnl TODO(emmatyping): check if we can set a C define for the minimum version
Expand All @@ -5428,7 +5429,9 @@ define([TEST_ZSTD_VERSION],[
#if ZSTD_VERSION_NUMBER < 10405
exit(1);
#endif
])], [have_libzstd=yes], [have_libzstd=no])
])], [py__zstd_too_old=no], [py__zstd_too_old=yes])
dnl Important to record that we have libzstd
have_libzstd=yes
])

PKG_CHECK_MODULES([LIBZSTD], [libzstd], [TEST_ZSTD_VERSION()], [
Copy link
Member
@ned-deily ned-deily May 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and 2. There's something funky going on here with the two calls to TEST_ZSTD_VERSION() such that version number test fails and the failure is sort of hidden in config.log:

configure:22400: checking for libzstd
configure:22407: $PKG_CONFIG --exists --print-errors "libzstd"
configure:22410: $? = 0
configure:22424: $PKG_CONFIG --exists --print-errors "libzstd"
configure:22427: $? = 0
configure:22727: result: yes
configure:22730: checking if libzstd is new enough
configure:22756: gcc -o conftest    conftest.c -ldl  >&5
conftest.c:325:10: fatal error: 'zstd.h' file not found
  325 | #include "zstd.h"
      |          ^~~~~~~~
1 error generated.
configure:22756: $? = 1
configure: program exited with status 1

This later results in:
checking for stdlib extension module _zstd... disabled

I suspect the problem is the call to TEST_ZSTD_VERSION() on line 5437 fails because at that point CPPFLAGS is no longer set to include $LIBZSTD_CFLAGS and the test compile would succeed only if the header files are installed in a default system location, like /usr/include, which won't be the case, for example, on builds on macOS.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Postscript: a likely confirmation of my suspicion above. For our CI runs on macOS, we use Homebrew to supply third-party libs (like libzstd) not supplied by Apple. If you look at CI logs for current PRs (i.e. after this PR was merged), _zstd is disabled (as above) on the ghcr.io/cirruslabs/macos-runner:sonoma runner but succeeds on the macos-13 runner. The difference is that the former is Apple Silicon (arm64) hardware where Homebrew installs to a non-standard /opt/homebrew prefix, whereas the macos-13 runner is Intel hardware where Homebrew continues to install to the legacy /usr/local prefix which, no doubt, is included in the compiler's default search path.

Expand Down Expand Up @@ -8094,7 +8097,7 @@ PY_STDLIB_MOD([_bz2], [], [test "$have_bzip2" = yes],
[$BZIP2_CFLAGS], [$BZIP2_LIBS])
PY_STDLIB_MOD([_lzma], [], [test "$have_liblzma" = yes],
[$LIBLZMA_CFLAGS], [$LIBLZMA_LIBS])
PY_STDLIB_MOD([_zstd], [], [test "$have_libzstd" = yes],
PY_STDLIB_MOD([_zstd], [test "$py__zstd_too_old" = no], [test "$have_libzstd" = yes],
[$LIBZSTD_CFLAGS], [$LIBZSTD_LIBS])

dnl OpenSSL bindings
Expand Down
Loading
0