8000 gh-132983: [Draft] Run refleak tests against _zstd with tests by emmatyping · Pull Request #133282 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content
8000

gh-132983: [Draft] Run refleak tests against _zstd with tests #133282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 51 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
9814e3b
Add _zstd module
emmatyping Apr 26, 2025
fda87c8
Add _zstd to modules
emmatyping Apr 26, 2025
887e564
Fix path for compression.zstd module
emmatyping Apr 26, 2025
cdba656
Ignore _zstd module like _io
emmatyping Apr 26, 2025
6b67e9b
Expand module state macros to improve code quality
emmatyping Apr 26, 2025
a99a5d2
Remove backticks suggested in review
emmatyping Apr 27, 2025
02cd17a
Use critical sections to lock object state
emmatyping Apr 27, 2025
54eca74
Remove compress/decompress and mark module as not reliant on the GIL
emmatyping Apr 27, 2025
f605956
Lift critical section to avoid clang warning
emmatyping Apr 27, 2025
2eadc65
Respond to comments by picnixz
emmatyping Apr 27, 2025
8eac354
Call out pyzstd explicitly in license description
emmatyping Apr 27, 2025
26775be
Use a much more robust implementation...
emmatyping Apr 27, 2025
eae460f
Use PyList_GetItemRef for thread safety purposes
emmatyping Apr 27, 2025
2ab5e4a
Use a macro for the minimum supported version
emmatyping Apr 27, 2025
d5bf1c1
remove const from primivite types
emmatyping Apr 27, 2025
9e92b9f
Use PyMem_New in another spot
emmatyping Apr 27, 2025
47f815a
Simplify error handling in _get_frame_size
emmatyping Apr 27, 2025
6a4f7b8
Another simplification of error handling in get_frame_info
emmatyping Apr 27, 2025
d7b3805
Rename _module_state to mod_state
emmatyping Apr 27, 2025
c225ea6
Rewrite comment explaining the context of the code
emmatyping Apr 28, 2025
6e8c61c
Add link to pyzstd
emmatyping Apr 28, 2025
e52ad06
Add TODO about refactoring dict training code
emmatyping Apr 28, 2025
2a1ad8b
Use PyModule_AddObjectRef over PyModule_AddObject
emmatyping Apr 28, 2025
94473b9
Check result of OutputBufferGrow
emmatyping Apr 28, 2025
e2b2515
Simplify return logic in `add_constant_to_type`
emmatyping Apr 29, 2025
cd2f085
Ignore return value of _zstd_clear()
emmatyping Apr 29, 2025
79e174f
Remove redundant comments
emmatyping Apr 29, 2025
ce6f79c
Remove __reduce__ from ZstdDict
emmatyping Apr 29, 2025
e15dd85
Use PyUnicode_FromFormat instead of a buffer
emmatyping Apr 29, 2025
685a3d1
Don't use C constants/types in error messages
emmatyping Apr 29, 2025
1b9f786
Make error messages easier to understand for Python users
emmatyping Apr 29, 2025
40c653c
Lower minimum required version 1.4.0
emmatyping Apr 30, 2025
428677d
Use casts and make slot function signatures correct
emmatyping Apr 30, 2025
0962bbb
Be consistent with CPython on const usage
emmatyping Apr 30, 2025
85efc18
Make else clauses in line with PEP 7
emmatyping Apr 30, 2025
cadf6e4
Fix over-indented blocks in argument clinic
emmatyping Apr 30, 2025
e45c22a
Merge branch 'main' into 3.14-zstd-c-code
emmatyping Apr 30, 2025
b9415be
Merge branch 'main' into 3.14-zstd-c-code
emmatyping Apr 30, 2025
6760545
Add critical section around ZSTD_DCtx_setParameter
emmatyping May 1, 2025
c082d8a
Add a TODO about refactoring critical sections
emmatyping May 1, 2025
e825285
Use Py_UNREACHABLE
emmatyping May 1, 2025
f02ff5a
Move bytes operations out of Py_BEGIN_ALLOW_THREADS
emmatyping May 2, 2025
0d69c8c
Add TODO about ensuring a lock is held
emmatyping May 2, 2025
58b0008
Remove asserts that may not be correct
emmatyping May 2, 2025
c786c3c
Add TODO to make ZstdDict and others GC objects
emmatyping May 2, 2025
d15ced7
Add Python files
emmatyping Apr 30, 2025
dfa7649
Fix byteswarning in test
emmatyping Apr 30, 2025
dedf955
Make objects GC tracked
emmatyping May 2, 2025
9ac9101
Remove unused include
emmatyping May 2, 2025
65542c4
Fix some memory issues
emmatyping May 2, 2025
6328860
Fix refleaks on module and in ZstdDict
emmatyping May 2, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Be consistent with CPython on const usage
  • Loading branch information
emmatyping committed Apr 30, 2025
commit 0962bbb0dc04bcd23fef42fe26db1672a976446d
6 changes: 3 additions & 3 deletions Modules/_zstd/_zstdmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ module _zstd
/* Format error message and set ZstdError. */
void
set_zstd_error(const _zstd_state* const state,
const error_type type, const size_t zstd_ret)
error_type type, size_t zstd_ret)
{
char *msg;
assert(ZSTD_isError(zstd_ret));
Expand Down Expand Up @@ -610,7 +610,7 @@ add_parameters(PyObject *module)
static inline PyObject *
get_zstd_version_info(void)
{
const uint32_t ver = ZSTD_versionNumber();
uint32_t ver = ZSTD_versionNumber();
uint32_t major, minor, release;

major = ver / 10000;
Expand Down Expand Up @@ -723,7 +723,7 @@ add_type_to_module(PyObject *module, const char *name,
}

static inline int
add_constant_to_type(PyTypeObject *type, const char *name, const long value)
add_constant_to_type(PyTypeObject *type, const char *name, long value)
{
PyObject *temp;

Expand Down
10 changes: 5 additions & 5 deletions Modules/_zstd/_zstdmodule.h
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ mt_continue_should_break(ZSTD_inBuffer *in, ZSTD_outBuffer *out) {
/* Format error message and set ZstdError. */
extern void
set_zstd_error(const _zstd_state* const state,
const error_type type, const size_t zstd_ret);
const error_type type, size_t zstd_ret);

extern void
set_parameter_error(const _zstd_state* const state, int is_compress,
Expand All @@ -194,10 +194,10 @@ _PyZstd_set_d_parameters(ZstdDecompressor *self, PyObject *options);

extern PyObject *
decompress_impl(ZstdDecompressor *self, ZSTD_inBuffer *in,
const Py_ssize_t max_length,
const Py_ssize_t initial_size,
const decompress_type type);
Py_ssize_t max_length,
Py_ssize_t initial_size,
decompress_type type);

extern PyObject *
compress_impl(ZstdCompressor *self, Py_buffer *data,
const ZSTD_EndDirective end_directive);
ZSTD_EndDirective end_directive);
6 changes: 3 additions & 3 deletions Modules/_zstd/buffer.h
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Python module.
Return -1 on failure */
static inline int
_OutputBuffer_InitAndGrow(_BlocksOutputBuffer *buffer, ZSTD_outBuffer *ob,
const Py_ssize_t max_length)
Py_ssize_t max_length)
{
/* Ensure .list was set to NULL */
assert(buffer->list == NULL);
Expand All @@ -33,8 +33,8 @@ _OutputBuffer_InitAndGrow(_BlocksOutputBuffer *buffer, ZSTD_outBuffer *ob,
Return -1 on failure */
static inline int
_OutputBuffer_InitWithSize(_BlocksOutputBuffer *buffer, ZSTD_outBuffer *ob,
const Py_ssize_t max_length,
const Py_ssize_t init_size)
Py_ssize_t max_length,
Py_ssize_t init_size)
{
Py_ssize_t block_size;

Expand Down
8 changes: 4 additions & 4 deletions Modules/_zstd/compressor.c
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ _PyZstd_set_c_parameters(ZstdCompressor *self, PyObject *level_or_options,

/* Integer compression level */
if (PyLong_Check(level_or_options)) {
const int level = PyLong_AsInt(level_or_options);
int level = PyLong_AsInt(level_or_options);
if (level == -1 && PyErr_Occurred()) {
PyErr_Format(PyExc_ValueError,
"Compression level should be an int value between %d and %d.",
Expand Down Expand Up @@ -75,7 +75,7 @@ _PyZstd_set_c_parameters(ZstdCompressor *self, PyObject *level_or_options,
return -1;
}

const int key_v = PyLong_AsInt(key);
int key_v = PyLong_AsInt(key);
if (key_v == -1 && PyErr_Occurred()) {
PyErr_SetString(PyExc_ValueError,
"Key of options dict should be a CParameter attribute.");
Expand All @@ -84,7 +84,7 @@ _PyZstd_set_c_parameters(ZstdCompressor *self, PyObject *level_or_options,

// TODO(emmatyping): check bounds when there is a value error here for better
// error message?
const int value_v = PyLong_AsInt(value);
int value_v = PyLong_AsInt(value);
if (value_v == -1 && PyErr_Occurred()) {
PyErr_SetString(PyExc_ValueError,
"Value of option dict should be an int.");
Expand Down Expand Up @@ -399,7 +399,7 @@ _zstd_ZstdCompressor___init___impl(ZstdCompressor *self, PyObject *level,

PyObject *
compress_impl(ZstdCompressor *self, Py_buffer *data,
const ZSTD_EndDirective end_directive)
ZSTD_EndDirective end_directive)
{
ZSTD_inBuffer in;
ZSTD_outBuffer out;
Expand Down
24 changes: 12 additions & 12 deletions Modules/_zstd/decompressor.c
7E38
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ _PyZstd_set_d_parameters(ZstdDecompressor *self, PyObject *options)
}

/* Both key & value should be 32-bit signed int */
const int key_v = PyLong_AsInt(key);
int key_v = PyLong_AsInt(key);
if (key_v == -1 && PyErr_Occurred()) {
PyErr_SetString(PyExc_ValueError,
"Key of options dict should be a DParameter attribute.");
Expand All @@ -96,7 +96,7 @@ _PyZstd_set_d_parameters(ZstdDecompressor *self, PyObject *options)

// TODO(emmatyping): check bounds when there is a value error here for better
// error message?
const int value_v = PyLong_AsInt(value);
int value_v = PyLong_AsInt(value);
if (value_v == -1 && PyErr_Occurred()) {
PyErr_SetString(PyExc_ValueError,
"Value of options dict should be an int.");
Expand Down Expand Up @@ -269,9 +269,9 @@ _PyZstd_load_d_dict(ZstdDecompressor *self, PyObject *dict)
*/
PyObject *
decompress_impl(ZstdDecompressor *self, ZSTD_inBuffer *in,
const Py_ssize_t max_length,
const Py_ssize_t initial_size,
const decompress_type type)
Py_ssize_t max_length,
Py_ssize_t initial_size,
decompress_type type)
{
size_t zstd_ret;
ZSTD_outBuffer out;
Expand Down Expand Up @@ -370,7 +370,7 @@ decompress_impl(ZstdDecompressor *self, ZSTD_inBuffer *in,

void
decompressor_reset_session(ZstdDecompressor *self,
const decompress_type type)
decompress_type type)
{
/* Reset variables */
self->in_begin = 0;
Expand All @@ -392,7 +392,7 @@ decompressor_reset_session(ZstdDecompressor *self,

PyObject *
stream_decompress(ZstdDecompressor *self, Py_buffer *data, Py_ssize_t max_length,
const decompress_type type)
decompress_type type)
{
Py_ssize_t initial_buffer_size = -1;
ZSTD_inBuffer in;
Expand Down Expand Up @@ -449,21 +449,21 @@ stream_decompress(ZstdDecompressor *self, Py_buffer *data, Py_ssize_t max_length
use_input_buffer = 1;

/* Unconsumed data size in input_buffer */
const size_t used_now = self->in_end - self->in_begin;
size_t used_now = self->in_end - self->in_begin;
assert(self->in_end > self->in_begin);

/* Number of bytes we can append to input buffer */
const size_t avail_now = self->input_buffer_size - self->in_end;
size_t avail_now = self->input_buffer_size - self->in_end;
assert(self->input_buffer_size >= self->in_end);

/* Number of bytes we can append if we move existing contents to
beginning of buffer */
const size_t avail_total = self->input_buffer_size - used_now;
size_t avail_total = self->input_buffer_size - used_now;
assert(self->input_buffer_size >= used_now);

if (avail_total < (size_t) data->len) {
char *tmp;
const size_t new_size = used_now + data->len;
size_t new_size = used_now + data->len;

/* Allocate with new size */
tmp = PyMem_Malloc(new_size);
Expand Down Expand Up @@ -537,7 +537,7 @@ stream_decompress(ZstdDecompressor *self, Py_buffer *data, Py_ssize_t max_length
self->in_end = 0;
}
} else {
const size_t data_size = in.size - in.pos;
size_t data_size = in.size - in.pos;

self->needs_input = 0;

Expand Down
0