8000 gh-132983: Add `compression.zstd` and Python tests by emmatyping · Pull Request #133365 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-132983: Add compression.zstd and Python tests #133365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 56 commits into from
May 6, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
f4cd026
Add Python files
emmatyping Apr 30, 2025
4168895
Fix byteswarning in test
emmatyping Apr 30, 2025
a22fa9b
Remove shape tests
emmatyping May 3, 2025
e70e03b
Make namedtuples dataclasses
emmatyping May 4, 2025
cbf0ef8
Apply suggestions from AA-Turner
emmatyping May 4, 2025
298b369
Clean up chunk calculations in train_dict
emmatyping May 4, 2025
a22be68
Fix _CLValues instantiation
emmatyping May 4, 2025
b30ed02
More cleanup of train_/finalize_dict
emmatyping May 4, 2025
307a894
Have train_/finalize_dict take tuple not list
emmatyping May 4, 2025
dd716a4
Ensure trailing data raises errors
emmatyping May 4, 2025
9b4765b
Remove paramter bounds caching and unsupported...
emmatyping May 4, 2025
214cd60
Use kwargs for code clarity
emmatyping May 4, 2025
e1f53b1
Clean up imports in zstd tests
emmatyping May 4, 2025
4c00026
Use _1K instead of 1024 in tests
emmatyping May 4, 2025
99653d2
Move compression.zstd.zstdfile to compression.zstd._zstdfile
emmatyping May 4, 2025
e403a25
Change compressLevel_values to COMPRESSION_LEVEL_DEFAULT
emmatyping May 4, 2025
63625bc
Fix tests for change in error message
emmatyping May 4, 2025
1ea4b9a
Make parameter names snake case
emmatyping May 4, 2025
ad05da8
Replace compressionLevel_values re-export with COMPRESSION_LEVEL_DEFAULT
emmatyping May 4, 2025
e82e23d
Move zstd_support_multithread to tests and rename
emmatyping May 4, 2025
7801b6b
Update module docstring for compression.zstd
emmatyping May 4, 2025
df5d827
Clarify Strategy stability in docstring
emmatyping May 4, 2025
4ff48da
Fix formatting in tarfile
emmatyping May 4, 2025
c68a896
Remove zstd_support_multithread from __all__
emmatyping May 4, 2025
326400d
Add test_name from upstream
emmatyping May 4, 2025
2c0c9a1
Don't close tarfile if there is a BaseException
emmatyping May 4, 2025
49f3821
Use options kwarg in tests
emmatyping May 4, 2025
8ba6bda
Use options kwarg in tests in more places
emmatyping May 4, 2025
129d5e6
Adopt suggestions by Tomas R. for _zstdfile
emmatyping May 4, 2025
7d54d35
Formatting fixes in zstd tests
emmatyping May 4, 2025
03795ec
Improve docstrings for (de)compress
emmatyping May 4, 2025
01fcfcb
Fix some line length issues
emmatyping May 4, 2025
f04494c
Improve docstring on C/DParameter.bounds()
emmatyping May 4, 2025
caa40b1
Improve docstrings and formatting
emmatyping May 4, 2025
4584ec5
Add missing f string prefix
emmatyping May 4, 2025
3cafdc6
Fix weird indent in _zstdfile.py
emmatyping May 4, 2025
8cb0846
Use io.open instead of builtins.open
emmatyping May 5, 2025
c7d5d67
Remove _READER_CLASS from ZstdFile
emmatyping May 5, 2025
a56a22e
Adopt many suggestions from AA-Turner for ZstdFile
emmatyping May 5, 2025
7e919c8
Set self._buffer to None
emmatyping May 5, 2025
389faed
Move _nbytes to _zstdfile.py
emmatyping May 5, 2025
006ef2e
Move test_zstd to file
emmatyping May 5, 2025
c846b78
Rename C/DParameter to (De)CompressionParameter
emmatyping May 5, 2025
fa0cb0c
regen clinic
AA-Turner May 5, 2025
74e4d2b
Fix whitespace issue
emmatyping May 5, 2025
03fff3d
Remove makefile test dir
emmatyping May 5, 2025
a99c5dd
swap order of parameters in _get_param_bounds
AA-Turner May 5, 2025
a12a031
Merge branch 'main' into 3.14-zstd-python-code
emmatyping May 5, 2025
bf94aad
Sort imports
AA-Turner May 5, 2025
5b45ec7
Improve docstrings
AA-Turner May 5, 2025
b0eca5a
Remove comments
AA-Turner May 5, 2025
c0d0e10
Remove unused private variables
AA-Turner May 5, 2025
10f0cff
Misc changes (positional-only, style, error messages)
AA-Turner May 5, 2025
7f8c350
whitespace
AA-Turner May 5, 2025
bf4b07d
Remove _set_parameter_types
AA-Turner May 5, 2025
eaf46a8
Revert "Remove _set_parameter_types"
emmatyping May 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove _set_parameter_types
Python, in general, works on protocols rather than nominal or specific types. I think this check is overly restrictive, with only minor benefit. Users wanting validation can use static type checkers.
  • Loading branch information
AA-Turner committed May 5, 2025
commit bf4b07daade1a55da151db571fc581d80899e8db
4 changes: 0 additions & 4 deletions Lib/compression/zstd/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,3 @@ class Strategy(enum.IntEnum):
btopt = _zstd._ZSTD_btopt
btultra = _zstd._ZSTD_btultra
btultra2 = _zstd._ZSTD_btultra2


# Check validity of the CompressionParameter & DecompressionParameter types
_zstd._set_parameter_types(CompressionParameter, DecompressionParameter)
46 changes: 0 additions & 46 deletions Modules/_zstd/_zstdmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -510,49 +510,12 @@ _zstd__get_frame_info_impl(PyObject *module, Py_buffer *frame_buffer)
return Py_BuildValue("KI", decompressed_size, dict_id);
}

/*[clinic input]
_zstd._set_parameter_types

c_parameter_type: object(subclass_of='&PyType_Type')
CompressionParameter IntEnum type object
d_parameter_type: object(subclass_of='&PyType_Type')
DecompressionParameter IntEnum type object

Internal function, set CompressionParameter/DecompressionParameter types for validity check.
[clinic start generated code]*/

static PyObject *
_zstd__set_parameter_types_impl(PyObject *module, PyObject *c_parameter_type,
PyObject *d_parameter_type)
/*[clinic end generated code: output=a13d4890ccbd2873 input=4535545d903853d3]*/
{
_zstd_state* const mod_state = get_zstd_state(module);

if (!PyType_Check(c_parameter_type) || !PyType_Check(d_parameter_type)) {
PyErr_SetString(PyExc_ValueError,
"The two arguments should be CompressionParameter and "
"DecompressionParameter types.");
return NULL;
}

Py_XDECREF(mod_state->CParameter_type);
Py_INCREF(c_parameter_type);
mod_state->CParameter_type = (PyTypeObject*)c_parameter_type;

Py_XDECREF(mod_state->DParameter_type);
Py_INCREF(d_parameter_type);
mod_state->DParameter_type = (PyTypeObject*)d_parameter_type;

Py_RETURN_NONE;
}

static PyMethodDef _zstd_methods[] = {
_ZSTD__TRAIN_DICT_METHODDEF
_ZSTD__FINALIZE_DICT_METHODDEF
_ZSTD__GET_PARAM_BOUNDS_METHODDEF
_ZSTD_GET_FRAME_SIZE_METHODDEF
_ZSTD__GET_FRAME_INFO_METHODDEF
_ZSTD__SET_PARAMETER_TYPES_METHODDEF

{0}
};
Expand Down Expand Up @@ -766,9 +729,6 @@ static int _zstd_exec(PyObject *module) {
ADD_STR_TO_STATE_MACRO(write);
ADD_STR_TO_STATE_MACRO(flush);

mod_state->CParameter_type = NULL;
mod_state->DParameter_type = NULL;

/* Add variables to module */
if (add_vars_to_module(module) < 0) {
return -1;
Expand Down Expand Up @@ -852,9 +812,6 @@ _zstd_traverse(PyObject *module, visitproc visit, void *arg)
Py_VISIT(mod_state->ZstdDecompressor_type);

Py_VISIT(mod_state->ZstdError);

Py_VISIT(mod_state->CParameter_type);
Py_VISIT(mod_state->DParameter_type);
return 0;
}

Expand All @@ -876,9 +833,6 @@ _zstd_clear(PyObject *module)
Py_CLEAR(mod_state->ZstdDecompressor_type);

Py_CLEAR(mod_state->ZstdError);

Py_CLEAR(mod_state->CParameter_type);
Py_CLEAR(mod_state->DParameter_type);
return 0;
}

Expand Down
3 changes: 0 additions & 3 deletions Modules/_zstd/_zstdmodule.h
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,6 @@ struct _zstd_state {
PyTypeObject *ZstdCompressor_type;
PyTypeObject *ZstdDecompressor_type;
PyObject *ZstdError;

PyTypeObject *CParameter_type;
PyTypeObject *DParameter_type;
};

typedef struct {
Expand Down
76 changes: 1 addition & 75 deletions Modules/_zstd/clinic/_zstdmodule.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 0 additions & 8 deletions Modules/_zstd/compressor.c
6D40
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,6 @@ _PyZstd_set_c_parameters(ZstdCompressor *self, PyObject *level_or_options,
Py_ssize_t pos = 0;

while (PyDict_Next(level_or_options, &pos, &key, &value)) {
/* Check key type */
if (Py_TYPE(key) == mod_state->DParameter_type) {
PyErr_SetString(PyExc_TypeError,
"Key of compression option dict should "
"NOT be DecompressionParameter.");
return -1;
}

int key_v = PyLong_AsInt(key);
if (key_v == -1 && PyErr_Occurred()) {
PyErr_SetString(PyExc_ValueError,
Expand Down
8 changes: 0 additions & 8 deletions Modules/_zstd/decompressor.c
Original file line number Diff line number Diff line change
Expand Up @@ -80,14 +80,6 @@ _PyZstd_set_d_parameters(ZstdDecompressor *self, PyObject *options)

pos = 0;
while (PyDict_Next(options, &pos, &key, &value)) {
/* Check key type */
if (Py_TYPE(key) == mod_state->CParameter_type) {
PyErr_SetString(PyExc_TypeError,
"Key of decompression options dict should "
"NOT be CompressionParameter.");
return -1;
}

/* Both key & value should be 32-bit signed int */
int key_v = PyLong_AsInt(key);
if (key_v == -1 && PyErr_Occurred()) {
Expand Down
Loading
0