8000 gh-129349: Accept bytes in bytes.fromhex()/bytearray.fromhex() by lordmauve · Pull Request #129844 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-129349: Accept bytes in bytes.fromhex()/bytearray.fromhex() #129844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Mar 12, 2025
Merged

gh-129349: Accept bytes in bytes.fromhex()/bytearray.fromhex() #129844

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
81d7ea2
Accept bytes in bytes.fromhex()/bytearray.fromhex()
lordmauve Feb 8, 2025
ea02542
Update documentation
lordmauve Feb 8, 2025
76d28ce
Apply suggestions from @picnixz in #129844
lordmauve Feb 9, 2025
cff091c
Fix sphinx-lint
lordmauve Mar 7, 2025
be92188
Mention bytearray in docs and blurb
lordmauve Mar 7, 2025
b0e3eb5
Use buffer protocol to support all byte-like objects
lordmauve Mar 7, 2025
ef92b4e
Insert missing braces
lordmauve Mar 7, 2025
685f583
Update Doc/library/stdtypes.rst
vstinner Mar 8, 2025
f859c3c
Add test for invalid type in bytes.fromhex()
lordmauve Mar 9, 2025
31b9ab0
PEP-7
lordmauve Mar 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Doc/library/stdtypes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2744,6 +2744,10 @@ data and are closely related to string objects in a variety of other ways.
:meth:`bytes.fromhex` now skips all ASCII whitespace in the string,
not just spaces.

.. versionchanged:: next
:meth:`bytes.fromhex` now accepts ASCII :class:`bytes` and
:term:`bytes-like objects <bytes-like object>` as input.

A reverse conversion function exists to transform a bytes object into its
hexadecimal representation.

Expand Down Expand Up @@ -2829,6 +2833,10 @@ objects.
:meth:`bytearray.fromhex` now skips all ASCII whitespace in the string,
not just spaces.

.. versionchanged:: next
:meth:`bytearray.fromhex` now accepts ASCII :class:`bytes` and
:term:`bytes-like objects <bytes-like object>` as input.

A reverse conversion function exists to transform a bytearray object into its
hexadecimal representation.

Expand Down
4 changes: 4 additions & 0 deletions Doc/whatsnew/3.14.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,10 @@ Other language changes
(with :func:`format` or :ref:`f-strings`).
(Contrubuted by Sergey B Kirpichev in :gh:`87790`.)

* The :func:`bytes.fromhex` and :func:`bytearray.fromhex` methods now accept
ASCII :class:`bytes` and :term:`bytes-like objects <bytes-like object>`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: are bytes also bytes-like objects? if so, you can just link the term. Or more generally, isn't it objects that support the buffer protocol?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but it seems less accessible to users to just link bytes-like object because that dives into a description of the buffer protocol which is lower level than the audience I was writing for in builtins docs.

Maybe the fault is with :term:\bytes-like object`` because it could just list types that duck-type like bytes.

So I hedged and did both.

Copy link
Member
@picnixz picnixz Mar 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could sa "that support the buffer protocol such as memoryviews" and add a link to whatever example of a type that supports the buffer protocol you used

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to say "bytes and bytes-like", it's more explicit.

(Contributed by Daniel Pope in :gh:`129349`.)

* ``\B`` in :mod:`regular expression <re>` now matches empty input string.
Now it is always the opposite of ``\b``.
(Contributed by Serhiy Storchaka in :gh:`124130`.)
Expand Down
23 changes: 22 additions & 1 deletion Lib/test/test_bytes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,13 +450,34 @@ def test_fromhex(self):

# check that ASCII whitespace is ignored
self.assertEqual(self.type2test.fromhex(' 1A\n2B\t30\v'), b)
self.assertEqual(self.type2test.fromhex(b' 1A\n2B\t30\v'), b)
for c in "\x09\x0A\x0B\x0C\x0D\x20":
self.assertEqual(self.type2test.fromhex(c), self.type2test())
for c in "\x1C\x1D\x1E\x1F\x85\xa0\u2000\u2002\u2028":
self.assertRaises(ValueError, self.type2test.fromhex, c)

# Check that we can parse bytes and bytearray
tests = [
("bytes", bytes),
("bytearray", bytearray),
("memoryview", memoryview),
("array.array", lambda bs: array.array('B', bs)),
]
for name, factory in tests:
with self.subTest(name=name):
self.assertEqual(self.type2test.fromhex(factory(b' 1A 2B 30 ')), b)

# Invalid bytes are rejected
for u8 in b"\0\x1C\x1D\x1E\x1F\x85\xa0":
b = bytes([30, 31, u8])
self.assertRaises(ValueError, self.type2test.fromhex, b)

self.assertEqual(self.type2test.fromhex('0000'), b'\0\0')
self.assertRaises(TypeError, self.type2test.fromhex, b'1B')
with self.assertRaisesRegex(
TypeError,
r'fromhex\(\) argument must be str or bytes-like, not tuple',
):
self.type2test.fromhex(())
self.assertRaises(ValueError, self.type2test.fromhex, 'a')
self.assertRaises(ValueError, self.type2test.fromhex, 'rt')
self.assertRaises(ValueError, self.type2test.fromhex, '1a b cd')
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Core_and_Builtins/2025-02-08-09-55-33.gh-issue-129349.PkcG-l.rst
View file
Open in desktop
67E6
Original file line numberDiff line number Diff line change
@@ -0,0 +1,2 @@
:meth:`bytes.fromhex` and :meth:`bytearray.fromhex` now accepts ASCII
:class:`bytes` and :term:`bytes-like objects <bytes-like object>`.
6 changes: 3 additions & 3 deletions Objects/bytearrayobject.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2533,7 +2533,7 @@ bytearray_splitlines_impl(PyByteArrayObject *self, int keepends)
@classmethod
bytearray.fromhex

string: unicode
string: object
/

Create a bytearray object from a string of hexadecimal numbers.
Expand All @@ -2543,8 +2543,8 @@ Example: bytearray.fromhex('B9 01EF') -> bytearray(b'\\xb9\\x01\\xef')
[clinic start generated code]*/

static PyObject *
bytearray_fromhex_impl(PyTypeObject *type, PyObject *string)
/*[clinic end generated code: output=8f0f0b6d30fb3ba0 input=f033a16d1fb21f48]*/
bytearray_fromhex(PyTypeObject *type, PyObject *string)
/*[clinic end generated code: output=da84dc708e9c4b36 input=7e314e5b2d7ab484]*/
{
PyObject *result = _PyBytes_FromHex(string, type == &PyByteArray_Type);
if (type != &PyByteArray_Type && result != NULL) {
Expand Down
72 changes: 49 additions & 23 deletions Objects/bytesobject.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2484,7 +2484,7 @@ bytes_splitlines_impl(PyBytesObject *self, int keepends)
@classmethod
bytes.fromhex

string: unicode
string: object
/

Create a bytes object from a string of hexadecimal numbers.
Expand All @@ -2494,8 +2494,8 @@ Example: bytes.fromhex('B9 01EF') -> b'\\xb9\\x01\\xef'.
[clinic start generated code]*/

static PyObject *
bytes_fromhex_impl(PyTypeObject *type, PyObject *string)
/*[clinic end generated code: output=0973acc63661bb2e input=bf4d1c361670acd3]*/
bytes_fromhex(PyTypeObject *type, PyObject *string)
/*[clinic end generated code: output=d458ec88195da6b3 input=f37d98ed51088a21]*/
{
PyObject *result = _PyBytes_FromHex(string, 0);
if (type != &PyBytes_Type && result != NULL) {
Expand All @@ -2510,37 +2510,55 @@ _PyBytes_FromHex(PyObject *string, int use_bytearray)
char *buf;
Py_ssize_t hexlen, invalid_char;
unsigned int top, bot;
const Py_UCS1 *str, *end;
const Py_UCS1 *str, *start, *end;
_PyBytesWriter writer;
Py_buffer view;
view.obj = NULL;

_PyBytesWriter_Init(&writer);
writer.use_bytearray = use_bytearray;

assert(PyUnicode_Check(string));
hexlen = PyUnicode_GET_LENGTH(string);
if (PyUnicode_Check(string)) {
hexlen = PyUnicode_GET_LENGTH(string);

if (!PyUnicode_IS_ASCII(string)) {
const void *data = PyUnicode_DATA(string);
int kind = PyUnicode_KIND(string);
Py_ssize_t i;
if (!PyUnicode_IS_ASCII(string)) {
const void *data = PyUnicode_DATA(string);
int kind = PyUnicode_KIND(string);
Py_ssize_t i;

/* search for the first non-ASCII character */
for (i = 0; i < hexlen; i++) {
if (PyUnicode_READ(kind, data, i) >= 128)
break;
/* search for the first non-ASCII character */
for (i = 0; i < hexlen; i++) {
if (PyUnicode_READ(kind, data, i) >= 128)
break;
}
invalid_char = i;
goto error;
}
invalid_char = i;
goto error;
}

assert(PyUnicode_KIND(string) == PyUnicode_1BYTE_KIND);
str = PyUnicode_1BYTE_DATA(string);
assert(PyUnicode_KIND(string) == PyUnicode_1BYTE_KIND);
str = PyUnicode_1BYTE_DATA(string);
}
else if (PyObject_CheckBuffer(string)) {
if (PyObject_GetBuffer(string, &view, PyBUF_SIMPLE) != 0) {
return NULL;
}
hexlen = view.len;
str = view.buf;
}
else {
PyErr_Format(PyExc_TypeError,
"fromhex() argument must be str or bytes-like, not %T",
string);
return NULL;
}

/* This overestimates if there are spaces */
buf = _PyBytesWriter_Alloc(&writer, hexlen / 2);
if (buf == NULL)
return NULL;
if (buf == NULL) {
goto release_buffer;
}

start = str;
end = str + hexlen;
while (str < end) {
/* skip over spaces in the input */
Expand All @@ -2554,7 +2572,7 @@ _PyBytes_FromHex(PyObject *string, int use_bytearray)

top = _PyLong_DigitValue[*str];
if (top >= 16) {
invalid_char = str - PyUnicode_1BYTE_DATA(string);
invalid_char = str - start;
goto error;
}
str++;
Expand All @@ -2565,7 +2583,7 @@ _PyBytes_FromHex(PyObject *string, int use_bytearray)
if (str >= end){
invalid_char = -1;
} else {
invalid_char = str - PyUnicode_1BYTE_DATA(string);
invalid_char = str - start;
}
goto error;
}
Expand All @@ -2574,6 +2592,9 @@ _PyBytes_FromHex(PyObject *string, int use_bytearray)
*buf++ = (unsigned char)((top << 4) + bot);
}

if (view.obj != NULL) {
PyBuffer_Release(&view);
}
return _PyBytesWriter_Finish(&writer, buf);

error:
Expand All @@ -2586,6 +2607,11 @@ _PyBytes_FromHex(PyObject *string, int use_bytearray)
"fromhex() arg at position %zd", invalid_char);
}
_PyBytesWriter_Dealloc(&writer);

release_buffer:
if (view.obj != NULL) {
PyBuffer_Release(&view);
}
return NULL;
}

Expand Down
22 changes: 1 addition & 21 deletions Objects/clinic/bytearrayobject.c.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 1 addition & 21 deletions Objects/clinic/bytesobject.c.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
0