8000 gh-129349: Accept bytes in bytes.fromhex()/bytearray.fromhex() by lordmauve · Pull Request #129844 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-129349: Accept bytes in bytes.fromhex()/bytearray.fromhex() #129844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Mar 12, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Doc/library/stdtypes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2744,6 +2744,10 @@ data and are closely related to string objects in a variety of other ways.
:meth:`bytes.fromhex` now skips all ASCII whitespace in the string,
not just spaces.

.. versionchanged:: next
:meth:`bytes.fromhex` now accepts ASCII :class:`bytes` and
:term:`bytes-like objects <bytes-like object>` as input.

A reverse conversion function exists to transform a bytes object into its
hexadecimal representation.

Expand Down Expand Up @@ -2829,6 +2833,10 @@ objects.
:meth:`bytearray.fromhex` now skips all ASCII whitespace in the string,
not just spaces.

.. versionchanged:: next
:meth:`bytearray.fromhex` now accepts ASCII :class:`bytes` and
:term:`bytes-like objects <bytes-like object>` as input.

A reverse conversion function exists to transform a bytearray object into its
hexadecimal representation.

Expand Down
4 changes: 4 additions & 0 deletions Doc/whatsnew/3.14.rst
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,10 @@ Other language changes
(with :func:`format` or :ref:`f-strings`).
(Contrubuted by Sergey B Kirpichev in :gh:`87790`.)

* The :func:`bytes.fromhex` and :func:`bytearray.fromhex` methods now accept
ASCII :class:`bytes` and :term:`bytes-like objects <bytes-like object>`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: are bytes also bytes-like objects? if so, you can just link the term. Or more generally, isn't it objects that support the buffer protocol?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but it seems less accessible to users to just link bytes-like object because that dives into a description of the buffer protocol which is lower level than the audience I was writing for in builtins docs.

Maybe the fault is with :term:\bytes-like object`` because it could just list types that duck-type like bytes.

So I hedged and did both.

Copy link
Member
@picnixz picnixz Mar 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could sa "that support the buffer protocol such as memoryviews" and add a link to whatever example of a type that supports the buffer protocol you used

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to say "bytes and bytes-like", it's more explicit.

(Contributed by Daniel Pope in :gh:`129349`.)

* ``\B`` in :mod:`regular expression <re>` now matches empty input string.
Now it is always the opposite of ``\b``.
(Contributed by Serhiy Storchaka in :gh:`124130`.)
Expand Down
23 changes: 22 additions & 1 deletion Lib/test/test_bytes.py
Original file line number Diff line number Diff line change
Expand Up @@ -450,13 +450,34 @@ def test_fromhex(self):

# check that ASCII whitespace is ignored
self.assertEqual(self.type2test.fromhex(' 1A\n2B\t30\v'), b)
self.assertEqual(self.type2test.fromhex(b' 1A\n2B\t30\v'), b)
for c in "\x09\x0A\x0B\x0C\x0D\x20":
self.assertEqual(self.type2test.fromhex(c), self.type2test())
for c in "\x1C\x1D\x1E\x1F\x85\xa0\u2000\u2002\u2028":
self.assertRaises(ValueError, self.type2test.fromhex, c)

# Check that we can parse bytes and bytearray
tests = [
("bytes", bytes),
("bytearray", bytearray),
("memoryview", memoryview),
("array.array", lambda bs: array.array('B', bs)),
]
for name, factory in tests:
with self.subTest(name=name):
self.assertEqual(self.type2test.fromhex(factory(b' 1A 2B 30 ')), b)

# Invalid bytes are rejected
for u8 in b"\0\x1C\x1D\x1E\x1F\x85\xa0":
b = bytes([30, 31, u8])
self.assertRaises(ValueError, self.type2test.fromhex, b)

self.assertEqual(self.type2test.fromhex('0000'), b'\0\0')
self.assertRaises(TypeError, self.type2test.fromhex, b'1B')
with self.assertRaisesRegex(
TypeError,
r'fromhex\(\) argument must be str or bytes-like, not tuple',
):
self.type2test.fromhex(())
self.assertRaises(ValueError, self.type2test.fromhex, 'a')
self.assertRaises(ValueError, self.type2test.fromhex, 'rt')
self.assertRaises(ValueError, self.type2test.fromhex, '1a b cd')
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
:meth:`bytes.fromhex` and :meth:`bytearray.fromhex` now accepts ASCII
:class:`bytes` and :term:`bytes-like objects <bytes-like object>`.
6 changes: 3 additions & 3 deletions Objects/bytearrayobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -2533,7 +2533,7 @@ bytearray_splitlines_impl(PyByteArrayObject *self, int keepends)
@classmethod
bytearray.fromhex

string: unicode
string: object
/

Create a bytearray object from a string of hexadecimal numbers.
Expand All @@ -2543,8 +2543,8 @@ Example: bytearray.fromhex('B9 01EF') -> bytearray(b'\\xb9\\x01\\xef')
[clinic start generated code]*/

static PyObject *
bytearray_fromhex_impl(PyTypeObject *type, PyObject *string)
/*[clinic end generated code: output=8f0f0b6d30fb3ba0 input=f033a16d1fb21f48]*/
bytearray_fromhex(PyTypeObject *type, PyObject *string)
/*[clinic end generated code: output=da84dc708e9c4b36 input=7e314e5b2d7ab484]*/
{
PyObject *result = _PyBytes_FromHex(string, type == &PyByteArray_Type);
if (type != &PyByteArray_Type && result != NULL) {
Expand Down
72 changes: 49 additions & 23 deletions Objects/bytesobject.c
F438
Original file line number Diff line number Diff line change
Expand Up @@ -2484,7 +2484,7 @@ bytes_splitlines_impl(PyBytesObject *self, int keepends)
@classmethod
bytes.fromhex

string: unicode
string: object
/

Create a bytes object from a string of hexadecimal numbers.
Expand All @@ -2494,8 +2494,8 @@ Example: bytes.fromhex('B9 01EF') -> b'\\xb9\\x01\\xef'.
[clinic start generated code]*/

static PyObject *
bytes_fromhex_impl(PyTypeObject *type, PyObject *string)
/*[clinic end generated code: output=0973acc63661bb2e input=bf4d1c361670acd3]*/
bytes_fromhex(PyTypeObject *type, PyObject *string)
/*[clinic end generated code: output=d458ec88195da6b3 input=f37d98ed51088a21]*/
{
PyObject *result = _PyBytes_FromHex(string, 0);
if (type != &PyBytes_Type && result != NULL) {
Expand All @@ -2510,37 +2510,55 @@ _PyBytes_FromHex(PyObject *string, int use_bytearray)
char *buf;
Py_ssize_t hexlen, invalid_char;
unsigned int top, bot;
const Py_UCS1 *str, *end;
const Py_UCS1 *str, *start, *end;
_PyBytesWriter writer;
Py_buffer view;
view.obj = NULL;

_PyBytesWriter_Init(&writer);
writer.use_bytearray = use_bytearray;

assert(PyUnicode_Check(string));
hexlen = PyUnicode_GET_LENGTH(string);
if (PyUnicode_Check(string)) {
hexlen = PyUnicode_GET_LENGTH(string);

if (!PyUnicode_IS_ASCII(string)) {
const void *data = PyUnicode_DATA(string);
int kind = PyUnicode_KIND(string);
Py_ssize_t i;
if (!PyUnicode_IS_ASCII(string)) {
const void *data = PyUnicode_DATA(string);
int kind = PyUnicode_KIND(string);
Py_ssize_t i;

/* search for the first non-ASCII character */
for (i = 0; i < hexlen; i++) {
if (PyUnicode_READ(kind, data, i) >= 128)
break;
/* search for the first non-ASCII character */
for (i = 0; i < hexlen; i++) {
if (PyUnicode_READ(kind, data, i) >= 128)
break;
}
invalid_char = i;
goto error;
}
invalid_char = i;
goto error;
}

assert(PyUnicode_KIND(string) == PyUnicode_1BYTE_KIND);
str = PyUnicode_1BYTE_DATA(string);
assert(PyUnicode_KIND(string) == PyUnicode_1BYTE_KIND);
str = PyUnicode_1BYTE_DATA(string);
}
else if (PyObject_CheckBuffer(string)) {
if (PyObject_GetBuffer(string, &view, PyBUF_SIMPLE) != 0) {
return NULL;
}
hexlen = view.len;
str = view.buf;
}
else {
PyErr_Format(PyExc_TypeError,
"fromhex() argument must be str or bytes-like, not %T",
string);
return NULL;
}

/* This overestimates if there are spaces */
buf = _PyBytesWriter_Alloc(&writer, hexlen / 2);
if (buf == NULL)
return NULL;
if (buf == NULL) {
goto release_buffer;
}

start = str;
end = str + hexlen;
while (str < end) {
/* skip over spaces in the input */
Expand All @@ -2554,7 +2572,7 @@ _PyBytes_FromHex(PyObject *string, int use_bytearray)

top = _PyLong_DigitValue[*str];
if (top >= 16) {
invalid_char = str - PyUnicode_1BYTE_DATA(string);
invalid_char = str - start;
goto error;
}
str++;
Expand All @@ -2565,7 +2583,7 @@ _PyBytes_FromHex(PyObject *string, int use_bytearray)
if (str >= end){
invalid_char = -1;
} else {
invalid_char = str - PyUnicode_1BYTE_DATA(string);
invalid_char = str - start;
}
goto error;
}
Expand All @@ -2574,6 +2592,9 @@ _PyBytes_FromHex(PyObject *string, int use_bytearray)
*buf++ = (unsigned char)((top << 4) + bot);
}

if (view.obj != NULL) {
PyBuffer_Release(&view);
}
return _PyBytesWriter_Finish(&writer, buf);

error:
Expand All @@ -2586,6 +2607,11 @@ _PyBytes_FromHex(PyObject *string, int use_bytearray)
"fromhex() arg at position %zd", invalid_char);
}
_PyBytesWriter_Dealloc(&writer);

release_buffer:
if (view.obj != NULL) {
PyBuffer_Release(&view);
}
return NULL;
}

Expand Down
22 changes: 1 addition & 21 deletions Objects/clinic/bytearrayobject.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 1 addition & 21 deletions Objects/clinic/bytesobject.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
0