Use-after-free in `unicode_escape` decoder with error handler #133767

sethmlarson · 2025-05-09T14:57:25Z

Crash report

What happened?

When using .decode("unicode_escape") with an error handler there is a use-after-free segfault.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Output from running 'python -VV' on the command line:

No response

Linked PRs

gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler #129648
[3.14] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) #133942
[3.13] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) #133944
[3.12] Fix use-after-free in the unicode-escape decoder with error handler (GH-133767) #134255
[3.12] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944) #134337
[3.11] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944) #134341
[3.10] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944) #134345
[3.9] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944) #134346

The text was updated successfully, but these errors were encountered:

…rror handler (GH-129648) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal().

…h an error handler (pythonGH-129648) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…der with an error handler (pythonGH-129648) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…th an error handler (GH-129648) (GH-133942) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

bedevere-app · 2025-05-19T17:05:00Z

GH-134255 is a backport of this pull request to the 3.12 branch.

…th an error handler (GH-129648) (GH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…der with an error handler (pythonGH-129648) (pythonGH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…der with an error handler (pythonGH-129648) (pythonGH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…der with an error handler (pythonGH-129648) (pythonGH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) (cherry picked from commit 0c33e5b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…er with an error handler (pythonGH-129648) (pythonGH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) (cherry picked from commit 0c33e5b) (cherry picked from commit 8b528ca) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…th an error handler (GH-129648) (GH-133944) (#134337) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8)

…th an error handler (GH-129648) (GH-133944) (GH-134341) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…th an error handler (GH-129648) (GH-133944) (GH-134345) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) (cherry picked from commit 0c33e5b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…h an error handler (GH-129648) (GH-133944) (#134346) * [3.9] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) (cherry picked from commit 0c33e5b) (cherry picked from commit 8b528ca) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

… an error handler Cut disused recode_encoding logic in _PyBytes_DecodeEscape. All call sites pass NULL for `recode_encoding`, so this path is completely untested. That's been true since before Python 3.0. It adds significant complexity to this logic, so it's best to take it out. All call sites now have a literal NULL, and that's been true since commit 768921c eliminated a conditional (`foo ? bar : NULL`) at the call site in Python/ast.c where we're parsing a bytes literal. But even before then, that condition `foo` had been a constant since unadorned string literals started meaning Unicode, in commit 572dbf8 aka v3.0a1~1035 . The `unicode` parameter is already unused, so mark it as unused too. The code that acted on it was also taken out before Python 3.0, in commit 8d30cc0 aka v3.0a1~1031 . The function (PyBytes_DecodeEscape) is exposed in the API, but it's never been documented. Fixes: bsc#1243273 (CVE-2025-4516) Fixes: gh#python#133767 From-PR: gh#python/cpython!134346 Patch: CVE-2025-4516-DecodeError-handler.patch

mcepl · 2025-06-04T08:13:55Z

I have uploaded above my draft of the backport to 3.6. Any review or comments would be more than welcome.

sethmlarson added the type-crash A hard crash of the interpreter, possibly with a core dump label May 9, 2025

sethmlarson changed the title ~~Use-after-free in unicode escape decoder with error handler~~ Use-after-free in unicode_escape decoder with error handler May 9, 2025

serhiy-storchaka self-assigned this May 9, 2025

bedevere-app bot mentioned this issue May 9, 2025

gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler #129648

Merged

bedevere-app bot mentioned this issue May 12, 2025

[3.14] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) #133942

Merged

bedevere-app bot mentioned this issue May 12, 2025

[3.13] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) #133944

Merged

serhiy-storchaka added the release-blocker label May 13, 2025

ezio-melotti added this to Release and Deferred blockers 🚫 May 13, 2025

github-project-automation bot moved this to Todo in Release and Deferred blockers 🚫 May 13, 2025

bedevere-app bot mentioned this issue May 19, 2025

[3.12] Fix use-after-free in the unicode-escape decoder with error handler (GH-133767) #134255

Closed

bedevere-app bot mentioned this issue May 20, 2025

[3.12] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944) #134337

Merged

This was referenced May 20, 2025

[3.11] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944) #134341

Merged

[3.10] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944) #134345

Merged

bedevere-app bot mentioned this issue May 20, 2025

[3.9] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944) #134346

Merged

ambv closed this as completed Jun 3, 2025

github-project-automation bot moved this from Todo to Done in Release and Deferred blockers 🚫 Jun 3, 2025

xry111 mentioned this issue Jun 4, 2025

CVE-2025-4516 in Python 3 AOSC-Dev/aosc-os-abbs#11247

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Use-after-free in `unicode_escape` decoder with error handler #133767

Use-after-free in `unicode_escape` decoder with error handler #133767

Uh oh!

Uh oh!

Uh oh!

Use-after-free in unicode_escape decoder with error handler #133767

Use-after-free in unicode_escape decoder with error handler #133767

Comments

Uh oh!

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Uh oh!

Uh oh!

Use-after-free in `unicode_escape` decoder with error handler #133767

Use-after-free in `unicode_escape` decoder with error handler #133767