8000 Improve UTF-8 decode speed · Issue #126024 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

Improve UTF-8 decode speed #126024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
methane opened this issue Oct 27, 2024 · 0 comments · Fixed by #126025
Closed

Improve UTF-8 decode speed #126024

methane opened this issue Oct 27, 2024 · 0 comments · Fixed by #126025
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage type-feature A feature request or enhancement

Comments

@methane
Copy link
Member
methane commented Oct 27, 2024 •
edited by encukou
Loading

Feature or enhancement

Proposal:

DuckDB and orjson don't use PyUnicode_FromStringAndSize() but PyUnicode_4BYTE_DATA() because they think Python UTF-8 decoder is slow.

vs DuckDB

vs orjson

I don't create simple extension module that contains only orjson UTF-8 decoder because I am not familiar with Rust yet.

When running their benchmark suite, using PyUnicode_FromStringAndSize slows down decoding twitter.json. twitter.json contains many non-ASCII (almost UCS2) strings. Most of them are Japanese text in ~140 chars.

Why Python decoder slow & possible optimization

  • Current Python decoder tries ASCII string first. When input is not ASCII, decoder need to convert buffer string into latin1/UCS2/UCS4.
  • When UTF-8 is 120 bytes, decoder allocates 120 codepoitns. But if all codepoints are 3byte UTF-8 sequene, result string has 40 codepoints. Decoder need to reallocate string at end.

When text is long, reallocation cost is relatively small. But for short (~200byte) strings, reallocate is much slower than decoding.

We can avoide some reallocation without slowing down decoding ASCII?

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/53

Linked PRs
@methane methane added the type-feature A feature request or enhancement label Oct 27, 2024
@bedevere-app bedevere-app bot mentioned this issue Oct 27, 2024
Merged
@methane methane linked a pull request Oct 27, 2024 that will close this issue
gh-126024: optimize UTF-8 decoder for short non-ASCII string #126025
Merged
@methane methane added the performance Performance or resource usage label Oct 27, 2024
@Eclips4 Eclips4 added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Oct 27, 2024
@methane methane mentioned this issue Nov 1, 2024
Merged
methane added a commit that referenced this issue Nov 29, 2024
@methane
gh-126024: optimize UTF-8 decoder for short non-ASCII string (#126025)
322b486
@picnixz picnixz mentioned this issue Dec 3, 2024
Closed
@bedevere-app bedevere-app bot mentioned this issue Dec 3, 2024
Merged
colesbury pushed a commit that referenced this issue Dec 6, 2024
@picnixz
gh-126024: fix UBSan failure in unicodeobject.c:find_first_nonascii (…
36c6178 
…GH-127566)
colesbury added a commit to colesbury/cpython that referenced this issue Dec 6, 2024
@colesbury
Revert "pythongh-126024: fix UBSan failure in `unicodeobject.c:find_f…
8000
4b022a7 
…irst_nonascii` (pythonGH-127566)"

Some hypothesis tests are failing.

This reverts commit 36c6178.
This was referenced Dec 6, 2024
Closed
Closed
encukou pushed a commit that referenced this issue Dec 13, 2024
@methane
gh-126024: unicodeobject: optimize find_first_nonascii (GH-127790)
5dd775b 
Remove 1 branch.
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this issue Jan 8, 2025
@methane @srinivasreddy
pythongh-126024: optimize UTF-8 decoder for short non-ASCII string (p…
ea2aa0a 
…ython#126025)
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this issue Jan 8, 2025
@picnixz @srinivasreddy
pythongh-126024: fix UBSan failure in `unicodeobject.c:find_first_non…
3e0f748 
…ascii` (pythonGH-127566)
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this issue Jan 8, 2025
@methane @srinivasreddy
pythongh-126024: unicodeobject: optimize find_first_nonascii (pythonG…
2be555f 
…H-127790)

Remove 1 branch.
ebonnal pushed a commit to ebonnal/cpython that referenced this issue Jan 12, 2025
@methane @ebonnal
pythongh-126024: optimize UTF-8 decoder for short non-ASCII string (p…
19b9628 
…ython#126025)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

gh-126024: optimize UTF-8 decoder for short non-ASCII string methane/cpython
2 participants
@methane @Eclips4
0