8000 gh-110289: C API: Add PyUnicode_EqualToUTF8() function by serhiy-storchaka · Pull Request #110297 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-110289: C API: Add PyUnicode_EqualToUTF8() function #110297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Oct 11, 2023
Merged

gh-110289: C API: Add PyUnicode_EqualToUTF8() function #110297

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
d39945e
gh-110289: C API: Add PyUnicode_EqualToString() function
serhiy-storchaka Oct 3, 2023
4793161
Add tests and address review comments.
serhiy-storchaka Oct 3, 2023
8b24911
Merge branch 'main' into capi-PyUnicode_EqualToString
serhiy-storchaka Oct 3, 2023
c55f9ac
Apply suggestions from code review
serhiy-storchaka Oct 4, 2023
bdf2f1e
Address some of review comments and test the UTF-8 cache.
serhiy-storchaka Oct 4, 2023
7223c14
Address review comments.
serhiy-storchaka Oct 4, 2023
b271327
Apply suggestions from code review
serhiy-storchaka Oct 4, 2023
6f26ad6
Remove trailing spaces.
serhiy-storchaka Oct 4, 2023
dd124b8
Apply suggestions from code review
serhiy-storchaka Oct 4, 2023
76b9177
Use "z#" instead of "|y#".
serhiy-storchaka Oct 4, 2023
ee5781d
Apply suggestions from code review
serhiy-storchaka Oct 5, 2023
1a4eb7b
Add PyUnicode_EqualToUTF8AndSize().
serhiy-storchaka Oct 6, 2023
b124377
Apply suggestions from code review
serhiy-storchaka Oct 7, 2023
029f1a0
Add more parentheses.
serhiy-storchaka Oct 7, 2023
be2ffe8
Remove redundant arguments. 8000
serhiy-storchaka Oct 7, 2023
29b26f7
Merge branch 'main' into capi-PyUnicode_EqualToString
serhiy-storchaka Oct 7, 2023
78de49d
Turn docstrings into comments.
serhiy-storchaka Oct 10, 2023
fc79d5e
Merge branch 'main' into capi-PyUnicode_EqualToString
serhiy-storchaka Oct 10, 2023
19ad126
Add tests for empty strings.
serhiy-storchaka Oct 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address some of review comments and test the UTF-8 cache.
  • Loading branch information
@serhiy-storchaka
serhiy-storchaka committed Oct 4, 2023
commit bdf2f1e27cdc42ec976a7a23b83f0aade13a56ad
6 changes: 3 additions & 3 deletions Doc/c-api/unicode.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1398,10 +1398,10 @@ They all return ``NULL`` or ``-1`` if an exception occurs.

.. c:function:: int PyUnicode_EqualToUTF8(PyObject *unicode, const char *string)

Compare a Unicode object with a UTF-8 encoded C string and return true
if they are equal and false otherwise.
Compare a Unicode object with a UTF-8 encoded C string and return true (``1``)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Compare a Unicode object with a UTF-8 encoded C string and return true (``1``)
Compare a Unicode object with a UTF-8 encoded or ASCII encoding C string and return true (``1``)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "ASCII encoded"?

if they are equal and false (``0``) otherwise.
If the Unicode object contains null or surrogate characters or
the C string is not encoded to UTF-8 return 0.
the C string is not encoded to UTF-8 return false.

This function does not raise exceptions.

Expand Down
28 changes: 18 additions & 10 deletions Lib/test/test_capi/test_unicode.py
View file
Open in desktop
8000
Original file line number Diff line number Diff line change
Expand Up @@ -1302,26 +1302,34 @@ def test_comparewithasciistring(self):
def test_equaltoutf8(self):
"""Test PyUnicode_EqualToUTF8()"""
from _testcapi import unicode_equaltoutf8 as equaltoutf8
from _testcapi import unicode_asutf8andsize as asutf8andsize

strings = [
'abc', '\xa1\xa2\xa3', '\u4f60\u597d\u4e16',
'\U0001f600\U0001f601\U0001f602'
]
for s in strings:
# Call PyUnicode_AsUTF8AndSize() which creates the UTF-8
# encoded string cached in the Unicode object.
asutf8andsize(s, 0)
b = s.encode()
self.assertEqual(equaltoutf8(s, b), 1)
self.assertEqual(equaltoutf8(b.decode(), b), 1)
self.assertEqual(equaltoutf8(s, b), 1) # Use the UTF-8 cache.
s2 = b.decode() # New Unicode object without the UTF-8 cache.
self.assertEqual(equaltoutf8(s2, b), 1)
self.assertEqual(equaltoutf8(s + 'x', b + b'x'), 1)
self.assertEqual(equaltoutf8(s + 'x', b + b'y'), 0)
self.assertEqual(equaltoutf8(s + '\0', b + b'\0'), 0)
self.assertEqual(equaltoutf8(s, b + b'x'), 0)
self.assertEqual(equaltoutf8(s, b[:-1]), 0)
self.assertEqual(equaltoutf8(s, b[:-1] + b'x'), 0)

# surrogateescape
self.assertEqual(equaltoutf8('\udcfe', b'\xfe'), 0)
# surrogatepass
self.assertEqual(equaltoutf8('\udcfe', b'\xed\xb3\xbe'), 0)
self.assertEqual(equaltoutf8(s2, b + b'x'), 0)
self.assertEqual(equaltoutf8(s2, b[:-1]), 0)
self.assertEqual(equaltoutf8(s2, b[:-1] + b'x'), 0)

# Surrogate characters are always treated as not equal
self.assertEqual(equaltoutf8('\udcfe',
'\udcfe'.encode("utf8", "surrogateescape")), 0)
self.assertEqual(equaltoutf8('\udcfe',
'\udcfe'.encode("utf8", "surrogatepass")), 0)
self.assertEqual(equaltoutf8('\ud801',
'\ud801'.encode("utf8", "surrogatepass")), 0)

# CRASHES equaltoutf8(b'abc', b'abc')
# CRASHES equaltoutf8([], b'abc')
Expand Down
31 changes: 18 additions & 13 deletions Objects/unicodeobject.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10678,6 +10678,7 @@ PyUnicode_EqualToUTF8(PyObject *unicode, const char *str)
{
assert(_PyUnicode_CHECK(unicode));
assert(str);

if (PyUnicode_IS_ASCII(unicode)) {
size_t len = (size_t)PyUnicode_GET_LENGTH(unicode);
return strlen(str) == len &&
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to test the length first, to make the code more readable.

Like:

if (strlen(str) == len) {
    return 1;
}
return memcmp(...);

Same below.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the same in _PyUnicode_EqualToASCIIString().

How

if (!a) {
    return 0;
}
return b;

is more readable than simple return a && b;? It is what the && operator for.

Copy link
Member 67E6

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is more readable than simple return a && b;?

For me, it's easier to reason about a single test per line when I review code.

Keep a && b if you prefer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The readability problem as I see it, is that your && use has side effects; it is not a pure logic expression.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me, it's easier to reason about a single test per line when I review code.

Fortunately, every condition here is already on a separate line.

The readability problem as I see it, is that your && use has side effects; it is not a pure logic expression.

It is how && works in C. There is a lot of code like arg != NULL and PyDict_Check(arg) && PyDict_GET_SIZE(a A3E2 rg) > count. I do not think rewriting it in three ifs with gotos can improve readability.

Expand All @@ -10689,49 +10690,53 @@ PyUnicode_EqualToUTF8(PyObject *unicode, const char *str)
memcmp(PyUnicode_UTF8(unicode), str, len) == 0;
}

Py_UCS4 ch;
const unsigned char *s = (const unsigned char *)str;
Py_ssize_t len = PyUnicode_GET_LENGTH(unicode);
int kind = PyUnicode_KIND(unicode);
const void *data = PyUnicode_DATA(unicode);
/* Compare Unicode string and UTF-8 string */
for (Py_ssize_t i = 0; i < len; i++) {
ch = PyUnicode_READ(kind, data, i);
Py_UCS4 ch = PyUnicode_READ(kind, data, i);
if (ch == 0) {
return 0;
}
else if (ch < 0x80) {
if (ch != (unsigned char)*str++) {
if (s[0] != ch) {
return 0;
}
s += 1;
}
else if (ch < 0x800) {
if ((0xc0 | (ch >> 6)) != (unsigned char)*str++ ||
(0x80 | (ch & 0x3f)) != (unsigned char)*str++)
if (s[0] != (0xc0 | (ch >> 6)) ||
s[1] != (0x80 | (ch & 0x3f)))
{
return 0;
}
s += 2;
}
else if (ch < 0x10000) {
if (Py_UNICODE_IS_SURROGATE(ch) ||
(0xe0 | (ch >> 12)) != (unsigned char)*str++ ||
(0x80 | ((ch >> 6) & 0x3f)) != (unsigned char)*str++ ||
(0x80 | (ch & 0x3f)) != (unsigned char)*str++)
s[0] != (0xe0 | (ch >> 12)) ||
s[1] != (0x80 | ((ch >> 6) & 0x3f)) ||
s[2] != (0x80 | (ch & 0x3f)))
{
return 0;
}
s += 3;
}
else {
assert(ch <= MAX_UNICODE);
if ((0xf0 | (ch >> 18)) != (unsigned char)*str++ ||
(0x80 | ((ch >> 12) & 0x3f)) != (unsigned char)*str++ ||
(0x80 | ((ch >> 6) & 0x3f)) != (unsigned char)*str++ ||
(0x80 | (ch & 0x3f)) != (unsigned char)*str++)
if (s[0] != (0xf0 | (ch >> 18)) ||
s[1] != (0x80 | ((ch >> 12) & 0x3f)) ||
s[2] != (0x80 | ((ch >> 6) & 0x3f)) ||
s[3] != (0x80 | (ch & 0x3f)))
{
return 0;
}
s += 4;
}
}
return *str == 0;
return *s == 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose that if we return true at this point then we know that str is the utf8 representation of unicode, does it make sense to copy the contents into unicode->utf8 so that future operations can fast-path without needing to encode again?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It needs a separate research and discussion. The disadvantage is that it increases the consumed memory size, also it consumes some CPU time, so the benefit will be only if the UTF-8 cache is used in future.

If the idea turned out to be good, it can simply be implemented in the future.

Copy link
Contributor
@davidhewitt davidhewitt Oct 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes total sense. I guess this also sits in an awkward place where it's likely that the user is best suited to know whether or not they want the utf-8 cache populated, but it's also an implementation detail that we don't really want to expose to users. For now I'll just mark this comment as resolved. Edit I can't, probably lack permissions I guess.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyUnicode_EqualToUTF8() doesn't raise exception and cannot fail. Trying to allocate memory should not raise memory, but it sounds like a non-trivial side effect.

Worst case: 1 GB string, you call PyUnicode_EqualToUTF8() and suddenly, Python allocates 1 GB more. I would be surprised by this behavior.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's worth it to add a comment explaining why we don't cache the UTF-8 encoded string.

}

int
Expand Down
0