8000 bpo-39337: Add a test case for normalizing of codec names by shihai1991 · Pull Request #19069 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

bpo-39337: Add a test case for normalizing of codec names #19069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Oct 8, 2020
Merged

bpo-39337: Add a test case for normalizing of codec names #19069

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
e9ecac0
add a test case to against nomralizeing() of codecs
shihai1991 Mar 19, 2020
cf46290
remove redundant operation
shihai1991 Mar 19, 2020
59911e1
not clear all codec_search_paths
shihai1991 Mar 20, 2020
aa3c4d0
remove codecs_unregister()
shihai1991 Mar 21, 2020
aa4bf51
remove useless head file
shihai1991 Mar 21, 2020
8ce5956
remove useless import
shihai1991 Mar 21, 2020
41a1672
retrigger checks
shihai1991 Apr 2, 2020
00b300c
Merge remote-tracking branch 'origin/master' into bpo_1635741_1
shihai1991 Sep 8, 2020
f48e5d9
add codecs_unregister() in testinternalcapi
shihai1991 Sep 10, 2020
3795420
update code
shihai1991 Sep 10, 2020
e3b9365
Merge remote-tracking branch 'origin/master' into bpo_1635741_1
shihai1991 Sep 29, 2020
a2887f3
Using codecs.unregister() in test case
shihai1991 Sep 29, 2020
632a448
apply Tal's comment
shihai1991 Sep 29, 2020
bb6e78c
apply victor's comment
shihai1991 Oct 3, 2020
72e2435
apply tal's comment
shihai1991 Oct 3, 2020
26fcbe4
apply victor's comment
shihai1991 Oct 5, 2020
37b80e0
apply victor's comment
shihai1991 Oct 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
add a test case to against nomralizeing() of codecs
  • Loading branch information
@shihai1991
shihai1991 committed Mar 19, 2020
commit e9ecac068485efd7947340cbfaf819b9da1289ca
16 changes: 16 additions & 0 deletions Lib/test/test_codecs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from unittest import mock

from test import support
import _testinternalcapi

try:
import _testcapi
Expand Down Expand Up @@ -48,6 +49,7 @@ class CPINFOEXW(ctypes.Structure):
info = CPINFOEXW()
return GetCPInfoEx(cp, 0, info)


class Queue(object):
"""
queue: write bytes at one end, read bytes from the other end
Expand Down Expand Up @@ -3401,5 +3403,19 @@ def test_rot13_func(self):
'To be, or not to be, that is the question')


class ZNormalizedTest(unittest.TestCase):
"""Test the normalizestring function via codecs module"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that the codec names are normalized via the normalizestring function is an implementation detail, so the test should be written ignoring that detail, to the extent possible. This includes its documentation, so...

Suggested change
"""Test the normalizestring function via codecs module"""
"""Test codec name normalization"""

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I considered this details before. Why I am still leave this detail?
IMHO, leave enough details can help developers to maintain.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're welcome to add a comment after the doc-string. I wouldn't want the doc-string to mention normalizestring, though, unless we change the test to call normalizestring() directly instead of codecs.lookup(), which I don't recommend.

From my experience, however, such comments often become outdated when the implementation changes. And outdated comments causing confusion are usually worse than no comments.

Also, if someone needs to see the details, they'll likely need to go through the code starting at codecs.lookup() anyways. It's relatively straightforward to follow: codecs imports lookup from _codecs, implemented in Modules/_codecsmodule.c. lookup() is a simple wrapper for _PyCodec_Lookup from Python/codecs.c. With the imports and wrappers out of the way, _PyCodec_Lookup calls normalizestring() near the top of the function, immediately after a bit of initialization.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my experience, however, such comments often become outdated when the implementation changes. And outdated comments causing confusion are usually worse than no comments.

Thanks for your share, Tal. It's make sense :)

def test_normalized_encoding(self):
def search_function(encoding):
if encoding == "aaa_8":
return (1, 2, 3, 4)
else:
return (None, None, None, None)
_testinternalcapi.codecs_unregister()
codecs.register(search_function)
self.assertEqual((1, 2, 3, 4), codecs.lookup('AAA-8'))
self.assertEqual((None, None, None, None), codecs.lookup('BBB-8'))


if __name__ == "__main__":
unittest.main()
1 change: 0 additions & 1 deletion Modules/_testcapimodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2155,7 +2155,6 @@ codec_incrementaldecoder(PyObject *self, PyObject *args)
return PyCodec_IncrementalDecoder(encoding, errors);
}


/* Simple test of _PyLong_NumBits and _PyLong_Sign. */
static PyObject *
test_long_numbits(PyObject *self, PyObject *Py_UNUSED(ignored))
Expand Down
29 changes: 29 additions & 0 deletions Modules/_testinternalcapi.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

#include "Python.h"
#include "pycore_initconfig.h" // _Py_GetConfigsAsDict()
#include "pycore_pystate.h"


static PyObject *
Expand All @@ -29,9 +30,37 @@ get_recursion_depth(PyObject *self, PyObject *args)
}


static PyObject *
codecs_unregister(PyObject *self, PyObject *Py_UNUSED(args))
{
PyListObject *codec_search_path;
PyObject **item;
Py_ssize_t i;

PyThreadState *tstate = PyThreadState_Get();
PyInterpreterState *interp = PyThreadState_GetInterpreter(tstate);
codec_search_path = (PyListObject *)interp->codec_search_path;
item = codec_search_path->ob_item;

if (item != NULL) {
i = Py_SIZE(codec_search_path);
Py_SET_SIZE(codec_search_path, 0);
codec_search_path->ob_item = NULL;
codec_search_path->allocated = 0;
while (--i >= 0) {
Py_XDECREF(item[i]);
}
}
PyMem_FREE(item);

Py_RETURN_NONE;
}


static PyMethodDef TestMethods[] = {
{"get_configs", get_configs, METH_NOARGS},
{"get_recursion_depth", get_recursion_depth, METH_NOARGS},
{"codecs_unregister", codecs_unregister, METH_NOARGS},
{NULL, NULL} /* sentinel */
};

Expand Down
0