8000 Merge branch 'main' into run-pidfdchildwatcher-on-the-running-loop · python/cpython@d25f852 · GitHub
[go: up one dir, main page]
Skip to content

Commit d25f852

Browse files
JelleZijlstraJelleZijlstra
authored
Merge branch 'main' into run-pidfdchildwatcher-on-the-running-loop
2 parents 65a2db6 + 1523c9e commit d25f852

File tree

14 files changed

+140
-66
lines changed

14 files changed

+140
-66
lines changed

‎Doc/faq/design.rst

Lines changed: 30 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ reference or call the method from a particular class. In C++, if you want to
129129
use a method from a base class which is overridden in a derived class, you have
130130
to use the ``::`` operator -- in Python you can write
131131
``baseclass.methodname(self, <argument list>)``. This is particularly useful
132-
for :meth:`__init__` methods, and in general in cases where a derived class
132+
for :meth:`~object.__init__` methods, and in general in cases where a derived class
133133
method wants to extend the base class method of the same name and thus has to
134134
call the base class method somehow.
135135

@@ -232,7 +232,8 @@ Similar methods exist for bytes and bytearray objects.
232232
How fast are exceptions?
233233
------------------------
234234

235-
A try/except block is extremely efficient if no exceptions are raised. Actually
235+
A :keyword:`try`/:keyword:`except` block is extremely efficient if no exceptions
236+
are raised. Actually
236237
catching an exception is expensive. In versions of Python prior to 2.0 it was
237238
common to use this idiom::
238239

@@ -352,7 +353,7 @@ will probably run out of file descriptors::
352353
c = f.read(1)
353354

354355
Indeed, using CPython's reference counting and destructor scheme, each new
355-
assignment to *f* closes the previous file. With a traditional GC, however,
356+
assignment to ``f`` closes the previous file. With a traditional GC, however,
356357
those file objects will only get collected (and closed) at varying and possibly
357358
long intervals.
358359

@@ -376,10 +377,10 @@ Python to work with it.)
376377

377378
Traditional GC also becomes a problem when Python is embedded into other
378379
applications. While in a standalone Python it's fine to replace the standard
379-
malloc() and free() with versions provided by the GC library, an application
380-
embedding Python may want to have its *own* substitute for malloc() and free(),
380+
``malloc()`` and ``free()`` with versions provided by the GC library, an application
381+
embedding Python may want to have its *own* substitute for ``malloc()`` and ``free()``,
381382
and may not want Python's. Right now, CPython works with anything that
382-
implements malloc() and free() properly.
383+
implements ``malloc()`` and ``free()`` properly.
383384

384385

385386
Why isn't all memory freed when CPython exits?
@@ -401,14 +402,15 @@ Why are there separate tuple and list data types?
401402

402403
Lists and tuples, while similar in many respects, are generally used in
403404
fundamentally different ways. Tuples can be thought of as being similar to
404-
Pascal records or C structs; they're small collections of related data which may
405+
Pascal ``records`` or C ``structs``; they're small collections of related data which may
405406
be of different types which are operated on as a group. For example, a
406407
Cartesian coordinate is appropriately represented as a tuple of two or three
407408
numbers.
408409

409410
Lists, on the other hand, are more like arrays in other languages. They tend to
410411
hold a varying number of objects all of which have the same type and which are
411-
operated on one-by-one. For example, ``os.listdir('.')`` returns a list of
412+
operated on one-by-one. For example, :func:`os.listdir('.') <os.listdir>`
413+
returns a list of
412414
strings representing the files in the current directory. Functions which
413415
operate on this output would generally not break if you added another file or
414416
two to the directory.
@@ -444,9 +446,9 @@ far) under most circumstances, and the implementation is simpler.
444446

445447
Dictionaries work by computing a hash code for each key stored in the dictionary
446448
using the :func:`hash` built-in function. The hash code varies widely depending
447-
on the key and a per-process seed; for example, "Python" could hash to
448-
-539294296 while "python", a string that differs by a single bit, could hash
449-
to 1142331976. The hash code is then used to calculate a location in an
449+
on the key and a per-process seed; for example, ``'Python'`` could hash to
450+
``-539294296`` while ``'python'``, a string that differs by a single bit, could hash
451+
to ``1142331976``. The hash code is then used to calculate a location in an
450452
internal array where the value will be stored. Assuming that you're storing
451453
keys that all have different hash values, this means that dictionaries take
452454
constant time -- O(1), in Big-O notation -- to retrieve a key.
@@ -497,7 +499,8 @@ Some unacceptable solutions that have been proposed:
497499

498500
There is a trick to get around this if you need to, but use it at your own risk:
499501
You can wrap a mutable structure inside a class instance which has both a
500-
:meth:`__eq__` and a :meth:`__hash__` method. You must then make sure that the
502+
:meth:`~object.__eq__` and a :meth:`~object.__hash__` method.
503+
You must then make sure that the
501504
hash value for all such wrapper objects that reside in a dictionary (or other
502505
hash based structure), remain fixed while the object is in the dictionary (or
503506
other structure). ::
@@ -528,7 +531,7 @@ is True``) then ``hash(o1) == hash(o2)`` (ie, ``o1.__hash__() == o2.__hash__()``
528531
regardless of whether the object is in a dictionary or not. If you fail to meet
529532
these restrictions dictionaries and other hash based structures will misbehave.
530533

531-
In the case of ListWrapper, whenever the wrapper object is in a dictionary the
534+
In the case of :class:`!ListWrapper`, whenever the wrapper object is in a dictionary the
532535
wrapped list must not change to avoid anomalies. Don't do this unless you are
533536
prepared to think hard about the requirements and the consequences of not
534537
meeting them correctly. Consider yourself warned.
@@ -581,9 +584,9 @@ exhaustive test suites that exercise every line of code in a module.
581584
An appropriate testing discipline can help build large complex applications in
582585
Python as well as having interface specifications would. In fact, it can be
583586
better because an interface specification cannot test certain properties of a
584-
program. For example, the :meth:`append` method is expected to add new elements
587+
program. For example, the :meth:`list.append` method is expected to add new elements
585588
to the end of some internal list; an interface specification cannot test that
586-
your :meth:`append` implementation will actually do this correctly, but it's
589+
your :meth:`list.append` implementation will actually do this correctly, but it's
587590
trivial to check this property in a test suite.
588591

589592
Writing test suites is very helpful, and you might want to design your code to
@@ -599,14 +602,14 @@ Why is there no goto?
599602
In the 1970s people realized that unrestricted goto could lead
600603
to messy "spaghetti" code that was hard to understand and revise.
601604
In a high-level language, it is also unneeded as long as there
602-
are ways to branch (in Python, with ``if`` statements and ``or``,
603-
``and``, and ``if-else`` expressions) and loop (with ``while``
604-
and ``for`` statements, possibly containing ``continue`` and ``break``).
605+
are ways to branch (in Python, with :keyword:`if` statements and :keyword:`or`,
606+
:keyword:`and`, and :keyword:`if`/:keyword:`else` expressions) and loop (with :keyword:`while`
607+
and :keyword:`for` statements, possibly containing :keyword:`continue` and :keyword:`break`).
605608

606609
One can also use exceptions to provide a "structured goto"
607610
that works even across
608611
function calls. Many feel that exceptions can conveniently emulate all
609-
reasonable uses of the "go" or "goto" constructs of C, Fortran, and other
612+
reasonable uses of the ``go`` or ``goto`` constructs of C, Fortran, and other
610613
languages. For example::
611614

612615
class label(Exception): pass # declare a label
@@ -620,7 +623,7 @@ languages. For example::
620623
...
621624

622625
This doesn't allow you to jump into the middle of a loop, but that's usually
623-
considered an abuse of goto anyway. Use sparingly.
626+
considered an abuse of ``goto`` anyway. Use sparingly.
624627

625628

626629
Why can't raw strings (r-strings) end with a backslash?
@@ -652,7 +655,7 @@ If you're trying to build a pathname for a DOS command, try e.g. one of ::
652655
Why doesn't Python have a "with" statement for attribute assignments?
653656
---------------------------------------------------------------------
654657

655-
Python has a 'with' statement that wraps the execution of a block, calling code
658+
Python has a :keyword:`with` statement that wraps the execution of a block, calling code
656659
on the entrance and exit from the block. Some languages have a construct that
657660
looks like this::
658661

@@ -679,13 +682,13 @@ For instance, take the following incomplete snippet::
679682
with a:
680683
print(x)
681684

682-
The snippet assumes that "a" must have a member attribute called "x". However,
685+
The snippet assumes that ``a`` must have a member attribute called ``x``. However,
683686
there is nothing in Python that tells the interpreter this. What should happen
684-
if "a" is, let us say, an integer? If there is a global variable named "x",
685-
will it be used inside the with block? As you see, the dynamic nature of Python
687+
if ``a`` is, let us say, an integer? If there is a global variable named ``x``,
688+
will it be used inside the :keyword:`with` block? As you see, the dynamic nature of Python
686689
makes such choices much harder.
687690

688-
The primary benefit of "with" and similar language features (reduction of code
691+
The primary benefit of :keyword:`with` and similar language features (reduction of code
689692
volume) can, however, easily be achieved in Python by assignment. Instead of::
690693

691694
function(args).mydict[index][index].a = 21
@@ -714,7 +717,8 @@ Why don't generators support the with statement?
714717
For technical reasons, a generator used directly as a context manager
715718
would not work correctly. When, as is most common, a generator is used as
716719
an iterator run to completion, no closing is needed. When it is, wrap
717-
it as "contextlib.closing(generator)" in the 'with' statement.
720+
it as :func:`contextlib.closing(generator) <contextlib.closing>`
721+
in the :keyword:`with` statement.
718722

719723

720724
Why are colons required for the if/while/def/class statements?

‎Doc/library/urllib.parse.rst

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,8 @@ or on combining URL components into a URL string.
113113
+------------------+-------+-------------------------+------------------------+
114114
| :attr:`path` | 2 | Hierarchical path | empty string |
115115
+------------------+-------+-------------------------+------------------------+
116-
| :attr:`params` | 3 | No longer used | always an empty string |
116+
| :attr:`params` | 3 | Parameters for last | empty string |
117+
| | | path element | |
117118
+------------------+-------+-------------------------+------------------------+
118119
| :attr:`query` | 4 | Query component | empty string |
119120
+------------------+-------+-------------------------+------------------------+

‎Lib/asyncio/selector_events.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ def __init__(self, selector=None):
5858

5959
def _make_socket_transport(self, sock, protocol, waiter=None, *,
6060
extra=None, server=None):
61+
self._ensure_fd_no_transport(sock)
6162
return _SelectorSocketTransport(self, sock, protocol, waiter,
6263
extra, server)
6364

@@ -68,6 +69,7 @@ def _make_ssl_transport(
6869
ssl_handshake_timeout=constants.SSL_HANDSHAKE_TIMEOUT,
6970
ssl_shutdown_timeout=constants.SSL_SHUTDOWN_TIMEOUT,
7071
):
72+
self._ensure_fd_no_transport(rawsock)
7173
ssl_protocol = sslproto.SSLProtocol(
7274
self, protocol, sslcontext, waiter,
7375
server_side, server_hostname,
@@ -80,6 +82,7 @@ def _make_ssl_transport(
8082

8183
def _make_datagram_transport(self, sock, protocol,
8284
address=None, waiter=None, extra=None):
85+
self._ensure_fd_no_transport(sock)
8386
return _SelectorDatagramTransport(self, sock, protocol,
8487
address, waiter, extra)
8588

‎Lib/re/__init__.py

F438 Lines changed: 46 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,7 @@ def compile(pattern, flags=0):
229229
def purge():
230230
"Clear the regular expression caches"
231231
_cache.clear()
232+
_cache2.clear()
232233
_compile_repl.cache_clear()
233234

234235
def template(pattern, flags=0):
@@ -266,40 +267,64 @@ def escape(pattern):
266267
# --------------------------------------------------------------------
267268
# internals
268269

269-
_cache = {} # ordered!
270-
270+
# Use the fact that dict keeps the insertion order.
271+
# _cache2 uses the simple FIFO policy which has better latency.
272+
# _cache uses the LRU policy which has better hit rate.
273+
_cache = {} # LRU
274+
_cache2 = {} # FIFO
271275
_MAXCACHE = 512
276+
_MAXCACHE2 = 256
277+
assert _MAXCACHE2 < _MAXCACHE
278+
272279
def _compile(pattern, flags):
273280
# internal: compile pattern
274281
if isinstance(flags, RegexFlag):
275282
flags = flags.value
276283
try:
277-
return _cache[type(pattern), pattern, flags]
284+
return _cache2[type(pattern), pattern, flags]
278285
except KeyError:
279286
pass
280-
if isinstance(pattern, Pattern):
281-
if flags:
282-
raise ValueError(
283-
"cannot process flags argument with a compiled pattern")
284-
return pattern
285-
if not _compiler.isstring(pattern):
286-
raise TypeError("first argument must be string or compiled pattern")
287-
if flags & T:
288-
import warnings
289-
warnings.warn("The re.TEMPLATE/re.T flag is deprecated "
290-
"as it is an undocumented flag "
291-
"without an obvious purpose. "
292-
"Don't use it.",
293-
DeprecationWarning)
294-
p = _compiler.compile(pattern, flags)
295-
if not (flags & DEBUG):
287+
288+
key = (type(pattern), pattern, flags)
289+
# Item in _cache should be moved to the end if found.
290+
p = _cache.pop(key, None)
291+
if p is None:
292+
if isinstance(pattern, Pattern):
293+
if flags:
294+
raise ValueError(
295+
"cannot process flags argument with a compiled pattern")
296+
return pattern
297+
if not _compiler.isstring(pattern):
298+
raise TypeError("first argument must be string or compiled pattern")
299+
if flags & T:
300+
import warnings
301+
warnings.warn("The re.TEMPLATE/re.T flag is deprecated "
302+
"as it is an undocumented flag "
303+
"without an obvious purpose. "
304+
"Don't use it.",
305+
DeprecationWarning)
306+
p = _compiler.compile(pattern, flags)
307+
if flags & DEBUG:
308+
return p
296309
if len(_cache) >= _MAXCACHE:
297-
# Drop the oldest item
310+
# Drop the least recently used item.
311+
# next(iter(_cache)) is known to have linear amortized time,
312+
# but it is used here to avoid a dependency from using OrderedDict.
313+
# For the small _MAXCACHE value it doesn't make much of a difference.
298314
try:
299315
del _cache[next(iter(_cache))]
300316
except (StopIteration, RuntimeError, KeyError):
301317
pass
302-
_cache[type(pattern), pattern, flags] = p
318+
# Append to the end.
319+
_cache[key] = p
320+
321+
if len(_cache2) >= _MAXCACHE2:
322+
# Drop the oldest item.
323+
try:
324+
del _cache2[next(iter(_cache2))]
325+
except (StopIteration, RuntimeError, KeyError):
326+
pass
327+
_cache2[key] = p
303328
return p
304329

305330
@functools.lru_cache(_MAXCACHE)

‎Lib/test/test_asyncio/test_selector_events.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,10 @@ def setUp(self):
6161
def test_make_socket_transport(self):
6262
m = mock.Mock()
6363
self.loop.add_reader = mock.Mock()
64+
self.loop._ensure_fd_no_transport = mock.Mock()
6465
transport = self.loop._make_socket_transport(m, asyncio.Protocol())
6566
self.assertIsInstance(transport, _SelectorSocketTransport)
67+
self.assertEqual(self.loop._ensure_fd_no_transport.call_count, 1)
6668

6769
# Calling repr() must not fail when the event loop is closed
6870
self.loop.close()
@@ -78,8 +80,10 @@ def test_make_ssl_transport_without_ssl_error(self):
7880
self.loop.add_writer = mock.Mock()
7981
self.loop.remove_reader = mock.Mock()
8082
self.loop.remove_writer = mock.Mock()
83+
self.loop._ensure_fd_no_transport = mock.Mock()
8184
with self.assertRaises(RuntimeError):
8285
self.loop._make_ssl_transport(m, m, m, m)
86+
self.assertEqual(self.loop._ensure_fd_no_transport.call_count, 1)
8387

8488
def test_close(self):
8589
class EventLoop(BaseSelectorEventLoop):

‎Lib/test/test_clinic.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ def test_right_only(self):
153153
def test_have_left_options_but_required_is_empty(self):
154154
def fn():
155155
clinic.permute_optional_groups(['a'], [], [])
156-
self.assertRaises(AssertionError, fn)
156+
self.assertRaises(ValueError, fn)
157157

158158

159159
class ClinicLinearFormatTest(TestCase):

‎Misc/NEWS.d/next/Core and Builtins/2022-10-06-23-13-34.gh-issue-97997.JQaJKF.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Add running column offset to the tokenizer state to avoid calculating AST column information with pointer arithmetic.

‎Misc/NEWS.d/next/Library/2022-08-27-23-16-09.gh-issue-96346.jJX14I.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Use double caching for compiled RE patterns.

‎Misc/NEWS.d/next/Library/2022-08-29-12-35-28.gh-issue-96073.WaGstf.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
In :mod:`inspect`, fix overeager replacement of "`typing.`" in formatting annotations.
1+
In :mod:`inspect`, fix overeager replacement of "``typing.``" in formatting annotations.

‎Modules/clinic/posixmodule.c.h

Lines changed: 3 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Modules/posixmodule.c

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4539,12 +4539,13 @@ If dir_fd is not None, it should be a file descriptor open to a directory,
45394539
dir_fd may not be implemented on your platform.
45404540
If it is unavailable, using it will raise a NotImplementedError.
45414541
4542-
The mode argument is ignored on Windows.
4542+
The mode argument is ignored on Windows. Where it is used, the current umask
4543+
value is first masked out.
45434544
[clinic start generated code]*/
45444545

45454546
static PyObject *
45464547
os_mkdir_impl(PyObject *module, path_t *path, int mode, int dir_fd)
4547-
/*[clinic end generated code: output=a70446903abe821f input=e965f68377e9b1ce]*/
4548+
/*[clinic end generated code: output=a70446903abe821f input=a61722e1576fab03]*/
45484549
{
45494550
int result;
45504551
#ifdef HAVE_MKDIRAT

0 commit comments

Comments
 (0)
0