10000 bpo-2506: Add -X noopt command line option · python/cpython@445778f · GitHub
[go: up one dir, main page]
Skip to content

Commit 445778f

Browse files
pablogsal1st1vstinner
committed
bpo-2506: Add -X noopt command line option
Add -X noopt command line option to disable compiler optimizations. distutils ignores noopt: always enable compiler optimizations. * Add sys.flags.noopt flag. * Add 'noopt' keyword-only parameter to builtin compile(), importlib.util.cache_from_source() and py_compile.compile(). * importlib: SourceLoader gets an additional keyword-only '_noopt' parameter. It is used by py_compile.compile(). * importlib uses ``.noopt.pyc`` suffix for .pyc files if sys.flags.noopt is true. * Add PyConfig.optimize parameter. * Update subprocess._optim_args_from_interpreter_flags(). * Add support.requires_compiler_optimizations() * Update unit tests which rely on compiler optimizations for noopt: add @support.requires_compiler_optimizations. * Rename c_optimize to c_optimization_level in compiler.c Co-Authored-By: Yury Selivanov <yury@magic.io> Co-Authored-By: Victor Stinner <vstinner@python.org>
1 parent c3a651a commit 445778f

40 files changed

+3087
-2772
lines changed

‎Doc/c-api/init_config.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -580,6 +580,14 @@ PyConfig
580580
by the function calculating the :ref:`Path Configuration
581581
<init-path-config>`.
582582
583+
.. c:member:: int optimize
584+
585+
Should the compiler optimize bytecode? If equal to 0, disable the
586+
compiler optimizations and set :c:member:`~PyConfig.optimization_level`
587+
to 0. Set to 0 by :option:`-X noopt <-X>` command line option.
588+
589+
.. versionadded:: 3.10
590+
583591
.. c:member:: int optimization_level
584592
585593
Compilation optimization level:

‎Doc/library/functions.rst

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -242,7 +242,7 @@ are always available. They are listed here in alphabetical order.
242242
Class methods can now wrap other :term:`descriptors <descriptor>` such as
243243
:func:`property`.
244244

245-
.. function:: compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1)
245+
.. function:: compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1, noopt=None)
246246

247247
Compile the *source* into a code or AST object. Code objects can be executed
248248
by :func:`exec` or :func:`eval`. *source* can either be a normal string, a
@@ -286,6 +286,9 @@ are always available. They are listed here in alphabetical order.
286286
``__debug__`` is true), ``1`` (asserts are removed, ``__debug__`` is false)
287287
or ``2`` (docstrings are removed too).
288288

289+
If *noopt* is false, disable compiler optimizations and ignore *optimize*
290+
argument. If it is ``None``, use ``sys.flags.noopt`` value.
291+
289292
This function raises :exc:`SyntaxError` if the compiled source is invalid,
290293
and :exc:`ValueError` if the source contains null bytes.
291294

@@ -319,10 +322,13 @@ are always available. They are listed here in alphabetical order.
319322
Previously, :exc:`TypeError` was raised when null bytes were encountered
320323
in *source*.
321324

322-
.. versionadded:: 3.8
325+
.. versionchanged:: 3.8
323326
``ast.PyCF_ALLOW_TOP_LEVEL_AWAIT`` can now be passed in flags to enable
324327
support for top-level ``await``, ``async for``, and ``async with``.
325328

329+
.. versionachanged:: 3.10
330+
New *noopt* optional keyword-only parameter.
331+
326332

327333
.. class:: complex([real[, imag]])
328334

‎Doc/library/importlib.rst

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1410,7 +1410,7 @@ an :term:`importer`.
14101410

14111411
.. versionadded:: 3.4
14121412

1413-
.. function:: cache_from_source(path, debug_override=None, *, optimization=None)
1413+
.. function:: cache_from_source(path, debug_override=None, *, optimization=None, noopt=None)
14141414

14151415
Return the :pep:`3147`/:pep:`488` path to the byte-compiled file associated
14161416
with the source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return
@@ -1429,6 +1429,11 @@ an :term:`importer`.
14291429
``/foo/bar/__pycache__/baz.cpython-32.opt-2.pyc``. The string representation
14301430
of *optimization* can only be alphanumeric, else :exc:`ValueError` is raised.
14311431

1432+
The *noopt* parameter is used to specify if compiler optimization are
1433+
disabled. If it is true, *optimization* is ignored and ``.noopt`` suffix is
1434+
used (ex: ``baz.cpython-32.noopt.pyc``). If it is ``None``, use
1435+
``sys.flags.noopt`` value.
1436+
14321437
The *debug_override* parameter is deprecated and can be used to override
14331438
the system's value for ``__debug__``. A ``True`` value is the equivalent of
14341439
setting *optimization* to the empty string. A ``False`` value is the same as
@@ -1444,6 +1449,9 @@ an :term:`importer`.
14441449
.. versionchanged:: 3.6
14451450
Accepts a :term:`path-like object`.
14461451

1452+
.. versionchanged:: 3.10
1453+
Added *noopt* parameter.
1454+
14471455

14481456
.. function:: source_from_cache(path)
14491457

‎Doc/library/py_compile.rst

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,9 @@ byte-code cache files in the directory containing the source code.
6060
:func:`compile` function. The default of ``-1`` selects the optimization
6161
level of the current interpreter.
6262

63+
If *noopt* is true, disable compiler optimizations and ignore *optimize*
64+
argument. If it is ``None``, use ``sys.flags.noopt`` value.
65+
6366
*invalidation_mode* should be a member of the :class:`PycInvalidationMode`
6467
enum and controls how the generated bytecode cache is invalidated at
6568
runtime. The default is :attr:`PycInvalidationMode.CHECKED_HASH` if
@@ -92,6 +95,9 @@ byte-code cache files in the directory containing the source code.
9295
.. versionchanged:: 3.8
9396
The *quiet* parameter was added.
9497

98+
.. versionchanged:: 3.10
99+
The *noopt* parameter was added.
100+
95101

96102
.. class:: PycInvalidationMode
97103

‎Doc/library/sys.rst

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -439,6 +439,7 @@ always available.
439439
:const:`inspect` :option:`-i`
440440
:const:`interactive` :option:`-i`
441441
:const:`isolated` :option:`-I`
442+
:const:`noopt` :option:`-X noopt <-X>`
442443
:const:`optimize` :option:`-O` or :option:`-OO`
443444
:const:`dont_write_bytecode` :option:`-B`
444445
:const:`no_user_site` :option:`-s`
@@ -469,6 +470,9 @@ always available.
469470
Mode <devmode>` and the ``utf8_mode`` attribute for the new :option:`-X`
470471
``utf8`` flag.
471472

473+
.. versionchanged:: 3.10
474+
Added ``noopt`` attribute for the new :option:`-X noopt <-X>` option.
475+
472476

473477
.. data:: float_info
474478

‎Doc/using/cmdline.rst

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,7 @@ Miscellaneous options
454454
* ``-X pycache_prefix=PATH`` enables writing ``.pyc`` files to a parallel
455455
tree rooted at the given directory instead of to the code tree. See also
456456
:envvar:`PYTHONPYCACHEPREFIX`.
457+
* ``-X noopt`` disables the compiler optimizations.
457458

458459
It also allows passing arbitrary values and retrieving them through the
459460
:data:`sys._xoptions` dictionary.
@@ -477,9 +478,9 @@ Miscellaneous options
477478
The ``-X pycache_prefix`` option. The ``-X dev`` option now logs
478479
``close()`` exceptions in :class:`io.IOBase` destructor.
479480

480-
.. versionchanged:: 3.9
481-
Using ``-X dev`` option, check *encoding* and *errors* arguments on
482-
string encoding and decoding operations.
481+
.. versionadded:: 3.10
482+
The ``-X noopt`` option. Using ``-X dev`` option, check *encoding* and
483+
*errors* arguments on string encoding and decoding operations.
483484

484485
The ``-X showalloccount`` option has been removed.
485486

‎Doc/whatsnew/3.10.rst

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,11 @@ Other Language Changes
9393
:meth:`~object.__index__` method).
9494
(Contributed by Serhiy Storchaka in :issue:`37999`.)
9595

96+
* Added :option:`-X noopt <-X>` command line to disable compiler optimizations.
97+
:mod:`importlib` uses ``.noopt.pyc`` suffix for ``.pyc`` filenames if
98+
:data:`sys.flags.noopt` is true.
99+
(Contributed by Yury Selivanov, Victor Stinner and Pablo Galindo in :issue:`2506`)
100+
96101

97102
New Modules
98103
===========
@@ -109,6 +114,12 @@ base64
109114
Add :func:`base64.b32hexencode` and :func:`base64.b32hexdecode` to support the
110115
Base32 Encoding with Extended Hex Alphabet.
111116

117+
builtins
118+
--------
119+
120+
The :func:`compile` function gets a new optional keyword-only *noopt* parameter
121+
to disable compiler optimizations.
122+
112123
curses
113124
------
114125

@@ -126,6 +137,20 @@ Added the *root_dir* and *dir_fd* parameters in :func:`~glob.glob` and
126137
:func:`~glob.iglob` which allow to specify the root directory for searching.
127138
(Contributed by Serhiy Storchaka in :issue:`38144`.)
128139

140+
importlib
141+
---------
142+
143+
Add a new *noopt* optional keyword-only parameter to
144+
:func:`importlib.util.cache_from_source` to disable compiler optimizations.
145+
(Contributed by Yury Selivanov, Victor Stinner and Pablo Galindo in :issue:`2506`)
146+
147+
py_compile
148+
----------
149+
150+
Add a new *noopt* optional keyword-only parameter to :func:`py_compile.compile`
151+
to disable compiler optimizations.
152+
(Contributed by Yury Selivanov, Victor Stinner and Pablo Galindo in :issue:`2506`)
153+
129154
os
130155
--
131156

@@ -145,6 +170,11 @@ Add :data:`sys.orig_argv` attribute: the list of the original command line
145170
arguments passed to the Python executable.
146171
(Contributed by Victor Stinner in :issue:`23427`.)
147172

173+
174+
Add new :data:`sys.flags.noopt` flag for the new :option:`-X noopt <-X>` option
175+
(disable compiler optimizations).
176+
(Contributed by Yury Selivanov, Victor Stinner and Pablo Galindo in :issue:`2506`)
177+
148178
xml
149179
---
150180

‎Include/compile.h

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,4 +106,7 @@ PyAPI_FUNC(int) _PyAST_Optimize(struct _mod *, PyArena *arena, _PyASTOptimizeSta
106106
/* This doesn't need to match anything */
107107
#define Py_fstring_input 800
108108

109+
/* Value of 'optimize' to deactivate all optimizations */
110+
#define _PyCompiler_disable_all_optimizations -2
111+
109112
#endif /* !Py_COMPILE_H */

‎Include/cpython/initconfig.h

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -417,6 +417,17 @@ typedef struct {
417417
418418
_PyConfig_Write() initializes Py_GetArgcArgv() to this list. */
419419
PyWideStringList orig_argv;
420+
421+
/* Should the compiler optimize bytecode?
422+
423+
If equal to 0, disable compiler optimizations and set optimization_level
424+
to 0.
425+
426+
If equal to 1, enable compiler optimizations.
427+
428+
Set to 0 by -X noopt. */
429+
int optimize;
430+
420431
} PyConfig;
421432

422433
PyAPI_FUNC(void) PyConfig_InitPythonConfig(PyConfig *config);

‎Include/pythonrun.h

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,11 @@ PyAPI_FUNC(PyObject *) Py_CompileStringObject(
6262
PyObject *filename, int start,
6363
PyCompilerFlags *flags,
6464
int optimize);
65+
PyAPI_FUNC(PyObject *) _Py_CompileString(
66+
const char *str,
67+
PyObject *filename, int start,
68+
PyCompilerFlags *flags,
69+
int optimize);
6570
#endif
6671
PyAPI_FUNC(struct symtable *) Py_SymtableString(
6772
const char *str,

‎Lib/distutils/command/build_py.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -314,10 +314,10 @@ def get_outputs(self, include_bytecode=1):
314314
if include_bytecode:
315315
if self.compile:
316316
outputs.append(importlib.util.cache_from_source(
317-
filename, optimization=''))
317+
filename, optimization='', noopt=False))
318318
if self.optimize > 0:
319319
outputs.append(importlib.util.cache_from_source(
320-
filename, optimization=self.optimize))
320+
filename, optimization=self.optimize, noopt=False))
321321

322322
outputs += [
323323
os.path.join(build_dir, filename)

‎Lib/distutils/command/install_lib.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -166,10 +166,10 @@ def _bytecode_filenames(self, py_filenames):
166166
continue
167167
if self.compile:
168168
bytecode_files.append(importlib.util.cache_from_source(
169-
py_file, optimization=''))
169+
py_file, optimization='', noopt=False))
170170
if self.optimize > 0:
171171
bytecode_files.append(importlib.util.cache_from_source(
172-
py_file, optimization=self.optimize))
172+
py_file, optimization=self.optimize, noopt=False))
173173

174174
return bytecode_files
175175

‎Lib/distutils/tests/test_install_lib.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,9 +44,11 @@ def test_byte_compile(self):
4444
f = os.path.join(project_dir, 'foo.py')
4545
self.write_file(f, '# python file')
4646
cmd.byte_compile([f])
47-
pyc_file = importlib.util.cache_from_source('foo.py', optimization='')
47+
pyc_file = importlib.util.cache_from_source('foo.py', optimization='',
48+
noopt=False)
4849
pyc_opt_file = importlib.util.cache_from_source('foo.py',
49-
optimization=cmd.optimize)
50+
optimization=cmd.optimize,
51+
noopt=False)
5052
self.assertTrue(os.path.exists(pyc_file))
5153
self.assertTrue(os.path.exists(pyc_opt_file))
5254

‎Lib/distutils/util.py

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -444,10 +444,11 @@ def byte_compile (py_files,
444444
# dfile - purported source filename (same as 'file' by default)
445445
if optimize >= 0:
446446
opt = '' if optimize == 0 else optimize
447-
cfile = importlib.util.cache_from_source(
448-
file, optimization=opt)
447+
cfile = importlib.util.cache_from_source(file,
448+
optimization=opt,
449+
noopt=False)
449450
else:
450-
cfile = importlib.util.cache_from_source(file)
451+
cfile = importlib.util.cache_from_source(file, noopt=False)
451452
dfile = file
452453
if prefix:
453454
if file[:len(prefix)] != prefix:
@@ -462,7 +463,7 @@ def byte_compile (py_files,
462463
if force or newer(file, cfile):
463464
log.info("byte-compiling %s to %s", file, cfile_base)
464465
if not dry_run:
465-
compile(file, cfile, dfile)
466+
compile(file, cfile, dfile, noopt=False)
466467
else:
467468
log.debug("skipping byte-compilation of %s to %s",
468469
file, cfile_base)

‎Lib/importlib/_bootstrap_external.py

Lines changed: 28 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -290,6 +290,7 @@ def _write_atomic(path, data, mode=0o666):
290290
_RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c
291291

292292
_PYCACHE = '__pycache__'
293+
_NOOPT = 'noopt'
293294
_OPT = 'opt-'
294295

295296
SOURCE_SUFFIXES = ['.py'] # _setup() adds .pyw as needed.
@@ -298,7 +299,8 @@ def _write_atomic(path, data, mode=0o666):
298299
# Deprecated.
299300
DEBUG_BYTECODE_SUFFIXES = OPTIMIZED_BYTECODE_SUFFIXES = BYTECODE_SUFFIXES
300301

301-
def cache_from_source(path, debug_override=None, *, optimization=None):
302+
def cache_from_source(path, debug_override=None, *, optimization=None,
303+
noopt=None):
302304
"""Given the path to a .py file, return the path to its .pyc file.
303305
304306
The .py file does not need to exist; this simply returns the path to the
@@ -330,16 +332,21 @@ def cache_from_source(path, debug_override=None, *, optimization=None):
330332
if tag is None:
331333
raise NotImplementedError('sys.implementation.cache_tag is None')
332334
almost_filename = ''.join([(base if base else rest), sep, tag])
333-
if optimization is None:
334-
if sys.flags.optimize == 0:
335-
optimization = ''
336-
else:
337-
optimization = sys.flags.optimize
338-
optimization = str(optimization)
339-
if optimization != '':
340-
if not optimization.isalnum():
341-
raise ValueError('{!r} is not alphanumeric'.format(optimization))
342-
almost_filename = '{}.{}{}'.format(almost_filename, _OPT, optimization)
335+
if noopt is None:
336+
noopt = sys.flags.noopt
337+
if noopt:
338+
almost_filename = f'{almost_filename}.{_NOOPT}'
339+
else:
340+
if optimization is None:
341+
if sys.flags.optimize == 0:
342+
optimization = ''
343+
else:
344+
optimization = sys.flags.optimize
345+
optimization = str(optimization)
346+
if optimization != '':
347+
if not optimization.isalnum():
348+
raise ValueError('{!r} is not alphanumeric'.format(optimization))
349+
almost_filename = f'{almost_filename}.{_OPT}{optimization}'
343350
filename = almost_filename + BYTECODE_SUFFIXES[0]
344351
if sys.pycache_prefix is not None:
345352
# We need an absolute path to the py file to avoid the possibility of
@@ -398,13 +405,14 @@ def source_from_cache(path):
398405
raise ValueError(f'expected only 2 or 3 dots in {pycache_filename!r}')
399406
elif dot_count == 3:
400407
optimization = pycache_filename.rsplit('.', 2)[-2]
401-
if not optimization.startswith(_OPT):
408+
if optimization.startswith(_OPT):
409+
opt_level = optimization[len(_OPT):]
410+
if not opt_level.isalnum():
411+
raise ValueError(f"optimization level {optimization!r} "
412+
"is not an alphanumeric value")
413+
elif not optimization.startswith(_NOOPT):
402414
raise ValueError("optimization portion of filename does not start "
403-
f"with {_OPT!r}")
404-
opt_level = optimization[len(_OPT):]
405-
if not opt_level.isalnum():
406-
raise ValueError(f"optimization level {optimization!r} is not an "
407-
"alphanumeric value")
415+
f"with {_OPT!r} nor {_NOOPT!r}")
408416
base_filename = pycache_filename.partition('.')[0]
409417
return _path_join(head, base_filename + SOURCE_SUFFIXES[0])
410418

@@ -845,13 +853,14 @@ def get_source(self, fullname):
845853
name=fullname) from exc
846854
return decode_source(source_bytes)
847855

848-
def source_to_code(self, data, path, *, _optimize=-1):
856+
def source_to_code(self, data, path, *, _optimize=-1, _noopt=None):
849857
"""Return the code object compiled from source.
850858
851859
The 'data' argument can be any object type that compile() supports.
852860
"""
853861
return _bootstrap._call_with_frames_removed(compile, data, path, 'exec',
854-
dont_inherit=True, optimize=_optimize)
862+
dont_inherit=True, optimize=_optimize,
863+
noopt=_noopt)
855864

856865
def get_code(self, fullname):
857866
"""Concrete implementation of InspectLoader.get_code.

0 commit comments

Comments
 (0)
0