8000 gh-108740: Fix "make regen-all" race condition by vstinner · Pull Request #108741 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-108740: Fix "make regen-all" race condition #108741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Sep 6, 2023
Merged

gh-108740: Fix "make regen-all" race condition #108741

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3afb66f
gh-108740: Fix "make regen-all" race condition
vstinner Aug 31, 2023
6a909dd
Update Doc/using/configure.rst
vstinner Sep 2, 2023
af96ee6
Style
erlend-aasland Sep 6, 2023
a760e70
Use DEEPFREEZE_C in target rule as well
erlend-aasland Sep 6, 2023
418d72e
Revert "Use DEEPFREEZE_C in target rule as well"
erlend-aasland Sep 6, 2023
66e8d70
Use $(DEEPFREEZE_C)
vstinner Sep 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
gh-108740: Fix "make regen-all" race condition 
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script
  anymore.
* "make regen-deepfreeze" now only updates deepfreeze.c, it doesn't
  build deepfreeze.o anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are outdated, these commands are now safe to
  use.
* Document generates files in Doc/using/configure.rst
  • Loading branch information
@vstinner
vstinner committed Sep 6, 2023
commit 3afb66f665f813bbd59202f917a4f997576b1d7a
23 changes: 23 additions & 0 deletions Doc/using/configure.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,29 @@ See also :pep:`7` "Style Guide for C Code" and :pep:`11` "CPython platform
support".


Generated files
===============

To reduce build dependencies, Python source code contains multiple generated
files. Commands to regenerate all generated files::

make regen-all
make regen-stdlib-module-names
make regen-limited-abi
make regen-configure

The ``Makefile.pre.in`` documents generated files, their inputs, and tools used
to regenerate them. Search for ``regen-*`` make targets.

The ``make regen-configure`` command runs `tiran/cpython_autoconf
<https://github.com/tiran/cpython_autoconf>`_ container for reproducible build;
see container ``entry.sh`` script. The container is optional, the following
command can be run locally, the generated files depend on autoconf and aclocal
versions::

autoreconf -ivf -Werror


.. _configure-options:

Configure Options
Expand Down
17 changes: 9 additions & 8 deletions Makefile.pre.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -490,6 +490,7 @@ OBJECT_OBJS= \
Objects/weakrefobject.o \
@PERF_TRAMPOLINE_OBJ@

DEEPFREEZE_C = Python/deepfreeze/deepfreeze.c
DEEPFREEZE_OBJS = Python/deepfreeze/deepfreeze.o

##########################################################################
Expand Down Expand Up @@ -777,7 +778,6 @@ coverage-report: regen-token regen-frozen
.PHONY: clinic
clinic: check-clean-src $(srcdir)/Modules/_blake2/blake2s_impl.c
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/clinic/clinic.py --make --exclude Lib/test/clinic.test.c --srcdir $(srcdir)
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/build/generate_global_objects.py

.PHONY: clinic-tests
clinic-tests: check-clean-src $(srcdir)/Lib/test/clinic.test.c
Expand Down Expand Up @@ -1252,7 +1252,7 @@ regen-frozen: Tools/build/freeze_modules.py $(FROZEN_FILES_IN)
# Deepfreeze targets

.PHONY: regen-deepfreeze
regen-deepfreeze: $(DEEPFREEZE_OBJS)
regen-deepfreeze: $(DEEPFREEZE_C)

DEEPFREEZE_DEPS=$(srcdir)/Tools/build/deepfreeze.py Include/internal/pycore_global_strings.h $(FREEZE_MODULE_DEPS) $(FROZEN_FILES_OUT)

Expand Down Expand Up @@ -1284,8 +1284,6 @@ Python/deepfreeze/deepfreeze.c: $(DEEPFREEZE_DEPS)
Python/frozen_modules/frozen_only.h:frozen_only \
-o Python/deepfreeze/deepfreeze.c
# END: deepfreeze modules
@echo "Note: Deepfreeze may have added some global objects,"
@echo " so run 'make regen-global-objects' if necessary."

# We keep this renamed target around for folks with muscle memory.
.PHONY: regen-importlib
Expand All @@ -1295,10 +1293,11 @@ regen-importlib: regen-frozen
# Global objects

.PHONY: regen-global-objects
regen-global-objects: $(srcdir)/Tools/build/generate_global_objects.py
# Dependencies which can add and/or remove _Py_ID() identifiers:
# - deepfreeze.c
# - "make clinic"
regen-global-objects: $(srcdir)/Tools/build/generate_global_objects.py $(DEEPFREEZE_C) clinic
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/build/generate_global_objects.py
@echo "Note: Global objects can be added or removed by other tools (e.g. deepfreeze), "
@echo " so be sure to re-run regen-global-objects after those tools."

############################################################################
# ABI
Expand All @@ -1321,8 +1320,9 @@ regen-limited-abi: all
# Regenerate all generated files

.PHONY: regen-all
# "regen-global-objects" runs "clinic".
regen-all: regen-cases regen-typeslots \
regen-token regen-ast regen-keyword regen-sre regen-frozen clinic \
regen-token regen-ast regen-keyword regen-sre regen-frozen \
regen-pegen-metaparser regen-pegen regen-test-frozenmain \
regen-test-levenshtein regen-global-objects
@echo
Expand Down Expand Up @@ -2597,6 +2597,7 @@ recheck:
autoconf:
(cd $(srcdir); autoreconf -ivf -Werror)

# See https://github.com/tiran/cpython_autoconf container
.PHONY: regen-configure
regen-configure:
@if command -v podman >/dev/null; then RUNTIME="podman"; else RUNTIME="docker"; fi; \
Expand Down
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/Build/2023-09-01-01-39-26.gh-issue-108740.JHExAQ.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Fix a race condition in ``make regen-all``. The ``deepfreeze.c`` source and
files generated by Argument Clinic are now generated or updated before
generating "global objects". Previously, some identifiers may miss depending
on the order in which these files were generated. Patch by Victor Stinner.
0