8000 feat: data and pyi files in the venv (#2936) · dougthor42/rules_python@013acd9 · GitHub
[go: up one dir, main page]
Skip to content

Commit 013acd9

Browse files
aignasaignas
authored
feat: data and pyi files in the venv (bazel-contrib#2936)
This adds the remaining of the files into the venv and should get us reasonably close to handling 99% of the cases. The expected differences from this and a `venv` built by `uv` would be: * The `RECORD` files are excluded from the `venv`s for better cache hit rate in `bazel`. Topological ordering is removed because topo ordering doesn't provide the "closer target first" guarantees desired. For now, just use default ordering and document conflicts as undefined behavior. Internally, it continues to use first-wins (i.e. first in depset.to_list() order) semantics. Work towards bazel-contrib#2156
1 parent d98547e commit 013acd9

File tree

27 files changed

+296
-105
lines changed

27 files changed

+296
-105
lines changed

‎.bazelrc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@
44
# (Note, we cannot use `common --deleted_packages` because the bazel version command doesn't support it)
55
# To update these lines, execute
66
# `bazel run @rules_bazel_integration_test//tools:update_deleted_packages`
7-
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_points,examples/bzlmod/entry_points/tests,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/patches,examples/bzlmod/py_proto_library,examples/bzlmod/py_proto_library/example.com/another_proto,examples/bzlmod/py_proto_library/example.com/proto,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,examples/py_proto_library/example.com/another_proto,examples/py_proto_library/example.com/proto,gazelle,gazelle/manifest,gazelle/manifest/generate,gazelle/manifest/hasher,gazelle/manifest/test,gazelle/modules_mapping,gazelle/python,gazelle/pythonconfig,gazelle/python/private,tests/integration/compile_pip_requirements,tests/integration/compile_pip_requirements_test_from_external_repo,tests/integration/custom_commands,tests/integration/ignore_root_user_error,tests/integration/ignore_root_user_error/submodule,tests/integration/local_toolchains,tests/integration/pip_parse,tests/integration/pip_parse/empty,tests/integration/py_cc_toolchain_registered,tests/modules/other,tests/modules/other/nspkg_delta,tests/modules/other/nspkg_gamma,tests/modules/other/nspkg_single
8-
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_points,examples/bzlmod/entry_points/tests,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/patches,examples/bzlmod/py_proto_library,examples/bzlmod/py_proto_library/example.com/another_proto,examples/bzlmod/py_proto_library/example.com/proto,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,examples/py_proto_library/example.com/another_proto,examples/py_proto_library/example.com/proto,gazelle,gazelle/manifest,gazelle/manifest/generate,gazelle/manifest/hasher,gazelle/manifest/test,gazelle/modules_mapping,gazelle/python,gazelle/pythonconfig,gazelle/python/private,tests/integration/compile_pip_requirements,tests/integration/compile_pip_requirements_test_from_external_repo,tests/integration/custom_commands,tests/integration/ignore_root_user_error,tests/integration/ignore_root_user_error/submodule,tests/integration/local_toolchains,tests/integration/pip_parse,tests/integration/pip_parse/empty,tests/integration/py_cc_toolchain_registered,tests/modules/other,tests/modules/other/nspkg_delta,tests/modules/other/nspkg_gamma,tests/modules/other/nspkg_single
7+
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_points,examples/bzlmod/entry_points/tests,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/patches,examples/bzlmod/py_proto_library,examples/bzlmod/py_proto_library/example.com/another_proto,examples/bzlmod/py_proto_library/example.com/proto,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,examples/py_proto_library/example.com/another_proto,examples/py_proto_library/example.com/proto,gazelle,gazelle/manifest,gazelle/manifest/generate,gazelle/manifest/hasher,gazelle/manifest/test,gazelle/modules_mapping,gazelle/python,gazelle/pythonconfig,gazelle/python/private,tests/integration/compile_pip_requirements,tests/integration/compile_pip_requirements_test_from_external_repo,tests/integration/custom_commands,tests/integration/ignore_root_user_error,tests/integration/ignore_root_user_error/submodule,tests/integration/local_toolchains,tests/integration/pip_parse,tests/integration/pip_parse/empty,tests/integration/py_cc_toolchain_registered,tests/modules/another_module,tests/modules/other,tests/modules/other/nspkg_delta,tests/modules/other/nspkg_gamma,tests/modules/other/nspkg_single,tests/modules/other/simple_v1,tests/modules/other/simple_v2,tests/modules/other/with_external_data
8+
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_points,examples/bzlmod/entry_points/tests,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/patches,examples/bzlmod/py_proto_library,examples/bzlmod/py_proto_library/example.com/another_proto,examples/bzlmod/py_proto_library/example.com/proto,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,examples/py_proto_library/example.com/another_proto,examples/py_proto_library/example.com/proto,gazelle,gazelle/manifest,gazelle/manifest/generate,gazelle/manifest/hasher,gazelle/manifest/test,gazelle/modules_mapping,gazelle/python,gazelle/pythonconfig,gazelle/python/private,tests/integration/compile_pip_requirements,tests/integration/compile_pip_requirements_test_from_external_repo,tests/integration/custom_commands,tests/integration/ignore_root_user_error,tests/integration/ignore_root_user_error/submodule,tests/integration/local_toolchains,tests/integration/pip_parse,tests/integration/pip_parse/empty,tests/integration/py_cc_toolchain_registered,tests/modules/another_module,tests/modules/other,tests/modules/other/nspkg_delta,tests/modules/other/nspkg_gamma,tests/modules/other/nspkg_single,tests/modules/other/simple_v1,tests/modules/other/simple_v2,tests/modules/other/with_external_data
99

1010
test --test_output=errors
1111

‎MODULE.bazel

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ bazel_dep(name = "rules_multirun", version = "0.9.0", dev_dependency = True)
8686
bazel_dep(name = "bazel_ci_rules", version = "1.0.0", dev_dependency = True)
8787
bazel_dep(name = "rules_pkg", version = "1.0.1", dev_dependency = True)
8888
bazel_dep(name = "other", version = "0", dev_dependency = True)
89+
bazel_dep(name = "another_module", version = "0", dev_dependency = True)
8990

9091
# Extra gazelle plugin deps so that WORKSPACE.bzlmod can continue including it for e2e tests.
9192
# We use `WORKSPACE.bzlmod` because it is impossible to have dev-only local overrides.
@@ -116,6 +117,11 @@ local_path_override(
116117
path = "tests/modules/other",
117118
)
118119

120+
local_path_override(
121+
module_name = "another_module",
122+
path = "tests/modules/another_module",
123+
)
124+
119125
dev_python = use_extension(
120126
"//python/extensions:python.bzl",
121127
"python",

‎internal_dev_deps.bzl

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,11 @@ def rules_python_internal_deps():
4848
path = "tests/modules/other",
4949
)
5050

51+
local_repository(
52+
name = "another_module",
53+
path = "tests/modules/another_module",
54+
)
55+
5156
http_archive(
5257
name = "bazel_skylib",
5358
sha256 = "bc283cdfcd526a52c3201279cda4bc298652efa898b10b4db0837dc51652756f",

‎python/private/BUILD.bazel

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -450,11 +450,13 @@ bzl_library(
450450
":attributes_bzl",
451451
":common_bzl",
452452
":flags_bzl",
453+
":normalize_name_bzl",
453454
":precompile_bzl",
454455
":py_cc_link_params_info_bzl",
455456
":py_internal_bzl",
456457
":rule_builders_bzl",
457458
":toolchain_types_bzl",
459+
":version_bzl",
458460
"@bazel_skylib//lib:dicts",
459461
"@bazel_skylib//rules:common_settings",
460462
],

‎python/private/attributes.bzl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -260,7 +260,7 @@ The order of this list can matter because it affects the order that information
260260
from dependencies is merged in, which can be relevant depending on the ordering
261261
mode of depsets that are merged.
262262
263-
* {obj}`PyInfo.venv_symlinks` uses topological ordering.
263+
* {obj}`PyInfo.venv_symlinks` uses default ordering.
264264
265265
See {obj}`PyInfo` for more information about the ordering of its depsets and
266266
how its fields are merged.

‎python/private/common.bzl

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -331,7 +331,7 @@ def collect_runfiles(ctx, files = depset()):
331331
# If the target is a File, then add that file to the runfiles.
332332
# Otherwise, add the target's **data runfiles** to the runfiles.
333333
#
334-
# Note that, contray to best practice, the default outputs of the
334+
# Note that, contrary to best practice, the default outputs of the
335335
# targets in `data` are *not* added, nor are the default runfiles.
336336
#
337337
# This ends up being important for several reasons, some of which are
@@ -396,9 +396,8 @@ def create_py_info(
396396
implicit_pyc_files: {type}`depset[File]` Implicitly generated pyc files
397397
that a binary can choose to include.
398398
imports: depset of strings; the import path values to propagate.
399-
venv_symlinks: {type}`list[tuple[str, str]]` tuples of
400-
`(runfiles_path, site_packages_path)` for symlinks to create
401-
in the consuming binary's venv site packages.
399+
venv_symlinks: {type}`list[VenvSymlinkEntry]` instances for
400+
symlinks to create in the consuming binary's venv.
402401
403402
Returns:
404403
A tuple of the PyInfo instance and a depset of the

‎python/private/py_executable.bzl

Lines changed: 35 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -650,10 +650,6 @@ def _create_venv_symlinks(ctx, venv_dir_map):
650650

651651
# maps venv-relative path to the runfiles path it should point to
652652
entries = depset(
653-
# NOTE: Topological ordering is used so that dependencies closer to the
654-
# binary have precedence in creating their symlinks. This allows the
655-
# binary a modicum of control over the result.
656-
order = "topological",
657653
transitive = [
658654
dep[PyInfo].venv_symlinks
659655
for dep in ctx.attr.deps
@@ -680,43 +676,52 @@ def _create_venv_symlinks(ctx, venv_dir_map):
680676
return venv_files
681677

682678
def _build_link_map(entries):
683-
# dict[str kind, dict[str rel_path, str link_to_path]]
684-
link_map = {}
679+
# dict[str package, dict[str kind, dict[str rel_path, str link_to_path]]]
680+
pkg_link_map = {}
681+
682+
# dict[str package, str version]
683+
version_by_pkg = {}
684+
685685
for entry in entries:
686-
kind = entry.kind
687-
kind_map = link_map.setdefault(kind, {})
688-
if entry.venv_path in kind_map:
689-
# We ignore duplicates by design. The dependency closer to the
690-
# binary gets precedence due to the topological ordering.
686+
link_map = pkg_link_map.setdefault(entry.package, {})
687+
kind_map = link_map.setdefault(entry.kind, {})
688+
689+
if version_by_pkg.setdefault(entry.package, entry.version) != entry.version:
690+
# We ignore duplicates by design.
691+
continue
692+
elif entry.venv_path in kind_map:
693+
# We ignore duplicates by design.
691694
continue
692695
else:
693696
kind_map[entry.venv_path] = entry.link_to_path
694697

< 7802 /code>
695-
# An empty link_to value means to not create the site package symlink.
696-
# Because of the topological ordering, this allows binaries to remove
697-
# entries by having an earlier dependency produce empty link_to values.
698-
for kind, kind_map in link_map.items():
699-
for dir_path, link_to in kind_map.items():
700-
if not link_to:
701-
kind_map.pop(dir_path)
698+
# An empty link_to value means to not create the site package symlink. Because of the
699+
# ordering, this allows binaries to remove entries by having an earlier dependency produce
700+
# empty link_to values.
701+
for link_map in pkg_link_map.values():
702+
for kind, kind_map in link_map.items():
703+
for dir_path, link_to in kind_map.items():
704+
if not link_to:
705+
kind_map.pop(dir_path)
702706

703707
# dict[str kind, dict[str rel_path, str link_to_path]]
704708
keep_link_map = {}
705709

706710
# Remove entries that would be a child path of a created symlink.
707711
# Earlier entries have precedence to match how exact matches are handled.
708-
for kind, kind_map in link_map.items():
709-
keep_kind_map = keep_link_map.setdefault(kind, {})
710-
for _ in range(len(kind_map)):
711-
if not kind_map:
712-
break
713-
dirname, value = kind_map.popitem()
714-
keep_kind_map[dirname] = value
715-
prefix = dirname + "/" # Add slash to prevent /X matching /XY
716-
for maybe_suffix in kind_map.keys():
717-
maybe_suffix += "/" # Add slash to prevent /X matching /XY
718-
if maybe_suffix.startswith(prefix) or prefix.startswith(maybe_suffix):
719-
kind_map.pop(maybe_suffix)
712+
for link_map in pkg_link_map.values():
713+
for kind, kind_map in link_map.items():
714+
keep_kind_map = keep_link_map.setdefault(kind, {})
715+
for _ in range(len(kind_map)):
716+
if not kind_map:
717+
break
718+
dirname, value = kind_map.popitem()
719+
keep_kind_map[dirname] = value
720+
prefix = dirname + "/" # Add slash to prevent /X matching /XY
721+
for maybe_suffix in kind_map.keys():
722+
maybe_suffix += "/" # Add slash to prevent /X matching /XY
723+
if maybe_suffix.startswith(prefix) or prefix.startswith(maybe_suffix):
724+
kind_map.pop(maybe_suffix)
720725
return keep_link_map
721726

722727
def _map_each_identity(v):

‎python/private/py_info.bzl

Lines changed: 14 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -67,11 +67,24 @@ the venv to create the path under.
6767
6868
A runfiles-root relative path that `venv_path` will symlink to. If `None`,
6969
it means to not create a symlink.
70+
""",
71+
"package": """
72+
:type: str | None
73+
74+
Represents the PyPI package name that the code originates from. It is normalized according to the
75+
PEP440 with all `-` replaced with `_`, i.e. the same as the package name in the hub repository that
76+
it would come from.
7077
""",
7178
"venv_path": """
7279
:type: str
7380
7481
A path relative to the `kind` directory within the venv.
82+
""",
83+
"version": """
84+
:type: str | None
85+
86+
Represents the PyPI package version that the code originates from. It is normalized according to the
87+
PEP440 standard.
7588
""",
7689
},
7790
)
@@ -296,29 +309,9 @@ This field is currently unused in Bazel and may go away in the future.
296309
"venv_symlinks": """
297310
:type: depset[VenvSymlinkEntry]
298311
299-
A depset with `topological` ordering.
300-
301-
302-
Tuples of `(runfiles_path, site_packages_path)`. Where
303-
* `runfiles_path` is a runfiles-root relative path. It is the path that
304-
has the code to make importable. If `None` or empty string, then it means
305-
to not create a site packages directory with the `site_packages_path`
306-
name.
307-
* `site_packages_path` is a path relative to the site-packages directory of
308-
the venv for whatever creates the venv (typically py_binary). It makes
309-
the code in `runfiles_path` available for import. Note that this
310-
is created as a "raw" symlink (via `declare_symlink`).
311-
312312
:::{include} /_includes/experimental_api.md
313313
:::
314314
315-
:::{tip}
316-
The topological ordering means dependencies earlier and closer to the consumer
317-
have precedence. This allows e.g. a binary to add dependencies that override
318-
values from further way dependencies, such as forcing symlinks to point to
319-
specific paths or preventing symlinks from being created.
320-
:::
321-
322315
:::{versionadded} VERSION_NEXT_FEATURE
323316
:::
324317
""",
@@ -375,9 +368,6 @@ def _PyInfoBuilder_typedef():
375368
376369
:::{field} venv_symlinks
377370
:type: DepsetBuilder[tuple[str | None, str]]
378-
379-
NOTE: This depset has `topological` order
380-
:::
381371
"""
382372

383373
def _PyInfoBuilder_new():
@@ -417,7 +407,7 @@ def _PyInfoBuilder_new():
417407
transitive_pyc_files = builders.DepsetBuilder(),
418408
transitive_pyi_files = builders.DepsetBuilder(),
419409
transitive_sources = builders.DepsetBuilder(),
420-
venv_symlinks = builders.DepsetBuilder(order = "topological"),
410+
venv_symlinks = builders.DepsetBuilder(),
421411
)
422412
return self
423413

0 commit comments

Comments
 (0)
0