10BC0 feat: Expose Python C headers through the toolchain. · bazel-contrib/rules_python@37e2d5a · GitHub
[go: up one dir, main page]
Skip to content

Commit 37e2d5a

Browse files
rickeylevrickeylev
committed
feat: Expose Python C headers through the toolchain.
This allows getting a build's `cc_library` of Python headers through toolchain resolution instead of having to use the underlying toolchain's repository `:python_headers` target directly. Without this feature, it's not possible to reliably and correctly get the C information about the runtime a build is going to use. Existing solutions require carefully setting up repo names, external state, and/or using specific build rules. In comparison, with this feature, consumers are able to simply ask for the current headers via a special target or manually lookup the toolchain and pull the relevant information; toolchain resolution handles finding the correct headers. The basic way this works is by registering a second toolchain to carry C/C++ related information; as such, it is named `py_cc_toolchain`. The py cc toolchain has the same constraint settings as the regular py toolchain; an expected invariant is that there is a 1:1 correspondence between the two. This base functionality allows a consuming rule implementation to use toolchain resolution to find the Python C toolchain information. Usually what downstream consumers need are the headers to feed into another `cc_library` (or equivalent) target, so, rather than have every project reimplement the same "lookup and forward cc_library info" logic, this is provided by the `//python/cc:current_py_cc_headers` target. Targets that need the headers can then depend on that target as if it was a `cc_library` target. Work towards #824
1 parent 5b8fa22 commit 37e2d5a

24 files changed

+922
-6
lines changed

‎.bazelci/presubmit.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,8 +113,25 @@ tasks:
113113
<<: *reusable_config
114114
name: Test on RBE using minimum supported Bazel version
115115
platform: rbe_ubuntu1604
116+
build_flags:
117+
# BazelCI sets --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1,
118+
# which prevents cc toolchain autodetection. To workaround this,
119+
# we manually specify the build kite cc toolchain.
120+
- "--extra_toolchains=@buildkite_config//config:cc-toolchain"
121+
build_targets:
122+
- "//tests/cc/current_py_cc_headers:test_current_toolchain_headers"
123+
- "..."
124+
- "@rules_python//examples/wheel/..."
116125
test_flags:
117126
- "--test_tag_filters=-integration-test,-acceptance-test"
127+
# BazelCI sets --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1,
128+
# which prevents cc toolchain autodetection. To workaround this,
129+
# we manually specify the build kite cc toolchain.
130+
- "--extra_toolchains=@buildkite_config//config:cc-toolchain"
131+
test_targets:
132+
- "//tests/cc/current_py_cc_headers:test_current_toolchain_headers"
133+
- "..."
134+
- "@rules_python//examples/wheel/..."
118135
rbe:
119136
<<: *reusable_config
120137
name: Test on RBE

‎.bazelrc‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
# This lets us glob() up all the files inside the examples to make them inputs to tests
44
# (Note, we cannot use `common --deleted_packages` because the bazel version command doesn't support it)
55
# To update these lines, run tools/bazel_integration_test/update_deleted_packages.sh
6-
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
7-
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
6+
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
7+
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
88

99
test --test_output=errors
1010

‎python/BUILD.bazel‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ licenses(["notice"])
3333
filegroup(
3434
name = "distribution",
3535
srcs = glob(["**"]) + [
36+
"//python/cc:distribution",
3637
"//python/config_settings:distribution",
3738
"//python/constraints:distribution",
3839
"//python/private:distribution",

‎python/cc/BUILD.bazel‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Package for C/C++ specific functionality of the Python rules.
2+
3+
load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
4+
load("//python/private:current_py_cc_headers.bzl", "current_py_cc_headers")
5+
load("//python/private:util.bzl", "BZLMOD_ENABLED")
6+
7+
package(
8+
default_visibility = ["//:__subpackages__"],
9+
)
10+
11+
# This target provides the C headers for whatever the current toolchain is
12+
# for the consuming rule. It basically acts like a cc_library by forwarding
13+
# on the providers for the underlying cc_library that the toolchain is using.
14+
current_py_cc_headers(
15+
name = "current_py_cc_headers",
16+
# Building this directly will fail unless a py cc toolchain is registered,
17+
# and it's only under bzlmod that one is registered by default.
18+
tags = [] if BZLMOD_ENABLED else ["manual"],
19+
visibility = ["//visibility:public"],
20+
)
21+
22+
toolchain_type(
23+
name = "toolchain_type",
24+
visibility = ["//visibility:public"],
25+
)
26+
27+
bzl_library(
28+
name = "py_cc_toolchain_bzl",
29+
srcs = ["py_cc_toolchain.bzl"],
30+
visibility = ["//visibility:public"],
31+
deps = ["//python/private:py_cc_toolchain_bzl"],
32+
)
33+
34+
bzl_library(
35+
name = "py_cc_toolchain_info_bzl",
36+
srcs = ["py_cc_toolchain_info.bzl"],
37+
visibility = ["//visibility:public"],
38+
deps = ["//python/private:py_cc_toolchain_info_bzl"],
39+
)
40+
41+
filegroup(
42+
name = "distribution",
43+
srcs = glob(["**"]),
44+
)

‎python/cc/py_cc_toolchain.bzl‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Public entry point for py_cc_toolchain rule."""
16+
17+
load("//python/private:py_cc_toolchain_macro.bzl", _py_cc_toolchain = "py_cc_toolchain")
18+
19+
py_cc_toolchain = _py_cc_toolchain

‎python/cc/py_cc_toolchain_info.bzl‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Public entry point for PyCcToolchainInfo."""
16+
17+
load("//python/private:py_cc_toolchain_info.bzl", _PyCcToolchainInfo = "PyCcToolchainInfo")
18+
19+
PyCcToolchainInfo = _PyCcToolchainInfo

‎python/private/BUILD.bazel‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,25 @@ bzl_library(
5151
deps = ["@bazel_skylib//lib:types"],
5252
)
5353

54+
bzl_library(
55+
name = "py_cc_toolchain_bzl",
56+
srcs = [
57+
"py_cc_toolchain_macro.bzl",
58+
"py_cc_toolchain_rule.bzl",
59+
],
60+
visibility = ["//python/cc:__pkg__"],
61+
deps = [
62+
":py_cc_toolchain_info_bzl",
63+
":util_bzl",
64+
],
65+
)
66+
67+
bzl_library(
68+
name = "py_cc_toolchain_info_bzl",
69+
srcs = ["py_cc_toolchain_info.bzl"],
70+
visibility = ["//python/cc:__pkg__"],
71+
)
72+
5473
# @bazel_tools can't define bzl_library itself, so we just put a wrapper around it.
5574
bzl_library(
5675
name = "bazel_tools_bzl",

‎python/private/current_py_cc_headers.bzl‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Implementation of current_py_cc_headers rule."""
16+
17+
def _current_py_cc_headers_impl(ctx):
18+
py_cc_toolchain = ctx.toolchains["//python/cc:toolchain_type"].py_cc_toolchain
19+
return py_cc_toolchain.headers.providers_map.values()
20+
21+
current_py_cc_headers = rule(
22+
implementation = _current_py_cc_headers_impl,
23+
toolchains = ["//python/cc:toolchain_type"],
24+
provides = [CcInfo],
25+
doc = """\
26+
Provides the currently active Python toolchain's C headers.
27+
28+
This is basically a wrapper around the underlying `cc_library()` for the
29+
C headers for the consuming target's currently active Python toolchain.
30+
31+
To use, simply depend on this target where you would have wanted the
32+
toolchain's underlying `:python_headers` target.
33+
""",
34+
)

‎python/private/py_cc_toolchain_info.bzl‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Implementation of PyCcToolchainInfo."""
16+
17+
PyCcToolchainInfo = provider(
18+
doc = "C/C++ information about the Python runtime.",
19+
fields = {
20+
"headers": """\
21+
(struct) Information about the header files, with fields:
22+
* providers_map: a dict of string to provider instances. The key should be
23+
a fully qualified name (e.g. `@rules_foo//bar:baz.bzl#MyInfo`) of the
24+
provider to uniquely identify its type.
25+
26+
The following keys are always present:
27+
* CcInfo: the CcInfo provider instance for the headers.
28+
* DefaultInfo: the DefaultInfo provider instance for the headers.
29+
30+
A map is used to allow additional providers from the originating headers
31+
target (typically a `cc_library`) to be propagated to consumers (directly
32+
exposing a Target object can cause memory issues and is an anti-pattern).
33+
34+
When consuming this map, it's suggested to use `providers_map.values()` to
35+
return all providers; or copy the map and filter out or replace keys as
36+
appropriate. Note that any keys begining with `_` (underscore) are
37+
considered private and should be forward along as-is (this better allows
38+
e.g. `:current_py_cc_headers` to act as the underlying headers target it
39+
represents).
40+
""",
41+
"python_version": "(str) The Python Major.Minor version.",
42+
},
43+
)

‎python/private/py_cc_toolchain_macro.bzl‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Fronting macro for the py_cc_toolchain rule."""
16+
17+
load(":py_cc_toolchain_rule.bzl", _py_cc_toolchain = "py_cc_toolchain")
18+
load(":util.bzl", "add_tag")
19+
20+
# A fronting macro is used because macros have user-observable behavior;
21+
# using one from the onset avoids introducing those changes in the future.
22+
def py_cc_toolchain(**kwargs):
23+
"""Creates a py_cc_toolchain target.
24+
25+
Args:
26+
**kwargs: Keyword args to pass onto underlying rule.
27+
"""
28+
29+
# This tag is added to easily identify usages through other macros.
30+
add_tag(kwargs, "@rules_python//python:py_cc_toolchain")
31+
_py_cc_toolchain(**kwargs)

0 commit comments

Comments
 (0)
0