8000 Add pip3_import (#256) · sfraint/rules_python@9467740 · GitHub
[go: up one dir, main page]
Skip to content

Commit 9467740

Browse files
brandjonbrandjon
authored
Add pip3_import (bazel-contrib#256)
This adds `pip3_import` as a wrapper around `pip_import` that sets `python_interpreter` to `"python3"`. This is important for requesting the Python 3 version of a package on systems where the `"python"` command is Python 2 (which is most of them). We decline to add an analogous `pip2_import` wrapper at this time, because the command `"python2"` does not exist on all platforms by default (e.g. macOS). `piptool.py` is updated to prefix the names of the wheel repos (an implementation detail of rules_python) with the name given to `pip_import`. This is needed to avoid shadowing wheel repos when the same wheel name is used by separate `pip_import` invocations -- in particular when the same wheel is used for both PY2 and PY3. (Thanks to @joshclimacell for pointing this detail out in his prototype 90a70d5.) Regenerated the .par files and docs. Also updated the README to better explain the structure of the packaging rules. This includes mentioning `pip3_import`, concerns around versioning / hermeticity, and not depending on the wheel repo names (use `requirement()` instead).
1 parent 7b222cf commit 9467740

File tree

8 files changed

+124
-47
lines changed

8 files changed

+124
-47
lines changed

‎README.md

Lines changed: 69 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,17 @@
55

66
## Recent updates
77

8-
* 2019-07-26: The canonical name of this repo has been changed from `@io_bazel_rules_python` to just `@rules_python`, in accordance with [convention](https://docs.bazel.build/versions/master/skylark/deploying.html#workspace). Please update your WORKSPACE file and labels that reference this repo accordingly.
8+
* 2019-11-15: Added support for `pip3_import` (and more generally, a
9+
`python_interpreter` attribute to `pip_import`). The canonical naming for wheel
10+
repositories has changed to accomodate loading wheels for both `pip_import` and
11+
`pip3_import` in the same build. To avoid breakage, please use `requirement()`
12+
instead of depending directly on wheel repo labels.
13+
14+
* 2019-07-26: The canonical name of this repo has been changed from
15+
`@io_bazel_rules_python` to just `@rules_python`, in accordance with
16+
[convention](https://docs.bazel.build/versions/master/skylark/deploying.html#workspace).
17+
Please update your `WORKSPACE` file and labels that reference this repo
18+
accordingly.
919

1020
## Overview
1121

@@ -95,7 +105,7 @@ git_repository(
95105
# above.
96106
```
97107

98-
Once you've imported the rule set into your WORKSPACE using any of these
108+
Once you've imported the rule set into your `WORKSPACE` using any of these
99109
methods, you can then load the core rules in your `BUILD` files with:
100110

101111
``` python
@@ -109,35 +119,67 @@ py_binary(
109119

110120
## Using the packaging rules
111121

122+
The packaging rules create two kinds of repositories: A central repo that holds
123+
downloaded wheel files, and individual repos for each wheel's extracted
124+
contents. Users only need to interact with the central repo; the wheel repos
125+
are essentially an implementation detail. The central repo provides a
126+
`WORKSPACE` macro to create the wheel repos, as well as a function to call in
127+
`BUILD` files to translate a pip package name into the label of a `py_library`
128+
target in the appropriate wheel repo.
129+
112130
### Importing `pip` dependencies
113131

114-
The packaging rules are designed to have developers continue using
115-
`requirements.txt` to express their dependencies in a Python idiomatic manner.
116-
These dependencies are imported into the Bazel dependency graph via a
117-
two-phased process in `WORKSPACE`:
132+
Adding pip dependencies to your `WORKSPACE` is a two-step process. First you
133+
declare the central repository using `pip_import`, which invokes pip to read
134+
a `requirements.txt` file and download the appropriate wheels. Then you load
135+
the `pip_install` function from the central repo, and call it to create the
136+
individual wheel repos.
137+
138+
**Important:** If you are using Python 3, load and call `pip3_import` instead.
118139

119140
```python
120141
load("@rules_python//python:pip.bzl", "pip_import")
121142

122-
# This rule translates the specified requirements.txt into
123-
# @my_deps//:requirements.bzl, which itself exposes a pip_install method.
124-
pip_import(
143+
# Create a central repo that knows about the dependencies needed for
144+
# requirements.txt.
145+
pip_import( # or pip3_import
125146
name = "my_deps",
126147
requirements = "//path/to:requirements.txt",
127148
)
128149

129-
# Load the pip_install symbol for my_deps, and create the dependencies'
130-
# repositories.
150+
# Load the central repo's install function from its `//:requirements.bzl` file,
151+
# and call it.
131152
load("@my_deps//:requirements.bzl", "pip_install")
132153
pip_install()
133154
```
134155

156+
Note that since pip is executed at WORKSPACE-evaluation time, Bazel has no
157+
information about the Python toolchain and cannot enforce that the interpreter
158+
used to invoke pip matches the interpreter used to run `py_binary` targets. By
159+
default, `pip_import` uses the system command `"python"`, which on most
160+
platforms is a Python 2 interpreter. This can be overridden by passing the
161+
`python_interpreter` attribute to `pip_import`. `pip3_import` just acts as a
162+
wrapper that sets `python_interpreter` to `"python3"`.
163+
164+
You can have multiple `pip_import`s in the same workspace, e.g. for Python 2
165+
and Python 3. This will create multiple central repos that have no relation to
166+
one another, and may result in downloading the same wheels multiple times.
167+
168+
As with any repository rule, if you would like to ensure that `pip_import` is
169+
reexecuted in order to pick up a non-hermetic change to your environment (e.g.,
170+
updating your system `python` interpreter), you can completely flush out your
171+
repo cache with `bazel clean --expunge`.
172+
135173
### Consuming `pip` dependencies
136174

137-
Once a set of dependencies has been imported via `pip_import` and `pip_install`
138-
we can start consuming them in our `py_{binary,library,test}` rules. In support
139-
of this, the generated `requirements.bzl` also contains a `requirement` method,
140-
which can be used directly in `deps=[]` to reference an imported `py_library`.
175+
Each extracted wheel repo contains a `py_library` target representing the
176+
wheel's contents. Rather than depend on this target's label directly -- which
177+
would require hardcoding the wheel repo's mangled name into your BUILD files --
178+
you should instead use the `requirement()` function defined in the central
179+
repo's `//:requirements.bzl` file. This function maps a pip package name to a
180+
label. (["Extras"](
181+
https://packaging.python.org/tutorials/installing-packages/#installing-setuptools-extras)
182+
can be referenced using the `pkg[extra]` syntax.)
141183

142184
```python
143185
load("@my_deps//:requirements.bzl", "requirement")
@@ -147,46 +189,36 @@ py_library(
147189
srcs = ["mylib.py"],
148190
deps = [
149191
":myotherlib",
150-
# This takes the name as specified in requirements.txt
151-
requirement("importeddep"),
192+
requirement("some_pip_dep"),
193+
requirement("anohter_pip_dep[some_extra]"),
152194
]
153195
)
154196
```
155197

156-
### Canonical `whl_library` naming
157-
158-
It is notable that `whl_library` rules imported via `pip_import` are canonically
159-
named, following the pattern: `pypi__{distribution}_{version}`. Characters in
160-
these components that are illegal in Bazel label names (e.g. `-`, `.`) are
161-
replaced with `_`.
162-
163-
This canonical naming helps avoid redundant work to import the same library
164-
multiple times. It is expected that this naming will remain stable, so folks
165-
should be able to reliably depend directly on e.g. `@pypi__futures_3_1_1//:pkg`
166-
for dependencies, however, it is recommended that folks stick with the
167-
`requirement` pattern in case the need arises for us to make changes to this
168-
format in the future.
169-
170-
["Extras"](
171-
https://packaging.python.org/tutorials/installing-packages/#installing-setuptools-extras)
172-
will have a target of the extra name (in place of `pkg` above).
198+
For reference, the wheel repos are canonically named following the pattern:
199+
`@{central_repo_name}_pypi__{distribution}_{version}`. Characters in the
200+
distribution and version that are illegal in Bazel label names (e.g. `-`, `.`)
201+
are replaced with `_`. While this naming pattern doesn't change often, it is
202+
not guaranted to remain stable, so use of the `requirement()` function is
203+
recommended.
173204

174205
## Migrating from the bundled rules
175206

176207
The core rules are currently available in Bazel as built-in symbols, but this
177208
form is deprecated. Instead, you should depend on rules_python in your
178-
WORKSPACE file and load the Python rules from `@rules_python//python:defs.bzl`.
209+
`WORKSPACE` file and load the Python rules from
210+
`@rules_python//python:defs.bzl`.
179211

180212
A [buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md)
181-
fix is available to automatically migrate BUILD and .bzl files to add the
213+
fix is available to automatically migrate `BUILD` and `.bzl` files to add the
182214
appropriate `load()` statements and rewrite uses of `native.py_*`.
183215

184216
```sh
185217
# Also consider using the -r flag to modify an entire workspace.
186218
buildifier --lint=fix --warnings=native-py <files>
187219
```
188220

189-
Currently the WORKSPACE file needs to be updated manually as per [Getting
221+
Currently the `WORKSPACE` file needs to be updated manually as per [Getting
190222
started](#Getting-started) above.
191223

192224
Note that Starlark-defined bundled symbols underneath

‎docs/pip.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,36 @@ wheels.
9090
</table>
9191

9292

93+
<a name="#pip3_import"></a>
94+
95+
## pip3_import
96+
97+
<pre>
98+
pip3_import(<a href="#pip3_import-kwargs">kwargs</a>)
99+
</pre>
100+
101+
A wrapper around pip_import that uses the `python3` system command.
102+
103+
Use this for requirements of PY3 programs.
104+
105+
### Parameters
106+
107+
<table class="params-table">
108+
<colgroup>
109+
<col class="col-param" />
110+
<col class="col-description" />
111+
</colgroup>
112+
<tbody>
113+
<tr id="pip3_import-kwargs">
114+
<td><code>kwargs</code></td>
115+
<td>
116+
optional.
117+
</td>
118+
</tr>
119+
</tbody>
120+
</table>
121+
122+
93123
<a name="#pip_repositories"></a>
94124

95125
## pip_repositories

‎packaging/piptool.py

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,9 @@ def list_whls():
174174
whls = [Wheel(path) for path in list_whls()]
175175
possible_extras = determine_possible_extras(whls)
176176

177+
def repository_name(wheel):
178+
return args.name + "_" + wheel.repository_suffix()
179+
177180
def whl_library(wheel):
178181
# Indentation here matters. whl_library must be within the scope
179182
# of the function below. We also avoid reimporting an existing WHL.
1CF5
@@ -185,7 +188,7 @@ def whl_library(wheel):
185188
whl = "@{name}//:{path}",
186189
requirements = "@{name}//:requirements.bzl",
187190
extras = [{extras}]
188-
)""".format(name=args.name, repo_name=wheel.repository_name(),
191+
)""".format(name=args.name, repo_name=repository_name(wheel),
189192
python_interpreter=args.python_interpreter,
190193
path=wheel.basename(),
191194
extras=','.join([
@@ -195,11 +198,11 @@ def whl_library(wheel):
195198

196199
whl_targets = ','.join([
197200
','.join([
198-
'"%s": "@%s//:pkg"' % (whl.distribution().lower(), whl.repository_name())
201+
'"%s": "@%s//:pkg"' % (whl.distribution().lower(), repository_name(whl))
199202
] + [
200203
# For every extra that is possible from this requirements.txt
201204
'"%s[%s]": "@%s//:%s"' % (whl.distribution().lower(), extra.lower(),
202-
whl.repository_name(), extra)
205+
repository_name(whl), extra)
203206
for extra in possible_extras.get(whl, [])
204207
])
205208
for whl in whls

‎packaging/whl.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,9 @@ def version(self):
4242
parts = self.basename().split('-')
4343
return parts[1]
4444

45-
def repository_name(self):
46-
# Returns the canonical name of the Bazel repository for this package.
45+
def repository_suffix(self):
46+
# Returns a canonical suffix that will form part of the name of the Bazel
47+
# repository for this package.
4748
canonical = 'pypi__{}_{}'.format(self.distribution(), self.version())
4849
# Escape any illegal characters with underscore.
4950
return re.sub('[-.+]', '_', canonical)

‎packaging/whl_test.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ def test_grpc_whl(self):
3434
self.assertEqual(wheel.version(), '1.6.0')
3535
self.assertEqual(set(wheel.dependencies()),
3636
set(['enum34', 'futures', 'protobuf', 'six']))
37-
self.assertEqual('pypi__grpcio_1_6_0', wheel.repository_name())
37+
self.assertEqual('pypi__grpcio_1_6_0', wheel.repository_suffix())
3838
self.assertEqual([], wheel.extras())
3939

4040
def test_futures_whl(self):
@@ -44,7 +44,7 @@ def test_futures_whl(self):
4444
self.assertEqual(wheel.distribution(), 'futures')
4545
self.assertEqual(wheel.version(), '3.1.1')
4646
self.assertEqual(set(wheel.dependencies()), set())
47-
self.assertEqual('pypi__futures_3_1_1', wheel.repository_name())
47+
self.assertEqual('pypi__futures_3_1_1', wheel.repository_suffix())
4848
self.assertEqual([], wheel.extras())
4949

5050
def test_whl_with_METADATA_file(self):
@@ -54,7 +54,7 @@ def test_whl_with_METADATA_file(self):
5454
self.assertEqual(wheel.distribution(), 'futures')
5555
self.assertEqual(wheel.version(), '2.2.0')
5656
self.assertEqual(set(wheel.dependencies()), set())
57-
self.assertEqual('pypi__futures_2_2_0', wheel.repository_name())
57+
self.assertEqual('pypi__futures_2_2_0', wheel.repository_suffix())
5858

5959
@patch('platform.python_version', return_value='2.7.13')
6060
def test_mock_whl(self, *args):
@@ -65,7 +65,7 @@ def test_mock_whl(self, *args):
6565
self.assertEqual(wheel.version(), '2.0.0')
6666
self.assertEqual(set(wheel.dependencies()),
6767
set(['funcsigs', 'pbr', 'six']))
68-
self.assertEqual('pypi__mock_2_0_0', wheel.repository_name())
68+
self.assertEqual('pypi__mock_2_0_0', wheel.repository_suffix())
6969

7070
@patch('platform.python_version', return_value='3.3.0')
7171
def test_mock_whl_3_3(self, *args):
@@ -103,7 +103,7 @@ def test_google_cloud_language_whl(self, *args):
103103
self.assertEqual(set(wheel.dependencies()),
104104
set(expected_deps))
105105
self.assertEqual('pypi__google_cloud_language_0_29_0',
106-
wheel.repository_name())
106+
wheel.repository_suffix())
107107
self.assertEqual([], wheel.extras())
108108

109109
@patch('platform.python_version', return_value='3.4.0')

‎python/pip.bzl

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,17 @@ py_binary(
102102
""",
103103
)
104104

105+
# We don't provide a `pip2_import` that would use the `python2` system command
106+
# because this command does not exist on all platforms. On most (but not all)
107+
# systems, `python` means Python 2 anyway. See also #258.
108+
109+
def pip3_import(**kwargs):
110+
"""A wrapper around pip_import that uses the `python3` system command.
111+
112+
Use this for requirements of PY3 programs.
113+
"""
114+
pip_import(python_interpreter = "python3", **kwargs)
115+
105116
def pip_repositories():
106117
"""Pull in dependencies needed to use the packaging rules."""
107118

‎tools/piptool.par

11.1 KB
Binary file not shown.

‎tools/whltool.par

2.38 KB
Binary file not shown.

0 commit comments

Comments
 (0)
0