Description
Currently, rules_python does not have bzlmod support to use multiple Python versions. The following example demonstrates the current "WORKSPACE" support: https://github.com/bazelbuild/rules_python/tree/main/examples/multi_python_versions
Design
Dependencies
In order to have support we need extensions that create the following data relationships:
First, we have 2 or more versions of Python; say Python 3.9, 3.10, and 3.11.
Second, each of those Python versions has a host-specific interpreter. You can see the interpreter extension used in the following code:
This extension creates a symlink to the host-specific interpreter so that pip can utilize the correct interpreter.
So we have
- multiple Python versions
- multiple Python host (OS and platform) specific Interpreter binaries
- symlinks to each of those binaries
The above is a one to one to one relationship.
Lastly, we have requirements files. Currently ONLY one requirement lock file per Python version is supported. This does not match the functionality of using a single version of Python since we can define an OS-specific requirements lock file. See:
How "WORKSPACE" support works
Starting with the WORKSPACE file:
First, we call
Which adds the bazel_skylib dependency.
Next, we call:
Which adds various dependencies in the following data structure
So those two components are callable via an extension. Pretty trivial.
Next we call
python_ register_mutiple_toolchains() does:
For each Python version, it calls the macro
https://github.com/bazelbuild/rules_python/blob/16126d0ebfee074a3a8fe216b20cc19e1b3603c1/python/repositories.bzl#L595
And it names the call with a specific name.
Does three primary things. Again this is done for each version of Python that is passed into
python_ register_mutiple_toolchains.
First, it calls
Which creates components like py_runtime and py_pairs for each platform (Windows, OSX, Linux)
Next, it uses
Lastly, we have:
To create the py_binary and py_test targets for each Python version.
Next, the rule calls the same macro for the "default" Python version:
Lastly, it calls:
The above call is critical to how the pip repos are created.
Is a repo rule that creates pip.bzl that contains:
A load in the WORKSPACE file then uses this, which we cannot do in bzlmod.
Before we get into that, let's recap.
For each version of Python, multiple repositories are created that contain:
- each Python version has multiple toolchains defined for each platform
py_runtimeandpy_testfor each version
With bzlmod, we cannot have a load statement in the MODULES.bazel file. We must create a repo to map information passed to the next pip call. The important components are:
The host interpreter is set. We will need a repo containing a mapping of each host interpreter for each Python version.
The rule multi_pip_parse is dynamically created. This calls a private repository rule. Here is an example of the pip.bzl file:
# Generated by python/private/toolchains_repo.bzl
load("@rules_python//python:pip.bzl", "pip_parse", _multi_pip_parse = "multi_pip_parse")
def multi_pip_parse(name, requirements_lock, **kwargs):
return _multi_pip_parse(
name = name,
python_versions = ["3.8", "3.9", "3.10", "3.11"],
requirements_lock = requirements_lock,
**kwargs
)Because we cannot do:
I do not believe that we can call the repository rule directly. The implementation between pip and bzlmod pip differs.
We will also need to map the interpreters, which is easily doable.