Skip to content

Commit 1c58124

Browse files
chrislovecnmchrislovecnm
authored
feat(bzlmod)!: Calling pip multiple times allowing for multiple Python versions (bazel-contrib#1254)
Implementing the capability to call pip bzlmod extension multiple times. We can now call the pip extension with the same hub name multiple times. This allows a user to have multiple different requirement files based on the Python version. With workspace, we need incompatible aliases because you get @pip//tabulate or @pip_tabulate//:pkg. The requirement() macro hides this, but changing the flag becomes a breaking change if you've manually referenced things. With bzlmod, though, the @pip_tabulate style isn't a realistic option (you'd have to use_repo everything, which is a huge pain). So we have chosen to have `@pip//tabulate`. This commit implements that capability for pip.parse to determine the Python version from programmatically the provided interpreter. See https://github.com/bazelbuild/rules_python/blob/76aab0f91bd614ee1b2ade310baf28c38695c522/python/extensions/pip.bzl#L88 For more in-depth implementation details. INTERFACE CHANGE:: - pip.parse argument python_version is introduced but not required. When possible, it is inferred. BREAKING CHANGE: * `pip.parse()` extension: the `name` attribute is renamed `hub_name`. This is to reflect that the user-provided name isn't unique to each `pip.parse()` call. We now have a hub that is created, and each pip.parse call creates a spoke. * `pip.parse()` extension: the `incompatible_generate_aliases` arg is removed; behavior has changed to assume it is always True. * `pip.parse()` calls are collected under the same `hub_name` to support multiple Python versions under the same resulting repo name (the hub name0. Close: bazel-contrib#1229
1 parent 3903d1a commit 1c58124

24 files changed

+1269
-286
lines changed

‎.bazelrc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
# This lets us glob() up all the files inside the examples to make them inputs to tests
44
# (Note, we cannot use `common --deleted_packages` because the bazel version command doesn't support it)
55
# To update these lines, run tools/bazel_integration_test/update_deleted_packages.sh
6-
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_point,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
7-
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_point,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
6+
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
7+
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
88

99
test --test_output=errors
1010

‎BZLMOD_SUPPORT.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,8 @@ A second example, in [examples/bzlmod_build_file_generation](examples/bzlmod_bui
3131

3232
This rule set does not have full feature partity with the older `WORKSPACE` type configuration:
3333

34-
1. Multiple pip extensions are not yet supported, as demonstrated in [this](examples/multi_python_versions) example.
35-
2. Gazelle does not support finding deps in sub-modules. For instance we can have a dep like ` "@our_other_module//other_module/pkg:lib",` in a `py_test` definition.
34+
1. Gazelle does not support finding deps in sub-modules. For instance we can have a dep like ` "@our_other_module//other_module/pkg:lib",` in a `py_test` definition.
35+
2. We have some features that are still not fully flushed out, and the user interface may change.
3636

3737
Check ["issues"](/bazelbuild/rules_python/issues) for an up to date list.
3838

‎README.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -96,8 +96,7 @@ use_repo(interpreter, "interpreter")
9696

9797
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
9898
pip.parse(
99-
name = "pip",
100-
incompatible_generate_aliases = True,
99+
hub_name = "pip",
101100
python_interpreter_target = "@interpreter//:python",
102101
requirements_lock = "//:requirements_lock.txt",
103102
requirements_windows = "//:requirements_windows.txt",
@@ -201,7 +200,7 @@ central external repo and individual wheel external repos.
201200

202201
```python
203202
pip.parse(
204-
name = "my_deps",
203+
hub_name = "my_deps",
205204
requirements_lock = "//:requirements_lock.txt",
206205
)
207206

‎docs/pip_repository.md

Lines changed: 23 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎examples/bzlmod/BUILD.bazel

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,17 +8,28 @@
88
load("@bazel_skylib//rules:build_test.bzl", "build_test")
99
load("@pip//:requirements.bzl", "all_requirements", "all_whl_requirements", "requirement")
1010
load("@python_3_9//:defs.bzl", py_test_with_transition = "py_test")
11+
load("@python_aliases//3.10:defs.bzl", compile_pip_requirements_3_10 = "compile_pip_requirements")
12+
load("@python_aliases//3.9:defs.bzl", compile_pip_requirements_3_9 = "compile_pip_requirements")
1113
load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
12-
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
1314

1415
# This stanza calls a rule that generates targets for managing pip dependencies
1516
# with pip-compile.
16-
compile_pip_requirements(
17-
name = "requirements",
17+
compile_pip_requirements_3_9(
18+
name = "requirements_3_9",
1819
extra_args = ["--allow-unsafe"],
1920
requirements_in = "requirements.in",
20-
requirements_txt = "requirements_lock.txt",
21-
requirements_windows = "requirements_windows.txt",
21+
requirements_txt = "requirements_lock_3_9.txt",
22+
requirements_windows = "requirements_windows_3_9.txt",
23+
)
24+
25+
# This stanza calls a rule that generates targets for managing pip dependencies
26+
# with pip-compile.
27+
compile_pip_requirements_3_10(
28+
name = "requirements_3_10",
29+
extra_args = ["--allow-unsafe"],
30+
requirements_in = "requirements.in",
31+
requirements_txt = "requirements_lock_3_10.txt",
32+
requirements_windows = "requirements_windows_3_10.txt",
2233
)
2334

2435
# The rules below are language specific rules defined in

‎examples/bzlmod/MODULE.bazel

Lines changed: 23 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -11,16 +11,6 @@ local_path_override(
1111
path = "../..",
1212
)
1313

14-
# This name is generated by python.toolchain(), and is later passed
15-
# to use_repo() and interpreter.install().
16-
PYTHON_NAME_39 = "python_3_9"
17-
18-
INTERPRETER_NAME_39 = "interpreter_39"
19-
20-
PYTHON_NAME_310 = "python_3_10"
21-
22-
INTERPRETER_NAME_310 = "interpreter_310"
23-
2414
# We next initialize the python toolchain using the extension.
2515
# You can set different Python versions in this block.
2616
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
@@ -41,38 +31,34 @@ python.toolchain(
4131
python_version = "3.10",
4232
)
4333

44-
# You only need to load this repositories if you are using muiltple Python versions.
45-
# See the tests folder for various examples.
46-
use_repo(python, PYTHON_NAME_39, "python_aliases")
34+
# You only need to load this repositories if you are using multiple Python versions.
35+
# See the tests folder for various examples on using multiple Python versions.
36+
# The names "python_3_9" and "python_3_10" are autmatically created by the repo
37+
# rules based on the python_versions.
38+
use_repo(python, "python_3_10", "python_3_9", "python_aliases")
4739

48-
# The interpreter extension discovers the platform specific Python binary.
49-
# It creates a symlink to the binary, and we pass the label to the following
50-
# pip.parse call.
51-
interpreter = use_extension("@rules_python//python/extensions:interpreter.bzl", "interpreter")
52-
interpreter.install(
53-
name = INTERPRETER_NAME_39,
54-
python_name = PYTHON_NAME_39,
55-
)
40+
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
5641

57-
# This second call is only needed if you are using mulitple different
58-
# Python versions/interpreters.
59-
interpreter.install(
60-
name = INTERPRETER_NAME_310,
61-
python_name = PYTHON_NAME_310,
42+
# For each pip setup we call pip.parse. We can pass in various options
43+
# but typically we are passing in a requirements and an interpreter.
44+
# If you provide the python_version we will attempt to determine
45+
# the interpreter target automatically. Otherwise use python_interpreter_target
46+
# to override the lookup.
47+
pip.parse(
48+
hub_name = "pip",
49+
python_version = "3.9",
50+
requirements_lock = "//:requirements_lock_3_9.txt",
51+
requirements_windows = "//:requirements_windows_3_9.txt",
6252
)
63-
use_repo(interpreter, INTERPRETER_NAME_39, INTERPRETER_NAME_310)
64-
65-
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
6653
pip.parse(
67-
name = "pip",
68-
# Intentionally set it false because the "true" case is already
69-
# covered by examples/bzlmod_build_file_generation
70-
incompatible_generate_aliases = False,
71-
python_interpreter_target = "@{}//:python".format(INTERPRETER_NAME_39),
72-
requirements_lock = "//:requirements_lock.txt",
73-
requirements_windows = "//:requirements_windows.txt",
54+
hub_name = "pip",
55+
python_version = "3.10",
56+
requirements_lock = "//:requirements_lock_3_10.txt",
57+
requirements_windows = "//:requirements_windows_3_10.txt",
7458
)
75-
use_repo(pip, "pip")
59+
60+
# we use the pip_39 repo because entry points are not yet supported.
61+
use_repo(pip, "pip", "pip_39")
7662

7763
bazel_dep(name = "other_module", version = "", repo_name = "our_other_module")
7864
local_path_override(

‎examples/bzlmod/entry_point/BUILD.bazel

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
load("@pip//:requirements.bzl", "entry_point")
1+
load("@pip_39//:requirements.bzl", "entry_point")
22
load("@rules_python//python:defs.bzl", "py_test")
33

44
alias(

‎examples/bzlmod/libs/my_lib/BUILD.bazel

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
load("@pip//:requirements.bzl", "requirement")
2+
load("@rules_python//python:defs.bzl", "py_library")
3+
4+
py_library(
5+
name = "my_lib",
6+
srcs = ["__init__.py"],
7+
visibility = ["@//tests:__pkg__"],
8+
deps = [requirement("websockets")],
9+
)

‎examples/bzlmod/libs/my_lib/__init__.py

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import websockets
16+
17+
18+
def websockets_is_for_python_version(sanitized_version_check):
19+
# We are checking that the name of the repository folders
20+
# match the expexted generated names. If we update the folder
21+
# structure or naming we will need to modify this test
22+
return f"pip_{sanitized_version_check}_websockets" in websockets.__file__

‎examples/bzlmod/requirements.in

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
websockets
12
requests~=2.25.1
23
s3cmd~=2.1.0
34
yamllint>=1.28.0

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy