Skip to content

Commit 6dabe02

Browse files
rickeylevrickeylev
committed
fix: Don't require default Python version for pip hubs
This fixes the issue where a sub-module was required to always have a pip.parse() call configured for the default Python version if they used any pip.parse() call. Such a requirement puts sub-modules in an impossible situation: If they don't have the default version, they'll get an error. If they register the default version, but also register a specific version, they'll potentially cause an error if a root module changes the default to match their specific version (becaues two pip.parse() calls for the same version are made). The requirement to have the default version registered for a pip hub was only present to satisfy the `whl_library_alias` repository rule, which needed a Ptyhon version to map `//conditions:default` to. To fix, the `whl_library_alias` rule's `default_version` arg is made optional. When None is passed, the `//condiitions:default` condition is replaced with a `no_match_error` setting. This prevents the pip hub from being used with the version-unaware rules, but that makes sense: no wheels were setup for that version, so it's not like there is something that can be used anyways. Fixes #1320
1 parent 5c5ab5b commit 6dabe02

File tree

10 files changed

+148
-41
lines changed

10 files changed

+148
-41
lines changed

‎.bazelrc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
# This lets us glob() up all the files inside the examples to make them inputs to tests
44
# (Note, we cannot use `common --deleted_packages` because the bazel version command doesn't support it)
55
# To update these lines, run tools/bazel_integration_test/update_deleted_packages.sh
6-
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/whl_mods,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
7-
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/whl_mods,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
6+
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
7+
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
88

99
test --test_output=errors
1010

‎examples/bzlmod/other_module/BUILD.bazel

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
load("@python_versions//3.11:defs.bzl", compile_pip_requirements_311 = "compile_pip_requirements")
2+
3+
# NOTE: To update the requirements, you need to uncomment the rules_python
4+
# override in the MODULE.bazel.
5+
compile_pip_requirements_311(
6+
name = "requirements",
7+
requirements_in = "requirements.in",
8+
requirements_txt = "requirements_lock_3_11.txt",
9+
)

‎examples/bzlmod/other_module/MODULE.bazel

Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,19 +6,31 @@ module(
66
# that the parent module uses.
77
bazel_dep(name = "rules_python", version = "")
88

9-
# It is not best practice to use a python.toolchian in
10-
# a submodule. This code only exists to test that
11-
# we support doing this. This code is only for rules_python
12-
# testing purposes.
9+
# The story behind this commented out override:
10+
# This override is necessary to generate/update the requirements file
11+
# for this module. This is because running it via the outer
12+
# module doesn't work -- the `requirements.update` target can't find
13+
# the correct file to update.
14+
# Running in the submodule itself works, but submodules using overrides
15+
# is considered an error until Bazel 6.3, which prevents the outer module
16+
# from depending on this module.
17+
# So until 6.3 and higher is the minimum, we leave this commented out.
18+
# local_path_override(
19+
# module_name = "rules_python",
20+
# path = "../../..",
21+
# )
22+
1323
PYTHON_NAME_39 = "python_3_9"
1424

1525
PYTHON_NAME_311 = "python_3_11"
1626

1727
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
28+
1829
python.toolchain(
1930
configure_coverage_tool = True,
2031
python_version = "3.9",
2132
)
33+
2234
python.toolchain(
2335
configure_coverage_tool = True,
2436
# In a submodule this is ignored
@@ -29,6 +41,22 @@ python.toolchain(
2941
# created by the above python.toolchain calls.
3042
use_repo(
3143
python,
44+
"python_versions",
3245
PYTHON_NAME_39,
3346
PYTHON_NAME_311,
3447
)
48+
49+
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
50+
51+
pip.parse(
52+
hub_name = "other_module_pip",
53+
# NOTE: This version must be different than the root module's
54+
# default python version.
55+
# This is testing that a sub-module can use pip.parse() and only specify
56+
# Python versions that DON'T include whatever the root-module's default
57+
# Python version is.
58+
python_version = "3.11",
59+
requirements_lock = ":requirements_lock_3_11.txt",
60+
)
61+
62+
use_repo(pip, "other_module_pip")

‎examples/bzlmod/other_module/other_module/pkg/BUILD.bazel

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
1-
load("@python_3_11//:defs.bzl", py_binary_311 = "py_binary")
1+
load(
2+
"@python_3_11//:defs.bzl",
3+
py_binary_311 = "py_binary",
4+
)
25
load("@rules_python//python:defs.bzl", "py_library")
36

47
py_library(
@@ -13,11 +16,16 @@ py_library(
1316
# used only when you need to support multiple versions of Python
1417
# in the same project.
1518
py_binary_311(
16-
name = "lib_311",
17-
srcs = ["lib.py"],
19+
name = "bin",
20+
srcs = ["bin.py"],
1821
data = ["data/data.txt"],
22+
main = "bin.py",
1923
visibility = ["//visibility:public"],
20-
deps = ["@rules_python//python/runfiles"],
24+
deps = [
25+
":lib",
26+
"@other_module_pip//absl_py",
27+
"@rules_python//python/runfiles",
28+
],
2129
)
2230

2331
exports_files(["data/data.txt"])

‎examples/bzlmod/other_module/other_module/pkg/bin.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
import sys
2+
3+
import absl
4+
5+
print("Python version:", sys.version)
6+
print("Module 'absl':", absl)

‎examples/bzlmod/other_module/requirements.in

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
absl-py

‎examples/bzlmod/other_module/requirements_lock_3_11.txt

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#
2+
# This file is autogenerated by pip-compile with Python 3.11
3+
# by the following command:
4+
#
5+
# bazel run //other_module/pkg:requirements.update
6+
#
7+
absl-py==1.4.0 \
8+
--hash=sha256:0d3fe606adfa4f7db64792dd4c7aee4ee0c38ab75dfd353b7a83ed3e957fcb47 \
9+
--hash=sha256:d2c244d01048ba476e7c080bd2c6df5e141d211de80223460d5b3b8a2a58433d
10+
# via -r other_module/pkg/requirements.in

‎examples/bzlmod/tests/other_module/BUILD.bazel

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Tests to verify the root module can interact with the "other_module"
2+
# submodule.
3+
#
4+
# Note that other_module is seen as "our_other_module" due to repo-remapping
5+
# in the root module.
6+
7+
load("@bazel_skylib//rules:build_test.bzl", "build_test")
8+
9+
build_test(
10+
name = "other_module_bin_build_test",
11+
targets = [
12+
"@our_other_module//other_module/pkg:bin",
13+
],
14+
)

‎python/extensions/pip.bzl

Lines changed: 11 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -291,23 +291,18 @@ def _pip_impl(module_ctx):
291291

292292
for hub_name, whl_map in hub_whl_map.items():
293293
for whl_name, version_map in whl_map.items():
294-
if DEFAULT_PYTHON_VERSION not in version_map:
295-
fail((
296-
"Default python version '{version}' missing in pip " +
297-
"hub '{hub}': update your pip.parse() calls so that " +
298-
'includes `python_version = "{version}"`'
299-
).format(
300-
version = DEFAULT_PYTHON_VERSION,
301-
hub = hub_name,
302-
))
294+
if DEFAULT_PYTHON_VERSION in version_map:
295+
whl_default_version = DEFAULT_PYTHON_VERSION
296+
else:
297+
whl_default_version = None
303298

304299
# Create the alias repositories which contains different select
305300
# statements These select statements point to the different pip
306301
# whls that are based on a specific version of Python.
307302
whl_library_alias(
308303
name = hub_name + "_" + whl_name,
309304
wheel_name = whl_name,
310-
default_version = DEFAULT_PYTHON_VERSION,
305+
default_version = whl_default_version,
311306
version_map = version_map,
312307
)
313308

@@ -363,7 +358,7 @@ is used, this attribute defaults to that version of Python.
363358
mandatory = False,
364359
doc = """\
365360
A dict of labels to wheel names that is typically generated by the whl_modifications.
366-
The labels are JSON config files describing the modifications.
361+
The labels are JSON config files describing the modifications.
367362
""",
368363
),
369364
}, **pip_repository_attrs)
@@ -396,7 +391,7 @@ executable.""",
396391
),
397392
"copy_files": attr.string_dict(
398393
doc = """\
399-
(dict, optional): A mapping of `src` and `out` files for
394+
(dict, optional): A mapping of `src` and `out` files for
400395
[@bazel_skylib//rules:copy_file.bzl][cf]""",
401396
),
402397
"data": attr.string_list(
@@ -457,10 +452,10 @@ the BUILD files for wheels.
457452
attrs = _pip_parse_ext_attrs(),
458453
doc = """\
459454
This tag class is used to create a pip hub and all of the spokes that are part of that hub.
460-
This tag class reuses most of the pip attributes that are found in
455+
This tag class reuses most of the pip attributes that are found in
461456
@rules_python//python/pip_install:pip_repository.bzl.
462-
The exceptions are it does not use the args 'repo_prefix',
463-
and 'incompatible_generate_aliases'. We set the repository prefix
457+
The exceptions are it does not use the args 'repo_prefix',
458+
and 'incompatible_generate_aliases'. We set the repository prefix
464459
for the user and the alias arg is always True in bzlmod.
465460
""",
466461
),
@@ -484,7 +479,7 @@ def _whl_mods_repo_impl(rctx):
484479

485480
_whl_mods_repo = repository_rule(
486481
doc = """\
487-
This rule creates json files based on the whl_mods attribute.
482+
This rule creates json files based on the whl_mods attribute.
488483
""",
489484
implementation = _whl_mods_repo_impl,
490485
attrs = {

‎python/pip.bzl

Lines changed: 51 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,25 @@ load(":versions.bzl", "MINOR_MAPPING")
2222
compile_pip_requirements = _compile_pip_requirements
2323
package_annotation = _package_annotation
2424

25+
_NO_MATCH_ERROR_MESSAGE_TEMPLATE = """\
26+
No matching wheel for current configuration's Python version.
27+
28+
The current build configuration's Python version doesn't match any of the Python
29+
versions available for this wheel. This wheel supports the following Python versions:
30+
{supported_versions}
31+
32+
As matched by the `@{rules_python}//python/config_settings:is_python_<version>`
33+
configuration settings.
34+
35+
To determine the current configuration's Python version, run:
36+
`bazel config <config id>` (shown further below)
37+
and look for
38+
{rules_python}//python/config_settings:python_version
39+
40+
If the value is missing, then the "default" Python version is being used,
41+
which has a "null" version value and will not match version constraints.
42+
"""
43+
2544
def pip_install(requirements = None, name = "pip", **kwargs):
2645
"""Accepts a locked/compiled requirements file and installs the dependencies listed within.
2746
@@ -260,13 +279,10 @@ _multi_pip_parse = repository_rule(
260279

261280
def _whl_library_alias_impl(rctx):
262281
rules_python = rctx.attr._rules_python_workspace.workspace_name
263-
if rctx.attr.default_version not in rctx.attr.version_map:
264-
fail(
265-
"""
266-
Unable to find '{}' in your version map, you may need to update your requirement files.
267-
""".format(rctx.attr.version_map),
268-
)
269-
default_repo_prefix = rctx.attr.version_map[rctx.attr.default_version]
282+
if rctx.attr.default_version:
283+
default_repo_prefix = rctx.attr.version_map[rctx.attr.default_version]
284+
else:
285+
default_repo_prefix = None
270286
version_map = rctx.attr.version_map.items()
271287
build_content = ["# Generated by python/pip.bzl"]
272288
for alias_name in ["pkg", "whl", "data", "dist_info"]:
@@ -289,6 +305,7 @@ def _whl_library_render_alias_target(
289305
# is canonical, so we have to add a second @.
290306
if BZLMOD_ENABLED:
291307
rules_python = "@" + rules_python
308+
292309
alias = ["""\
293310
alias(
294311
name = "{alias_name}",
@@ -304,23 +321,42 @@ alias(
304321
),
305322
rules_python = rules_python,
306323
))
307-
alias.append("""\
308-
"//conditions:default": "{default_actual}",
309-
}}),
310-
visibility = ["//visibility:public"],
311-
)""".format(
324+
if default_repo_prefix:
312325
default_actual = "@{repo_prefix}{wheel_name}//:{alias_name}".format(
313326
repo_prefix = default_repo_prefix,
314327
wheel_name = wheel_name,
315328
alias_name = alias_name,
316-
),
317-
))
329+
)
330+
alias.append(' "//conditions:default": "{default_actual}",'.format(
331+
default_actual = default_actual,
332+
))
333+
334+
alias.append(" },") # Close select expression condition dict
335+
if not default_repo_prefix:
336+
supported_versions = sorted([python_version for python_version, _ in version_map])
337+
alias.append(' no_match_error="""{}""",'.format(
338+
_NO_MATCH_ERROR_MESSAGE_TEMPLATE.format(
339+
supported_versions = ", ".join(supported_versions),
340+
rules_python = rules_python,
341+
),
342+
))
343+
alias.append(" ),") # Close the select expression
344+
alias.append(' visibility = ["//visibility:public"],')
345+
alias.append(")") # Close the alias() expression
318346
return "\n".join(alias)
319347

320348
whl_library_alias = repository_rule(
321349
_whl_library_alias_impl,
322350
attrs = {
323-
"default_version": attr.string(mandatory = True),
351+
"default_version": attr.string(
352+
mandatory = False,
353+
doc = "Optional Python version in major.minor format, e.g. '3.10'." +
354+
"The Python version of the wheel to use when the versions " +
355+
"from `version_map` don't match. This allows the default " +
356+
"(version unaware) rules to match and select a wheel. If " +
357+
"not specified, then the default rules won't be able to " +
358+
"resolve a wheel and an error will occur.",
359+
),
324360
"version_map": attr.string_dict(mandatory = True),
325361
"wheel_name": attr.string(mandatory = True),
326362
"_rules_python_workspace": attr.label(default = Label("//:WORKSPACE")),

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy