fix(pypi): pull fewer wheels with experimental_index_url #3058

aignas · 2025-07-05T03:20:44Z

DO NOT MERGE: stacked on #3108, #3111, #3112.

Before this we would pull all of the wheels that the user target
configuration would be compatible with and that meant that it was not
customizable. This also meant that there were a lot of footguns in the
configuration where the select statements were not really foolproof.

With this PR we select only those sources that need to be for the
declared configurations.

Freethreaded support should be done by defining extra freethreaded
platforms using the new builder API. It is done as a followup in #3063.

This is also changing the default platforms to be only the fully supported
platforms. This makes the testing easier and avoids us running into
compatibility issues during the rollout.

Work towards #2747
Fixes #2759
Fixes #2849

aignas · 2025-07-05T03:22:23Z

Created this as WIP to signal that I am working on this and to invite feedback. The CI should be mostly happy but there are a few TODOs before we can fully merge and I want to test it a little bit more.

Freethreaded platform definition should be possible.
Probably something else that I need to think of more carefully.

TODO cleanup after this PR is proven to not break things:

Remove complex config_setting generation where we are generating them from whl filenames - it is just too complex to properly maintain and it has not proven to be effective here. In chore(pypi): remove unused config setting code #3065

Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.

aignas · 2025-07-06T12:46:43Z

The CI failure is a flake.

Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.

aignas · 2025-07-06T14:12:17Z

This would be ready to be reviewed, but because it is stacked on two PRs, let's get those out of the way first.

Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.

Stacked on bazel-contrib#3058 Since the parent PR is simplifying the code a great deal, we can remove the unused code and its tests. Hopefully this makes the whole codebase easier to contribute to. There is probably more cleanup that can be done, but since the CI is now green, it's a good start.

MODULE.bazel

examples/bzlmod/MODULE.bazel

python/private/pypi/extension.bzl

python/private/pypi/select_whl.bzl

keith · 2025-07-07T17:01:14Z

I hit this problem with this change (I don't support windows and use requirements_by_platform to exclude it too):

% bazel query 'deps(...)'
INFO: Repository rules_python++python+python_3_13_aarch64-pc-windows-msvc instantiated at:
  <builtin>: in <toplevel>
Repository rule python_repository defined at:
  /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:271:36: in <toplevel>
ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:76:34: An error occurred during the fetch of repository 'rules_python++python+python_3_13_aarch64-pc-windows-msvc':
   Traceback (most recent call last):
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl", line 76, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
ERROR: Evaluation of query "deps(...)" failed: preloading transitive closure failed: no such package '@@rules_python++python+python_3_13_aarch64-pc-windows-msvc//': java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
Loading: 2635 packages loaded

aignas · 2025-07-07T23:48:06Z

Thank you, I'll take a look later.

Marking this as a draft for now.

aignas · 2025-07-08T04:10:17Z

I hit this problem with this change (I don't support windows and use requirements_by_platform to exclude it too):

% bazel query 'deps(...)'
INFO: Repository rules_python++python+python_3_13_aarch64-pc-windows-msvc instantiated at:
  <builtin>: in <toplevel>
Repository rule python_repository defined at:
  /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:271:36: in <toplevel>
ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:76:34: An error occurred during the fetch of repository 'rules_python++python+python_3_13_aarch64-pc-windows-msvc':
   Traceback (most recent call last):
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl", line 76, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
ERROR: Evaluation of query "deps(...)" failed: preloading transitive closure failed: no such package '@@rules_python++python+python_3_13_aarch64-pc-windows-msvc//': java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
Loading: 2635 packages loaded

@keith, #3070 should fix your issue, I think. The toolchain pulling in query seems to be unrelated to pip integration, so if it works after #3070 and #3058 is merged, I think it is safe to say that the issue is resolved.

python/private/pypi/select_whl.bzl

aignas · 2025-07-18T13:43:35Z

@keith, @hartikainen, could you please retest this PR if it works with your setup.

Before this PR we would be constructing slightly different environments when the `env_marker_setting` is doing it in the analysis phase and when we are doing it in the repo phase due to how the defaults are handled. In this change we simply reuse the same select statements and add an extra helper that is allowing us to process that. Work towards bazel-contrib#2949 Prep for bazel-contrib#3058

aignas · 2025-07-21T01:54:37Z

I am going to split this into smaller PRs and keep this as the top of stack. Marking as draft to signal that this should not be merged.

This will be needed when we start selecting wheels entirely in the bzlmod extension evaluation phase (bazel-contrib#3058). This adds a few unit tests to just ensure that we conform to the spec even though the code is very simple. Work towards bazel-contrib#2747 Work towards bazel-contrib#2759 Work towards bazel-contrib#2849

DO NOT MERGE: stacked on bazel-contrib#3110 This PR only implements the selection algorithm where instead of selecting all wheels that are compatible with the set of target platforms, we select a single wheel that is most specialized for a particular *single* target platform. What is more, compared to the existing algorithm it does not assume a particular list of supported platforms and just fully implements the spec. Work towards bazel-contrib#2747 Work towards bazel-contrib#2759 Work towards bazel-contrib#2849

Before this PR we would be constructing slightly different environments when the `env_marker_setting` is doing it in the analysis phase and when we are doing it in the repo phase due to how the defaults are handled. In this change we simply reuse the same select statements and add an extra helper that is allowing us to process that. Work towards bazel-contrib#2949 Prep for bazel-contrib#3058

Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747

Before this we would pull all of the wheels that the user target configuration would be compatible with and that meant that it was not customizable. This also meant that there were a lot of footguns in the configuration where the select statements were not really foolproof. With this PR we select only those sources that need to be for the declared configurations. Freethreaded support is done by defining extra freethreaded platforms using the new builder API (bazel-contrib#3063). This is also changing the default platforms to be only the fully supported platforms. This makes the testing easier and avoids us running into compatibility issues during the roll out. Work towards bazel-contrib#2747 Fixes bazel-contrib#2759 Fixes bazel-contrib#2849

DO NOT MERGE: stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747

aignas · 2025-07-21T07:17:28Z

OK, the splitting is done.

aignas force-pushed the exp/pypi-simplify branch from 1d55c50 to c95dce6 Compare July 5, 2025 03:21

aignas changed the title ~~next: pypi stabilization of experimental_index_url~~ fix(pypi): pull fewer wheels with experimental_index_url Jul 5, 2025

aignas mentioned this pull request Jul 6, 2025

feat: freethreaded support for the builder API #3063

Open

aignas force-pushed the exp/pypi-simplify branch 2 times, most recently from dc50a12 to 855a673 Compare July 6, 2025 13:42

aignas mentioned this pull request Jul 6, 2025

chore(pypi): remove unused config setting code #3065

Draft

rickeylev reviewed Jul 6, 2025

View reviewed changes

aignas marked this pull request as ready for review July 7, 2025 13:15

aignas requested a review from groodt as a code owner July 7, 2025 13:15

This was referenced Jul 7, 2025

experimental_index_url slows down downloads by 25-50% #2849

Open

"Illegal ambiguous match" error with experimental_index_url #2759

Open

aignas marked this pull request as draft July 7, 2025 23:50

aignas marked this pull request as ready for review July 8, 2025 04:09

aignas mentioned this pull request Jul 8, 2025

Create a test that tests whl_library with an sdist #3071

Open

hartikainen reviewed Jul 8, 2025

View reviewed changes

python/private/pypi/select_whl.bzl Outdated Show resolved Hide resolved

aignas requested a review from hartikainen July 12, 2025 14:19

aignas marked this pull request as ready for review July 18, 2025 13:42

aignas requested a review from rickeylev July 19, 2025 23:43

aignas mentioned this pull request Jul 20, 2025

fix(pypi): reuse select dicts for constructing the env #3108

Open

aignas marked this pull request as draft July 21, 2025 01:54

aignas mentioned this pull request Jul 21, 2025

feat(pypi): add a standards compliant python_tag creator #3110

Open

aignas added 2 commits July 21, 2025 11:36

aignas force-pushed the exp/pypi-simplify branch from 2c2ff9e to f7641d5 Compare July 21, 2025 02:51

aignas mentioned this pull request Jul 21, 2025

feat(pypi): incrementally build platform configuration #3112

Open

aignas force-pushed the exp/pypi-simplify branch from f7641d5 to 8c916f3 Compare July 21, 2025 04:24

aignas force-pushed the exp/pypi-simplify branch from 8c916f3 to 89e58de Compare July 21, 2025 04:25

aignas added the type: pip pip/pypi integration label Jul 21, 2025

aignas marked this pull request as ready for review July 21, 2025 07:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(pypi): pull fewer wheels with experimental_index_url #3058

fix(pypi): pull fewer wheels with experimental_index_url #3058

aignas commented Jul 5, 2025 •

edited

Loading

Uh oh!

aignas commented Jul 5, 2025 •

edited

Loading

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

keith commented Jul 7, 2025 •

edited

Loading

Uh oh!

aignas commented Jul 7, 2025 •

edited

Loading

Uh oh!

aignas commented Jul 8, 2025

Uh oh!

Uh oh!

aignas commented Jul 18, 2025

Uh oh!

aignas commented Jul 21, 2025

Uh oh!

aignas commented Jul 21, 2025 •

edited

Loading

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Uh oh!

fix(pypi): pull fewer wheels with experimental_index_url #3058

Are you sure you want to change the base?

fix(pypi): pull fewer wheels with experimental_index_url #3058

Conversation

aignas commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

keith commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 8, 2025

Uh oh!

Uh oh!

aignas commented Jul 18, 2025

Uh oh!

aignas commented Jul 21, 2025

Uh oh!

aignas commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

aignas commented Jul 5, 2025 •

edited

Loading

aignas commented Jul 5, 2025 •

edited

Loading

keith commented Jul 7, 2025 •

edited

Loading

aignas commented Jul 7, 2025 •

edited

Loading

aignas commented Jul 21, 2025 •

edited

Loading