Skip to content

feat(toolchain): support freethreaded toolchains #2372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 27 commits into from
Nov 11, 2024
Merged

feat(toolchain): support freethreaded toolchains #2372

merged 27 commits into from
Nov 11, 2024

Conversation

aignas
Copy link
Collaborator

@aignas aignas commented Nov 5, 2024 •
edited
Loading

Before this PR freethreaded toolchains were not possible to be used,
this adds the minimum plumbing to get the things working. Coverage
support is also added.

Whilst at it:

  • Add plumbing to print checksums only for a particular python version.
  • Bump the remaining toolchain versions that used to use the 20241008 release
  • Pass around the loaded platform list so that we are only defining toolchains for the platforms that we have loaded the hermetic toolchain for.

Tested:

$ bazel run --//python/config_settings:python_version=3.13.0 --//python/config_settings:py_freethreaded="yes" //python/private:current_interpreter_executable
...
Python 3.13.0 experimental free-threading build (main, Oct 16 2024, 03:26:14) [Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Closes #2129.
Work towards #2386.

@aignas aignas requested a review from rickeylev as a code owner November 5, 2024 00:36
@aignas aignas mentioned this pull request Nov 5, 2024
Closed
@aignas aignas marked this pull request as draft November 5, 2024 00:40
@vfdev-5
Copy link
Contributor

vfdev-5 commented Nov 5, 2024

Seems like this PR is working to build XLA with Python free-threading using the following command:

bazel build \
    --@rules_python//python/config_settings:py_freethreaded="yes" \
    --define=use_enabled_free_threading=true \
    --test_output=all --spawn_strategy=sandboxed \
    --override_repository=rules_python=/rules_python \
    //xla/...

It downloads however two versions of python:

ls /root/.cache/bazel/_bazel_root/83d0ab0b1a11dcb5adba61b76e88291b/external/python                                                                                                         
...
python_x86_64-unknown-linux-gnu-freethreaded/                                                                            
python_x86_64-unknown-linux-gnu/
...

I assume that free-threaded version is used for building python extensions...

@aignas aignas marked this pull request as ready for review November 5, 2024 13:36
vfdev-5
vfdev-5 reviewed Nov 5, 2024
View reviewed changes
@aignas
Copy link
Collaborator Author

aignas commented Nov 5, 2024

@vfdev-5, I am not fully sure about why you have the toolchain in your cache. Could you please check it with bazel query or use --toolchain_resolution_debug=python to check?

@vfdev-5
Copy link
Contributor

vfdev-5 commented Nov 5, 2024

In my tests I deleted completely the cache and run xla build from the very beginning. In the console I saw it downloading python for platform x86_64-unknown-linux-gnu and after that x86_64-unknown-linux-gnu-freethreaded.
I'll retry with your suggestions to figure out

aignas
aignas commented Nov 6, 2024
View reviewed changes
Copy link
Collaborator Author

@aignas aignas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done with the self-review.

groodt
groodt approved these changes Nov 7, 2024
View reviewed changes
Copy link
Collaborator

@groodt groodt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM