can I register a in source build python as rules_python toolchain? #2172

comicfans · 2024-09-01T00:25:50Z

comicfans
Sep 1, 2024

Hello here, I'm new to bazel, I have a codebase with python extension written in C, currently it use rules_python pre-built interpreter

py = use_extension("@rules_python//python/extensions:python.bzl","python")
py.toolchain(
is_default = True,
python_version = "3.10"
)

pip.parse(
   hub_name = "pypi",
  python_version="3.10",
  requirements_lock....
)

and it also using pip.parse . now I'm building it with address sanitizer to check error, pre-build python can't load, I need to have a in source build python interpreter, so it use exactly same compiler and I can pass --with-address-sanitizer to configure command, I create following files:

MODULE.bazel

http_archive(
    name = "python_sanitizer",
    sha256 = "cefea32d3be89c02436711c95a45c7f8e880105514b78680c14fe76f5709a0f6",
    url = "https://www.python.org/ftp/python/3.10.14/Python-3.10.14.tgz",
    strip_prefix = "Python-3.10.14",
    patches = [
        "//third_party/python:configure_sanitizer.patch",
        "//third_party/python:3.10_tsan.patch",
    ],
    patch_args = ["-p1"],
    build_file = "//third_party/python:python.BUILD",
)

under third_party/python, I have python.BUILD

  load("@rules_foreign_cc//foreign_cc:defs.bzl", "configure_make")                                                                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                          
  package(default_visibility = ["//visibility:public"])                                                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                                                                                          
  filegroup(                                                                                                                                                                                                                                                                                                                                                              
      name = "all_srcs",                                                                                                                                                                                                                                                                                                                                                  
      srcs = glob(                                                                                                                                                                                                                                                                                                                                                     
          include = ["**"],                                                                                                                                                                                                                                                                                                                                            
          exclude = ["*.bazel"],                                                                                                                                                                                                                                                                                                                                          
      ),                                                                                                                                                                                                                                                                                                                                                               
  )                                                                                                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                                                                          
  configure_make(                                                                                                                                                                                                                                                                                                                                                         
      name = "python_sanitizer",                                                                                                                                                                                                                                                                                                                                       
      configure_options = [                                                                                                                                                                                                                                                                                                                                            
          "CFLAGS='-Dredacted=\"redacted\"'",                                                                                                                                                                                                                                                                                                                          
      ],                                                                                                                                                                                                                                                                                                                                                                  
        configure_command = "configure_sanitizer.sh",                                                                                                                                                                                                                                                                                                                                                        
      # rules_foreign_cc defaults the install_prefix to "python". This conflicts with the "python" executable that is generated.                                                                                                                                                                                                                                       
      install_prefix = "py_install",                                                                                                                                                                                                                                                                                                                                     
      lib_source = "@python_sanitizer:all_srcs",                                                                                                                                                                                                                                                                                                                       
      out_binaries = [                                                                                                                                                                                                                                                                                                                                                 
          "python3.10",                                                                                                                                                                                                                                                                                                                                                
      ],                                                                                                                                                                                                                                                                                                                                                               
      out_data_dirs = ["lib"],                                                                                                                                                                                                                                                                                                                                            
  )                                                                                                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                                                                         
  filegroup(                                                                                                                                                                                                                                                                                                                                                           
      name = "python3_bin",                                                                                                                                                                                                                                                                                                                                            
      srcs = [":python_sanitizer"]                                                                                                                                                                                                                                                                                                                                     
      output_group = "python3.10"                                                                                                                                                                                                                                                                                                                                      
  )     

# these script came from rules_foreign_cc example python

and configure_sanitizer.patch

diff --git a/configure_sanitizer.sh b/configure_sanitizer.sh
new file mode 100755
--- /dev/null
+++ b/configure_sanitizer.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
+
+if [[ $CFLAGS == *"-fsanitize=address"* ]]; then
+    set -- "$@" "--with-address-sanitizer"
+elif [[ $CFLAGS == *"-fsanitize=thread"* ]]; then
+    set -- "$@" "--with-thread-sanitizer"
+elif [[ $CFLAGS == *"-fsanitize=memory"* ]]; then
+    set -- "$@" "--with-memory-sanitizer"
+elif [[ $CFLAGS == *"-fsanitize=undefined"* ]]; then
+    set -- "$@" "--with-undefined-behavior-sanitizer"
+fi
+
+${SCRIPT_DIR}/configure "$@"

and 3.10_tsan.patch

From af968df68eb082b4543eb35d17d323a7a557435d Mon Sep 17 00:00:00 2001
From: "xinyu.wang" <comicfans44@gmail.com>
Date: Sun, 8 Sep 2024 01:16:43 +0200
Subject: [PATCH] thread sanitizer

---
 Doc/using/configure.rst      |  7 +++++++
 Include/Python.h             |  8 ++++++++
 Lib/test/support/__init__.py | 17 +++++++++++------
 Lib/test/test_io.py          |  9 ++++++---
 configure                    | 25 +++++++++++++++++++++++++
 configure.ac                 | 18 ++++++++++++++++++
 6 files changed, 75 insertions(+), 9 deletions(-)

diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst
index ab33e0a06f5..99033289d13 100644
--- a/Doc/using/configure.rst
+++ b/Doc/using/configure.rst
@@ -336,6 +336,13 @@ Debug options
 
    .. versionadded:: 3.6
 
+.. option:: --with-thread-sanitizer
+
+   Enable ThreadSanitizer data race detector, ``tsan``
+   (default is no).
+
+   .. versionadded:: 3.13
+
 
 Linker options
 --------------
diff --git a/Include/Python.h b/Include/Python.h
index d3186c32e35..035c75f1df3 100644
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -62,10 +62,18 @@
 #      define _Py_ADDRESS_SANITIZER
 #    endif
 #  endif
+#  if __has_feature(thread_sanitizer)
+#    if !defined(_Py_THREAD_SANITIZER)
+#      define _Py_THREAD_SANITIZER
+#    endif
+#  endif
 #elif defined(__GNUC__)
 #  if defined(__SANITIZE_ADDRESS__)
 #    define _Py_ADDRESS_SANITIZER
 #  endif
+#  if defined(__SANITIZE_THREAD__)
+#    define _Py_THREAD_SANITIZER
+#  endif
 #endif
 
 #include "pymath.h"
diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
index b7cf1e28581..5456f9a8705 100644
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -367,10 +367,10 @@ def wrapper(*args, **kw):
     return decorator
 
 
-def check_sanitizer(*, address=False, memory=False, ub=False):
+def check_sanitizer(*, address=False, memory=False, ub=False, thread=False):
     """Returns True if Python is compiled with sanitizer support"""
-    if not (address or memory or ub):
-        raise ValueError('At least one of address, memory, or ub must be True')
+    if not (address or memory or ub or thread):
+        raise ValueError('At least one of address, memory, ub or thread must be True')
 
 
     _cflags = sysconfig.get_config_var('CFLAGS') or ''
@@ -387,18 +387,23 @@ def check_sanitizer(*, address=False, memory=False, ub=False):
         '-fsanitize=undefined' in _cflags or
         '--with-undefined-behavior-sanitizer' in _config_args
     )
+    thread_sanitizer = (
+        '-fsanitize=thread' in _cflags or
+        '--with-thread-sanitizer' in _config_args
+    )
     return (
         (memory and memory_sanitizer) or
         (address and address_sanitizer) or
-        (ub and ub_sanitizer)
+        (ub and ub_sanitizer) or
+        (thread and thread_sanitizer)
     )
 
 
-def skip_if_sanitizer(reason=None, *, address=False, memory=False, ub=False):
+def skip_if_sanitizer(reason=None, *, address=False, memory=False, ub=False, thread=False):
     """Decorator raising SkipTest if running with a sanitizer active."""
     if not reason:
         reason = 'not working with sanitizers active'
-    skip = check_sanitizer(address=address, memory=memory, ub=ub)
+    skip = check_sanitizer(address=address, memory=memory, ub=ub, thread=thread)
     return unittest.skipIf(skip, reason)
 
 
diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
index 8dae85ac4f5..a9235b09545 100644
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -1547,7 +1547,8 @@ def test_truncate_on_read_only(self):
 class CBufferedReaderTest(BufferedReaderTest, SizeofTest):
     tp = io.BufferedReader
 
-    @skip_if_sanitizer(memory=True, address=True, reason= "sanitizer defaults to crashing "
+    @skip_if_sanitizer(memory=True, address=True, thread=True,
+                       reason="sanitizer defaults to crashing "
                        "instead of returning NULL for malloc failure.")
     def test_constructor(self):
         BufferedReaderTest.test_constructor(self)
@@ -1912,7 +1913,8 @@ def test_slow_close_from_thread(self):
 class CBufferedWriterTest(BufferedWriterTest, SizeofTest):
     tp = io.BufferedWriter
 
-    @skip_if_sanitizer(memory=True, address=True, reason= "sanitizer defaults to crashing "
+    @skip_if_sanitizer(memory=True, address=True, thread=True,
+                       reason="sanitizer defaults to crashing "
                        "instead of returning NULL for malloc failure.")
     def test_constructor(self):
         BufferedWriterTest.test_constructor(self)
@@ -2411,7 +2413,8 @@ def test_interleaved_readline_write(self):
 class CBufferedRandomTest(BufferedRandomTest, SizeofTest):
     tp = io.BufferedRandom
 
-    @skip_if_sanitizer(memory=True, address=True, reason= "sanitizer defaults to crashing "
+    @skip_if_sanitizer(memory=True, address=True, thread=True,
+                       reason="sanitizer defaults to crashing "
                        "instead of returning NULL for malloc failure.")
     def test_constructor(self):
         BufferedRandomTest.test_constructor(self)
diff --git a/configure b/configure
index 4b71c4e00f8..f74f50731aa 100755
--- a/configure
+++ b/configure
@@ -832,6 +832,7 @@ with_lto
 with_address_sanitizer
 with_memory_sanitizer
 with_undefined_behavior_sanitizer
+with_thread_sanitizer
 with_hash_algorithm
 with_tzpath
 with_libs
@@ -1560,6 +1561,8 @@ Optional Packages:
   --with-undefined-behavior-sanitizer
                           enable UndefinedBehaviorSanitizer undefined
                           behaviour detector, 'ubsan' (default is no)
+  --with-thread-sanitizer enable ThreadSanitizer data race detector, 'tsan'
+                          (default is no)
   --with-hash-algorithm=[fnv|siphash24]
                           select hash algorithm for use in Python/pyhash.c
                           (default is SipHash24)
@@ -9684,6 +9687,28 @@ with_ubsan="no"
 fi
 
 
+{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for --with-thread-sanitizer" >&5
+printf %s "checking for --with-thread-sanitizer... " >&6; }
+
+# Check whether --with-thread_sanitizer was given.
+if test ${with_thread_sanitizer+y}
+then :
+  withval=$with_thread_sanitizer;
+{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $withval" >&5
+printf "%s\n" "$withval" >&6; }
+BASECFLAGS="-fsanitize=thread $BASECFLAGS"
+LDFLAGS="-fsanitize=thread $LDFLAGS"
+with_tsan="yes"
+
+else $as_nop
+
+{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: no" >&5
+printf "%s\n" "no" >&6; }
+with_tsan="no"
+
+fi
+
+
 # Set info about shared libraries.
 
 
diff --git a/configure.ac b/configure.ac
index ac3be3850a9..6b74e837e86 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2612,6 +2612,24 @@ AC_MSG_RESULT(no)
 with_ubsan="no"
 ])
 
+AC_MSG_CHECKING([for --with-thread-sanitizer])
+AC_ARG_WITH(
+  [thread_sanitizer],
+  [AS_HELP_STRING(
+    [--with-thread-sanitizer],
+    [enable ThreadSanitizer data race detector, 'tsan' (default is no)]
+  )],
+[
+AC_MSG_RESULT([$withval])
+BASECFLAGS="-fsanitize=thread $BASECFLAGS"
+LDFLAGS="-fsanitize=thread $LDFLAGS"
+with_tsan="yes"
+],
+[
+AC_MSG_RESULT([no])
+with_tsan="no"
+])
+
 # Set info about shared libraries.
 AC_SUBST(SHLIB_SUFFIX)
 AC_SUBST(LDSHARED)
-- 
2.34.1

and BUILD

load("@rules_python//python:defs.bzl", "py_runtime", "py_runtime_pair")

py_runtime(
    name = "python3.10",
    files = ["@python_sanitizer"],
    interpreter = "@python_sanitizer//:python3_bin",
    python_version = "PY3",
)

py_runtime_pair(
    name = "py3.10",
    py3_runtime = ":python3.10",
)

toolchain(
    name = "python_3_10_host",
    toolchain = ":py3.10",
    #    toolchain_type = "@bazel_tools//tools/python:toolchain_type",
    toolchain_type = "@rules_python//python:toolchain_type",
)

now in root MODULE.bazel,

py = use_extension("@rules_python//python/extensions:python.bzl","python")
py.toolchain(
is_default = True,
python_version = "3.10"
)

register_toolchains("//third_party/python:python_3_10_host"). # keep everything as before, just add this line

pip.parse(
  hub_name...
  python_interpreter_target = "@python_sanitizer//:python3_bin",
  version = "3.10"
requirement...
)

now python target will use my in-source build python.

original part (I think I don't need them anymore, just keep them for easier reference):

if I replace

py = use_extension("@rules_python//python/extensions:python.bzl","python")
py.toolchain(
is_default = True,
python_version = "3.10"
)

with

register_toolchains("//third_party/python:python_3_10_host")

when I try to build some target, it said

ERROR: /home/xwang/.cache/bazel/_bazel_xwang/eec88c563263b4f71691ce77bb7b2ea3/external/rules_python~~pip~pypi/numpy/B
UILD.bazel:27:6: configurable attribute "actual" in @@rules_python~~pip~pypi//numpy:pkg doesn't match this configurat
ion: No matching wheel for current configuration's Python version.

The current build configuration's Python version doesn't match any of the Python
wheels available for this wheel. This wheel supports the following Python
configuration settings:
    //_config:is_python_3.10

To determine the current configuration's Python version, run:
    `bazel config <config id>` (shown further below)
and look for
    rules_python//python/config_settings:python_version

If the value is missing, then the "default" Python version is being used,
which has a "null" version value and will not match version constraints.


This instance of @@rules_python~~pip~pypi//numpy:pkg has configuration identifier 1578ff5. To inspect its configurati
on, run: bazel config 1578ff5.

For more help, see https://bazel.build/docs/configurable-attributes#faq-select-choose-condition

then I try to add

build --@rules_python//python/config_settings:python_version=3.10

to bazelrc
it gives another error:

ERROR: /home/xwang/.cache/bazel/_bazel_xwang/eec88c563263b4f71691ce77bb7b2ea3/external/rules_python~/python/cc/BUILD.bazel:15:22: Whil
e resolving toolchains for target @@rules_python~//python/cc:current_py_cc_headers (46af15e): No matching toolchains found for types @
@rules_python~//python/cc:toolchain_type.
To debug, rerun with --toolchain_resolution_debug='@@rules_python~//python/cc:toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro.

how can I resolve it? thanks for advise.

basically I still want to use pre-build python from rules_python for default build, only switch the python runtime to in source build python when detect -fsanitize=xxx in copts, is it possible ?

rickeylev · 2024-09-01T03:14:36Z

rickeylev
Sep 1, 2024
Maintainer

Ah, you're going to be pretty deep into defining a custom toolchain.

I don't have time right now to post in detail about how to go about doing this. Have a look at python/private/py_toolchain_suite.bzl for how the hermetic and local toolchains are defined. It'll show the toolchain types you need to define and what needs to be specified on them. In short, to satisfy current_py_cc_headers, the python/cc:toolchain_type toolchain needs to be configured with the headers.

Switching based on copts won't be possible. You could try to detect the global --copt flag, but you can't change the python toolchain based on what the copts an arbitrary cc library might be using.

I wouldn't worry about activating based on flags at this early step. Just leave target_compatible_with blank and target_settings blank, register your toolchain first, and get that working.

Good luck!

4 replies

comicfans Sep 1, 2024
Author

Thank you very much for the information! will check that part of code. I'm new to bazel and still puzzling about how everything works together

aignas Sep 1, 2024
Maintainer

To add, the pip.parse right now relies on a prebuilt python interpreter to parse Metadata and similar, so you should leave the pip.parse interpreter target as is and rely on the hermetic toolchain. In theory it would be possible to use python interpreter for this, but that code is still work in progress as of now.

comicfans Sep 1, 2024
Author

I think it's not important to me which interpreter is used for pip.parse, what I want is that in my custom config build, all bazel script on python target should kept the same (or with minimal change), and the in source build python will be used as include/lib to my extension, the executable used to run my extension.

comicfans Sep 8, 2024
Author

If I understand correctly, If I don't care about which interpreter is used in pip.parse, then I can leave rules_python toolchain as it is, and only register my own will works. now I managed to make it work, it always use my in-source build interpreter even for normal build, but I guess that only requires some target_compatible_with config , no need to touch python/private/py_toolchain_suite.bzl

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

can I register a in source build python as rules_python toolchain? #2172

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Uh oh!

can I register a in source build python as rules_python toolchain? #2172

Uh oh!

Uh oh!

comicfans Sep 1, 2024

Replies: 1 comment · 4 replies

Uh oh!

Uh oh!

rickeylev Sep 1, 2024 Maintainer

Uh oh!

comicfans Sep 1, 2024 Author

Uh oh!

aignas Sep 1, 2024 Maintainer

Uh oh!

comicfans Sep 1, 2024 Author

Uh oh!

comicfans Sep 8, 2024 Author

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

comicfans
Sep 1, 2024

Replies: 1 comment 4 replies

rickeylev
Sep 1, 2024
Maintainer

comicfans Sep 1, 2024
Author

aignas Sep 1, 2024
Maintainer

comicfans Sep 1, 2024
Author

comicfans Sep 8, 2024
Author