Skip to content

PIP Installation failure when using partially vendorred dependencies using bazel vendor #3079

New issue
New issue
Open
Bug
#3096
Open
PIP Installation failure when using partially vendorred dependencies using bazel vendor#3079
Bug
#3096
Labels
type: pippip/pypi integration
@omar-droubi

Description

@omar-droubi
omar-droubi
opened on Jul 11, 2025

🐞 bug report

Affected Rule

The issue is caused by the rule:

//python/private/pypi:whl_library.bzl --> _whl_library_impl

Is this a regression?

No (AFAIK) this has never worked.

Description

When using bazel vendor to partially vendor the python dependencies, in case the user runs bazel build on a target which doesn't have all of it's python dependencies vendorred already, then the whl installation steps fails. This is likely because python is getting confused where it's running from. This is similar to #2996. where PYTHONHOME is not set.

🔬 Minimal Reproduction

BUILD.bazel

  
load("@rules_python//python:defs.bzl", "py_binary")
py_binary(
    name = "Binary-01",
    srcs = ["Binary-01.py"],
    main = "Binary-01.py",
    deps = [
        "@example_python_deps//requests"
    ]
)
py_binary(
    name = "Binary-02",
    srcs = ["Binary-02.py"],
    main = "Binary-02.py",
    deps = [
        "@example_python_deps//pyyaml"
    ]
)

MODULE.bazel

  
module(
    name = "rules_python_vendor",
    version = "1.0.0",
)
bazel_dep(name = "rules_python", version = "1.5.1")
bazel_dep(name = "rules_cc", version = "0.1.2")
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.12",
    is_default = True,
)
use_repo(python, "python_3_12", "python_versions")
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "example_python_deps",
    requirements_lock = "//example:requirements.txt",
    python_version = "3.12",
)
use_repo(pip, "example_python_deps")
  • Only Vendor dependencies used directly by //Binary-01
    bazelisk vendor --vendor_dir=.bazel-vendor //example:Binary-01
  • Clear bazel cache
    bazelisk clean --expunge
  • Attempt running //example:Binary-02
    bazelisk run --vendor_dir=.bazel-vendor //example:Binary-02

🔥 Exception or Error


bazelisk run --vendor_dir=.bazel-vendor //example:Binary-02
Analyzing: target //example:Binary-02 (0 packages loaded, 0 targets configured)
INFO: Repository rules_python++pip+example_python_deps_312_pyyaml instantiated at:
  : in 
Repository rule whl_library defined at:
  /home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/pypi/whl_library.bzl:590:30: in 
ERROR: /home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/repo_utils.bzl:83:16: An error occurred during the fetch of repository 'rules_python++pip+example_python_deps_312_pyyaml':
   Traceback (most recent call last):
        File "/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/pypi/whl_library.bzl", line 248, column 59, in _whl_library_impl
                environment = _create_repository_execution_environment(rctx, python_interpreter, logger = logger)
        File "/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/pypi/whl_library.bzl", line 227, column 51, in _create_repository_execution_environment
                cppflags.extend(_get_toolchain_unix_cflags(rctx, python_interpreter, logger = logger))
        File "/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/pypi/whl_library.bzl", line 109, column 52, in _get_toolchain_unix_cflags
                stdout = pypi_repo_utils.execute_checked_stdout(
        File "/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/pypi/pypi_repo_utils.bzl", line 160, column 45, in _execute_checked_stdout
                return repo_utils.execute_checked_stdout(
        File "/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/repo_utils.bzl", line 232, column 28, in _execute_checked_stdout
                return _execute_checked(*args, **kwargs).stdout
        File "/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/repo_utils.bzl", line 228, column 29, in _execute_checked
                return _execute_internal(fail_on_error = True, *args, **kwargs)
        File "/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/repo_utils.bzl", line 157, column 27, in _execute_internal
                return logger.fail((
        File "/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/repo_utils.bzl", line 93, column 39, in lambda
                fail = lambda message_cb: _log(-1, "FAIL", message_cb, fail),
        File "/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python+/python/private/repo_utils.bzl", line 83, column 16, in _log
                printer("\nrules_python:{} {}:".format(
Error in fail: 
rules_python:whl_library(@@rules_python++pip+example_python_deps_312_pyyaml) FAIL: repo.execute: GetPythonVersionForUnixCflags: end: failure:
  command: /home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python -B -I -c "import sys; print(f'{sys.version_info[0]}.{sys.version_info[1]}', end='')"
  return code: 1
  working dir: 
  timeout: 
  environment: 

===== stderr start =====
Could not find platform independent libraries 
Could not find platform dependent libraries 
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python'
  isolated = 1
  environment = 0
  user site = 0
  safe_path = 1
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.12'
  sys._base_executable = '/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python312.zip',
    '/install/lib/python3.12',
    '/install/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f8aee7fd740 (most recent call first):
  
===== stderr end =====
ERROR: no such package '@@rules_python++pip+example_python_deps_312_pyyaml//': 
rules_python:whl_library(@@rules_python++pip+example_python_deps_312_pyyaml) FAIL: repo.execute: GetPythonVersionForUnixCflags: end: failure:
  command: /home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python -B -I -c "import sys; print(f'{sys.version_info[0]}.{sys.version_info[1]}', end='')"
  return code: 1
  working dir: 
  timeout: 
  environment: 

===== stderr start =====
Could not find platform independent libraries 
Could not find platform dependent libraries 
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python'
  isolated = 1
  environment = 0
  user site = 0
  safe_path = 1
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.12'
  sys._base_executable = '/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python312.zip',
    '/install/lib/python3.12',
    '/install/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f8aee7fd740 (most recent call first):
  
===== stderr end =====
ERROR: /home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++pip+example_python_deps/pyyaml/BUILD.bazel:5:12: @@rules_python++pip+example_python_deps//pyyaml:pkg depends on @@rules_python++pip+example_python_deps_312_pyyaml//:pkg in repository @@rules_python++pip+example_python_deps_312_pyyaml which failed to fetch. no such package '@@rules_python++pip+example_python_deps_312_pyyaml//': 
rules_python:whl_library(@@rules_python++pip+example_python_deps_312_pyyaml) FAIL: repo.execute: GetPythonVersionForUnixCflags: end: failure:
  command: /home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python -B -I -c "import sys; print(f'{sys.version_info[0]}.{sys.version_info[1]}', end='')"
  return code: 1
  working dir: 
  timeout: 
  environment: 

===== stderr start =====
Could not find platform independent libraries 
Could not find platform dependent libraries 
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python'
  isolated = 1
  environment = 0
  user site = 0
  safe_path = 1
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.12'
  sys._base_executable = '/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/home/omar/.cache/bazel/_bazel_omar/49da98361449be12cf1305ccde5773cd/external/rules_python++python+python_3_12_host/python'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python312.zip',
    '/install/lib/python3.12',
    '/install/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f8aee7fd740 (most recent call first):
  
===== stderr end =====
ERROR: Analysis of target '//example:Binary-02' failed; build aborted: Analysis failed
INFO: Elapsed time: 25.346s, Critical Path: 0.04s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

🌍 Your Environment

Operating System:

  
Ubuntu 22.04 LTS under WSL2

Output of bazel version:

  
8.3.1

Rules_python version:

  
1.5.1

Anything else relevant?
The following patch seems to fix the issue. I am not sure what impact will setting PYTHONHOME have:

  
diff --git a/python/private/pypi/whl_library.bzl b/python/private/pypi/whl_library.bzl
index c271449b..dbd16909 100644
--- a/python/private/pypi/whl_library.bzl
+++ b/python/private/pypi/whl_library.bzl
@@ -410,6 +410,7 @@ def _whl_library_impl(rctx):
                     )
                 ]
 
+        environment["PYTHONHOME"] = str(python_interpreter.dirname)
         pypi_repo_utils.execute_checked(
             rctx,
             op = "whl_library.ExtractWheel({}, {})".format(rctx.attr.name, whl_path),

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: pippip/pypi integration

    Type

    Bug

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
      pFad - Phonifier reborn

      Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

      Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


      Alternative Proxies:

      Alternative Proxy

      pFad Proxy

      pFad v3 Proxy

      pFad v4 Proxy