Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.32%. Comparing base (8f2dbf4) to head (3eb5d62).

@@           Coverage Diff           @@
##           master    #2478   +/-   ##
=======================================
  Coverage   90.32%   90.32%           
=======================================
  Files          16       16           
  Lines        2439     2439           
=======================================
  Hits         2203     2203           
  Misses        236      236           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Thanks!

So am I understanding correctly that binaries built with this change - including the wheels we publish - would leave the GIL off in free-threaded Python, potentially exposing the user to bugs if our code is not completely thread safe?

Is there a way we could make this a convenient option to set at build time, but still have the wheels we publish on PyPI require the GIL for now? I.e. I want the first round of people testing h5py without the GIL to build h5py from source, as a strong signal that you're testing something that may not work.

I can see the messaging in the Python docs is that free-threading is experimental, but I expect people are going to be really excited about this. I'd much rather enthusiastic non-experts see h5py as incompatible for now, than have it claim compatibility when it might not work.

freethreading python actually requires special wheels altogether, and there are still not built by default with cibuildwheel. This changes nothing for regular wheels, and only makes a difference for anyone we wants to build from source in freethreading Python. I very much agree that downstream testers should build from source (or maybe get a nightly if we can set this up for ubuntu), and I am not proposing we publish stable cp313t wheels for now !

Great, thanks!

So these markers allow it to build freethreading compatible binaries if you ask. How do you ask for that?

So these markers allow it to build freethreading compatible binaries if you ask.

Not quite: you can already build free-threading binaries without them, it's just that, by default the GIL is re-enabled when one imports any extension that doesn't explicitly declare that it's fine with having the GIL released, and you get the following warning:

RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'h5py._errors', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

So it is already possible to get the same effect (disable the warning) on the user side by invoking python with the appropriate flags (and maybe we should actually just let users do that for now, up for debate).

How do you ask for that?

Locally, one needs to install Python 3.13t. Cython 3.1.0 is also needed (but not released yet) so it should be built from source or installed from https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/ . Build isolation should also be turned off for this reason.

In cibuildwheel, free-threaded builts are controled by a flag which is off by default.

Also note that NumPy 2.1.0 has cp313 wheels for mac and linux, but not for windows, so if anyone wants to try it there, they'll also need to build numpy from source (which I guess will likely not work out of the box).

Right, sorry, replace 'free-threading binaries' with 'extension modules that don't trigger re-enabling the GIL'.

Thanks, I hadn't realised that it was possible to override the automatic re-enabling of the GIL. That will presumably be the easiest way to test h5py until we're ready to publish wheels that claim to be compatible with free-threading.

Good point, all I really want to do is to make sure that testing is possible downstream, but this way may give off the wrong message that we're completely ready. I'll try setting up a CI job instead, and draft this for now.

I think the first piece of the puzzle would be something like

diff --git a/.github/workflows/build_wheels.yml b/.github/workflows/build_wheels.yml
index 2e9a1b6a..6fff93df 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -118,6 +118,7 @@ jobs:
           TOX_TEST_LIMITED: ${{ env.TOX_TEST_LIMITED }}
           CIBW_PRERELEASE_PYTHONS: ${{ env.CIBW_PRERELEASE_PYTHONS }}
           CIBW_BEFORE_TEST: ${{ env.CIBW_BEFORE_TEST }}
+          CIBW_FREE_THREADED_SUPPORT: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') }}
         if: steps.triage.outputs.skip != '1'

       # And upload the results

However, as I mentionned earlier, building requires Cython 3.1 (unreleased) and it looks like it could land soon enough that it's not worth special casing it now.
Another (small) hurdle is that setting PYTHON_GIL=0 unconditionally doesn't work since the regular build for Python 3.13 will err with

Fatal Python error: config_read_gil: PYTHON_GIL / -X gil are not supported by this build

I do not think we should mark as "free-threading-safe" until we have done some testing forcing YOLO mode (e.g. PYTHON_GIL=0). I think it reasonable to expect down-stream to test in the same way until we are sure.

To start with we should add a cp313t job to the test matrix that runs the current test suite with PYTHON_GIL=0 and see what happens!

I except this will become significantly easier to setup soon: actions/setup-pyhon just merged support for free-threaded builds a couple days ago. We just need to wait for a new release (currently, the latest one is 5.4.0)

We just need to wait for a new release (currently, the latest one is 5.4.0)

FYI, v5.5.0 is available now and has full free-threading support.

+ CIBW_FREE_THREADED_SUPPORT: ...

Note that this just changed to CIBW_ENABLE: cpython-freethreading (see https://cibuildwheel.pypa.io/en/stable/options/#enable)

thanks for reminding me Ralf !

Are you planning to wait until the Cython 3.1.0 release before merging this? Even then, it's probably pretty aggressive to require that immediately as the minimum supported version. If not, you're probably a lot better off passing the freethreading_compatible=true dynamically rather than in directive comments. Something along the lines of:

compiler_directives = {}
if Version(cython_version) >= Version("3.1.0b1"):
    compiler_directives["freethreading_compatible"] = True

setup(
    ext_modules=cythonize(
        extensions,
        compiler_directives=compiler_directives,
    )
)

(from https://py-free-threading.github.io/porting-extensions/#__tabbed_1_3)

That way this PR will mergeable now, and you don't unnecessarily force distros onto a brand new Cython release.

Thanks for the suggestion. Indeed, I intend to polish this before undrafting; bumping our Cython requirement is a shortcut I'm taking to experiment quickly, not something I want to ship.

Failing on all platforms with error messages similar to

ERROR: h5py-3.13.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is not a supported wheel on this platform.

I would normally suspect an outdated pip to raise this error, but it also looks like cibuildwheel correctly installs the latest version (currently 25.0.1), so I don't quite get what's going on.

If you read the log carefully you'll see that the initial install of the built wheel is actually fine. It fails in the cibw_test_command, which employs tox:

  .pkg_external: remove tox env folder /Users/runner/work/h5py/h5py/.tox/.pkg_external
  py313-test-deps: install_package_deps> python -I -m pip install 'numpy>=1.19.3'
  py313-test-deps: install_package> python -I -m pip install --force-reinstall --no-deps /private/var/folders/r1/4882c7yd7wx9hcpc__xp04_m0000gn/T/cibw-run-g5gtcp95/cp313t-macosx_arm64/repaired_wheel/h5py-3.13.0-cp313-cp313t-macosx_11_0_arm64.whl
  ERROR: h5py-3.13.0-cp313-cp313t-macosx_11_0_arm64.whl is not a supported wheel on this platform.
  py313-test-deps: exit 1 (0.30 seconds) /Users/runner/work/h5py/h5py> python -I -m pip install --force-reinstall --no-deps /private/var/folders/r1/4882c7yd7wx9hcpc__xp04_m0000gn/T/cibw-run-g5gtcp95/cp313t-macosx_arm64/repaired_wheel/h5py-3.13.0-cp313-cp313t-macosx_11_0_arm64.whl pid=15744
  py313-test-deps: FAIL ✖ in 5.22 seconds

tox doesn't have free-threading support yet (tox-dev/tox#3391). You may want to drop that - tox inside cibuildwheel testing is anyway weird - as you can see from the CI log all dependencies are already installed successfully, and then tox fails on trying to pip install --force-reinstall them.

Ah ! I agree that running tox in a containerized context is weird, I just failed to realize that it might be where the problem was, thanks for pointing it out !
The issue is that, in our current CI setup, wheel building is 100% coupled with tox; it'd be indeed desirable to separate them, but we would either loose part of our test matrix or end up duplicating a lot of logic from tox.ini... that is, if we choose to keep using tox. I'm not a fan of this tool so I would lean towards just dropping it entirely, but obviously this needs discussion first (and I don't expect my admittedly radical take to be consensual).

that is, if we choose to keep using tox. I'm not a fan of this tool so I would lean towards just dropping it entirely, but obviously this needs discussion first

I have mixed feelings about tox, but I do see a definite advantage in trying to keep as much as possible of the 'how to build and test this' info in a format that's not tied to one specific CI provider, or even to running in CI at all. h5py has used quite a few different CI platforms over the years, and while for now it looks like Github Actions might be a good answer, it still seems sensible not to tie ourselves to it more than we have to.

The flip side of course is that we haven't actually done a great job of this; more of that detail lives in CI config than in tox.ini anyway. 🤷

(and I don't expect my admittedly radical take to be consensual).

no idea about preferences/concensus in this project, but I'd say it's far from radical at least. I quite dislike tox myself, as do quite a few people I collaborate with, and I consider the design fundamentally flawed. It makes it effectively impossible to use uv/conda/pixi/etc. (and people are really attached to their tool of choice here). Imho such tooling should be layered such that you can define your test tasks without strong coupling to environment setup. spin has the right idea there, you can use it within any environment manager - and absent a tool like spin you should be able to use pytest directly).

In terms of environment/dependency management, there's effectively very little of interest in tox.ini besides the numpy and Python versions by the way. The more interesting/challenging dependencies seem to be HDF5, MPI, and compilers - none of which can be controlled with tox.

FYI from colleagues I heard that in a couple of projects switching from tox to nox unblocked free-threading support. We don't know conclusively that there are no issues in nox (there's no issue, PR or docs that touches on this) but anecdotally installing packages and running tests works. nox is very similar to tox, so perhaps a less invasive change.

tox-dev/tox#3391 was just resolved in tox 4.26.0, and Cython 3.1 was shipped last week. Let's see where we're at.

This is starting to look good. It'd be great to also set PYTHON_GIL=0, but this can only be done conditionally since the gil-enabled build of Python 3.13 errors if this variable is defined (even if set to 1 ...). I think I'll post-pone this bit for now and finally undraft this first PR.

This looks pretty safe to merge (at least to me). I guess if no-one else merges it in a week, I'll do it?

If I've understood this correctly, the markers and the CIBW_ENABLE option would combine to create wheels which claim they are compatible with free-threading (i.e. importing h5py does not trigger re-enabling the GIL)? If that is right, please hold off on merging for now. I could of course exclude the cp313t wheel manually when doing a release, but I'd rather not have to remember that.

I think the first step is to get some realistic testing of h5py with the GIL disabled, before we claim it's compatible. Running our test suite is a good first step - I think this PR does this, though it would be good to ensure the GIL is not getting accidentally reenabled during the test run. But I don't think most of our existing tests really exercise multiple threads. We should either look for a realistic example using threads & h5py, or create our own stress-tests doing HDF5 operations from multiple threads. The callback-based iterators like group.visit() and dset.id.chunk_iter() could be useful for this, because we can pause in the middle of an HDF5 call. It would also be a good idea to test the file-like-object support, which involves HDF5 calling back into Python at a lower level.

This doesn't necessarily need to be part of our regular test suite for now; it could be a separate script that we run manually for starters. But I think testing with PYTHON_GIL=0 (or YOLO-mode, thanks @tacaswell ) should come before claiming compatibility.

I meant this a first step, mainly to unblock interested parties downstream so we can get feedback from actual applications; In other words, building cp3**t wheels doesn't equate to "claiming support". Well, maybe to some level, but in any case we'd need to clarify in docs/release notes what's actually supported and what's not.

Following the introduction of free-threading PyPI troves, my intention would be to get a release (or maybe just nightlies ?) with the level 1 support ("Experimental").

I don't have deep-enough knowledge of h5py to know where thread-safety may arise, so I'm not in a good position to add concurrency tests without help, but I'd gladly work on making some if you can give me a general idea of what should be stress-tested.

In terms of testing, I think the starting point is just to do a bunch of HDF5 operations from concurrent threads:

pytest-run-parallel does a lot of this automatically, so should smoke out most the obvious issues without having to write dedicated concurrent tests. See https://py-free-threading.github.io/testing/ for some more context. I'd start there; install it in your test env locally, add --parallel-threads=2 and see what fails. Once that is happy, push it a bit (say --parallel-threads=12).

Sounds reasonable. In fact, running our test suite with pytest-run-parallel with 4 threads immediately reveals thread safety issues (seg faults all over the place), so I'm convinced we shouldn't allow downstream testing without PYTHON_GIL=0 just yet.

Actually, if I only use 2 threads, I get clearer error messages.
Most of the exceptions look like these

(...)
PARALLEL FAILED h5py/tests/test_group.py::TestAdditionalMappingFuncs::test_update_dict - OSError: Unable to create link (name already exists)
PARALLEL FAILED h5py/tests/test_group.py::TestAdditionalMappingFuncs::test_update_iter - OSError: Unable to create link (name already exists)
PARALLEL FAILED h5py/tests/test_group.py::TestAdditionalMappingFuncs::test_update_kwargs - OSError: Unable to create link (name already exists)
PARALLEL FAILED h5py/tests/test_group.py::TestGet::test_get_class - ValueError: Unable to synchronously create group (name already exists)
PARALLEL FAILED h5py/tests/test_group.py::TestGet::test_get_default - ValueError: Unable to synchronously create group (name already exists)
PARALLEL FAILED h5py/tests/test_group.py::TestGet::test_get_link - ValueError: Unable to synchronously create group (name already exists)
PARALLEL FAILED h5py/tests/test_group.py::TestGet::test_get_link_class - ValueError: Unable to synchronously create group (name already exists)
(...)

I think they are all legit exceptions from HDF5 itself ? I guess these tests should be marked as @pytest.mark.thread_unsafe (defined in pytest-run-parallel) for now, and we should really be focusing on everything else that's failing.
I'm going to run experiments in that direction in a different branch.

It's clear now that most of this PR is premature, so, switching back to draft mode.

Yeah, if two threads try to create an object in the same file with the same name, it's expected that one of them will fail (with or without the GIL in play). That's not unsafe, just the tests are written with an assumption that doesn't hold when running multiple of the same in parallel.

Of course, my mistake. I'm trying to get rid of this assumption in #2588

I'm bouncing around my dependencies checking on 3.14 and free-threading support and found this PR. One thing you could add to this PR is the free-threading classifier in pyproject.toml:

Programming Language :: Python :: Free Threading :: 2 - Beta

See https://py-free-threading.github.io/porting/#define-and-document-thread-safety-guarantees

Thanks for working on this!

at this stage I would rather go with Programming Language :: Python :: Free Threading :: 1 - Unstable, and even that is premature in our case, but I'll circle back to it !