Allow the GIL to be enabled on the unix port, and add a CI job to test this configuration #17668

dpgeorge · 2025-07-13T12:48:32Z

Summary

Up until now, thread tests have only been run (under CI at least) on a small number of build configurations. PR #17655 enables the thread tests on a lot more of the existing builds, and this PR adds a new job which builds and tests the unix port with the GIL enabled.

(This PR builds on #17655 and includes all the commits from that PR.)

Running tests with the GIL enabled will help to test the esp32 port's threading implementation, which also uses the GIL.

Edit: updated some thread tests to add time.sleep(0) to the busy-wait loop, so that they pass when used with the native emitter.

Testing

Will be tested by CI.

Edit: new CI job runs correctly, and takes 1 minute 55 seconds. It runs the thread tests in bytecode and native emitter mode.

Trade-offs and Alternatives

Adds a new CI job which adds to the CI time, but it should only be a couple of minutes, and run in parallel.

github-actions · 2025-07-13T12:57:07Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

codecov · 2025-07-13T13:01:54Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.41%. Comparing base (0b698b8) to head (13aae9e).
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #17668      +/-   ##
==========================================
- Coverage   98.44%   98.41%   -0.04%     
==========================================
  Files         171      171              
  Lines       22208    22208              
==========================================
- Hits        21863    21855       -8     
- Misses        345      353       +8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dlech · 2025-07-13T17:09:44Z

tests/thread/mutate_bytearray.py

@@ -36,7 +37,7 @@ def th(n, lo, hi):

 # busy wait for threads to finish


Missed one.

But, TBH, I would still call all of them busy waits since the arg to sleep() is 0.

OK, I've reverted the change to the comment and just left them all as "busy wait for threads to finish".

dlech · 2025-07-13T17:17:47Z

tests/feature_check/target_info.py

+try:
+    import _thread
+
+    have_thread = "thread"


I was going to suggest to just make this "yes" here, but I see run-tests.py changes in a later commit, so it doesn't really matter.

Yes, it's changed later to be GIL or no-GIL.

projectgus

Changes look good to me. Suggest as part of this PR we also add some brief docs about the native emitter thread/scheduler starvation issue, and the sleep(0) workaround. I think the appropriate place is the dot point list in reference/speed_python.rst?

(While looking for that I also noticed that refernece/micropython.rst doesn't document the native annotation at all - it should probably have a short entry that links to the explanatory section in speed_python.rst? I can submit this in a follow-up, though.)

This is a workaround for the case where threading is enabled without a GIL. In such a configuration, creating a new global variable is not atomic and threads have race conditions resizing/accessing the global dict. Signed-off-by: Damien George <damien@micropython.org>

Builds with stackless enabled are slower than non-stackless, and this test takes around 2 minutes on the unix port of MicroPython with the standard unix parameters. So detect stackless mode and reduce the time of the test. Signed-off-by: Damien George <damien@micropython.org>

When detecting the target platform, check if the `_thread` module is available. If so, add the thread tests to the set of tests to run (unless the set of tests is explicitly given). With this change, the unix port no longer needs to explicitly run the set of thread tests, so that line has been removed from the Makefile. This change will make sure thread tests are run with other testing combinations. In particular, thread tests are now run: - on the unix port with the native emitter - on macOS builds - on unix qemu, the architectures MIPS, ARM and RISCV-64 Signed-off-by: Damien George <damien@micropython.org>

This commit adds a feature check to the test runner to detect if the target has the GIL or not. Currently this information is hard-coded: unix/PC targets and the rp2 port are considered to be GIL-less, and all other targets have the GIL enabled (if they have threading enabled). With the change here, some code is run to detect the GIL. That uses the fact that native code won't bounce the GIL and can hog it, and times how long code runs. Signed-off-by: Damien George <damien@micropython.org>

The qemu emulation introduces enough overhead that the `tests/thread/stress_aes.py` test overruns the default timeout. So increase it to allow this test to pass. Signed-off-by: Damien George <damien@micropython.org>

This test passes sometimes and fails other times. Eventually that should be fixed, but for now just skip this test. Signed-off-by: Damien George <damien@micropython.org>

The unix port can now be built with the GIL enabled, by passing MICROPY_PY_THREAD_GIL=1 on the make command line. Signed-off-by: Damien George <damien@micropython.org>

Signed-off-by: Damien George <damien@micropython.org>

The native emitter will not release/bounce the GIL when running code, so if it runs tight loops then no other threads get a chance to run (if the GIL is enabled). So for the thread tests, explicitly include a call to `time.sleep(0)` (or equivalent) to bounce the GIL and give other threads a chance to run. For some tests (eg `thread_coop.py`) the whole point of the test is to test that the GIL is correctly bounced. So for those cases for the use of the bytecode emitter for the busy functions. Signed-off-by: Damien George <damien@micropython.org>

Signed-off-by: Damien George <damien@micropython.org>

dpgeorge · 2025-07-17T05:07:20Z

Suggest as part of this PR we also add some brief docs about the native emitter thread/scheduler starvation issue, and the sleep(0) workaround.

I've now added this in the last commit.

While looking for that I also noticed that refernece/micropython.rst doesn't document the native annotation at all - it should probably have a short entry that links to the explanatory section in speed_python.rst?

Yes, would be good to document bytecode/native/viper decorators in docs/library/micropython.rst. Feel free to do that (but not urgent).

dpgeorge added port-unix tests Relates to tests/ directory in source labels Jul 13, 2025

dpgeorge added this to the release-1.26.0 milestone Jul 13, 2025

dlech reviewed Jul 13, 2025

View reviewed changes

dpgeorge force-pushed the tests-add-unix-gil-build branch from 526493b to 62f720f Compare July 14, 2025 12:23

dpgeorge requested a review from projectgus July 14, 2025 23:29

projectgus approved these changes Jul 15, 2025

View reviewed changes

hmaerki mentioned this pull request Jul 15, 2025

ESP32_DEVKIT / RUN-TESTS_STANDARD_NATIVE / thread/mutate_bytearray.py - Device crashed octoprobe/testbed_micropython#38

Open

dpgeorge added 11 commits July 17, 2025 14:39

tools/ci.sh: Increase timeout for unix qemu test runs.

c8b4484

The qemu emulation introduces enough overhead that the `tests/thread/stress_aes.py` test overruns the default timeout. So increase it to allow this test to pass. Signed-off-by: Damien George <damien@micropython.org>

tools/ci.sh: Skip thread/stress_heap.py test on macOS test run.

8d3bce7

This test passes sometimes and fails other times. Eventually that should be fixed, but for now just skip this test. Signed-off-by: Damien George <damien@micropython.org>

tools/ci.sh: Skip thread/stress_recurse.py on unix qemu test runs.

86551c9

This test passes sometimes and fails other times. Eventually that should be fixed, but for now just skip this test. Signed-off-by: Damien George <damien@micropython.org>

unix: Allow the GIL to be enabled.

17f8b58

The unix port can now be built with the GIL enabled, by passing MICROPY_PY_THREAD_GIL=1 on the make command line. Signed-off-by: Damien George <damien@micropython.org>

github/workflows: Add new CI job to test unix port with GIL enabled.

e924ac5

Signed-off-by: Damien George <damien@micropython.org>

docs/reference/speed_python: Document schedule/GIL limitation of native.

13aae9e

Signed-off-by: Damien George <damien@micropython.org>

dpgeorge force-pushed the tests-add-unix-gil-build branch from 62f720f to 13aae9e Compare July 17, 2025 05:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Allow the GIL to be enabled on the unix port, and add a CI job to test this configuration #17668

Allow the GIL to be enabled on the unix port, and add a CI job to test this configuration #17668

dpgeorge commented Jul 13, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 13, 2025

Uh oh!

codecov bot commented Jul 13, 2025 •

edited

Loading

Uh oh!

dlech Jul 13, 2025

Uh oh!

dpgeorge Jul 14, 2025

Uh oh!

dlech Jul 13, 2025

Uh oh!

dpgeorge Jul 14, 2025

Uh oh!

projectgus left a comment

Uh oh!

dpgeorge commented Jul 17, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

		@@ -36,7 +37,7 @@ def th(n, lo, hi):

		# busy wait for threads to finish

Uh oh!

Allow the GIL to be enabled on the unix port, and add a CI job to test this configuration #17668

Are you sure you want to change the base?

Allow the GIL to be enabled on the unix port, and add a CI job to test this configuration #17668

Conversation

dpgeorge commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Trade-offs and Alternatives

Uh oh!

github-actions bot commented Jul 13, 2025

Uh oh!

codecov bot commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dlech Jul 13, 2025

Choose a reason for hiding this comment

Uh oh!

dpgeorge Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

dlech Jul 13, 2025

Choose a reason for hiding this comment

Uh oh!

dpgeorge Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

projectgus left a comment

Choose a reason for hiding this comment

Uh oh!

dpgeorge commented Jul 17, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

dpgeorge commented Jul 13, 2025 •

edited

Loading

codecov bot commented Jul 13, 2025 •

edited

Loading