gh-135427: Fix DeprecationWarning for os.fork when run in threads with -Werror #136796
Conversation
|
All commit authors signed the Contributor License Agreement. |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
|
I have created a NEWS entry for this fix, but actually this change has indeed a little impact on Python users, so maybe |
…nto fix-issue-135427
|
Many tests are failing with a warning - as now the Many of the failing tests do not fail where one would expect to have more than one thread running. For example, in When printing there the number of threads using Therefore, we should find a way to suppress the warning of running a fork within a thread in all the tests that fail. Petr suggests to consider using the If anyone knows where this extra thread comes from, it will be appreciated if this is explained :-) |
That is ironically exactly why we have this warning - threads in a process can come from everywhere including external libraries outside of our control. I do think you're on the right track in terms of how to enable this to be something using |
…ead. Started to fix tests.
| @@ -1182,6 +1182,7 @@ async def runner(): | |||
| @support.requires_fork() | |||
| class TestFork(unittest.IsolatedAsyncioTestCase): | |||
|
|
|||
| @warnings_helper.ignore_warnings(category=DeprecationWarning) # gh-135427 | |||
There was a problem hiding this comment.
The whole class has fork tests, maybe add decorator to class itself?
There was a problem hiding this comment.
Yes. I am still learning... I tried to avoid putting decorator to the classes themselves, as I thought that it is more correct that if someone adds a new test, that developer should see the depracation warning.
But what you suggest will save many lines of code.
And also it seems that in some cases the test may run with forks or without them, so I am not sure that it is correct to decorate the functions themselves. For example, ProcessPoolExecutorTest extends ExecutorTest but also ThreadPoolExecutorTest extends ExecutorTest. So the tests in ExecutorTest must be decorated only if it is extended by ProcessPoolExecutorTest.
| @@ -1182,6 +1182,7 @@ async def runner(): | |||
| @support.requires_fork() | |||
| class TestFork(unittest.IsolatedAsyncioTestCase): | |||
|
|
|||
| @warnings_helper.ignore_warnings(category=DeprecationWarning) # gh-135427 | |||
There was a problem hiding this comment.
I still consider to create a dedicated ignore_fork_deprecated_warnings() decorator which will use
warnings.filterwarnings('ignore',
message=".*fork.*may lead to deadlocks in the child.*",
category=DeprecationWarning)This will prevent that in the future all the ignore_warning on DeprecatedWarning will not affect other deprecated warnings.
…ific decorator to ignore only the fork in thread deprecation warnings
|
@encukou, @gpshead - I am not sure, but maybe we should indeed create two PRs. The one that is here for I think that the differences between the two PRs will only be the change in What do you think? |
Misc/NEWS.d/next/Library/2025-07-19-11-53-19.gh-issue-135427.iJM_X2.rst
Outdated
Show resolved
Hide resolved
Lib/test/support/warnings_helper.py
Outdated
| return decorator | ||
|
|
||
|
|
||
| def ignore_fork_in_thread_deprecation_warnings(test): |
There was a problem hiding this comment.
That's quite a bit of code added. I tried simplifying it:
- Use
contextlib.contextmanagerforignore_warnings. It turns out that context managers created this way can also be used as decorators! (I learned that myself quite recently) - Use
filterwarningsinstead ofsimplefilterinignore_warnings, and add amessageargument - Then use
contextlib.contextmanagerforignore_fork_in_thread_deprecation_warningstoo. Sadly this means adding()in all the call sites... but a context manager can also simplify thewith warnings.catch_warnings(): warnings.filterwarnings(...) - Don't worry about
asyncfunctions -- there's only one of those as far as I can see, and theforkin it can be put in a context manager
@contextlib.contextmanager
def ignore_warnings(*, category, message=''):
"""Decorator to suppress warnings.
Can also be used as a context manager. This is not preferred,
because it makes diffs more noisy and tools like 'git blame' less useful.
But, it's useful for async functions.
"""
with warnings.catch_warnings():
warnings.filterwarnings('ignore', category=category, message='')
yield
@contextlib.contextmanager
def ignore_fork_in_thread_deprecation_warnings():
"""Suppress deprecation warnings related to forking in multi-threaded code.
See gh-135427
Can be used as decorator (preferred) or context manager.
"""
with ignore_warnings(
message=".*fork.*may lead to deadlocks in the child.*",
category=DeprecationWarning,
):
yieldThen adjust all the decorator calls to add (). And maybe remove the issue reference -- it turned out to be quite noisy; it should be enough if the function references it, in one place.
- @warnings_helper.ignore_fork_in_thread_deprecation_warnings # gh-135427
+ @warnings_helper.ignore_fork_in_thread_deprecation_warnings()|
I think this should go in 3.15 only, and go through the full testing period. |
Co-authored-by: Petr Viktorin <encukou@gmail.com>
…JM_X2.rst Co-authored-by: Petr Viktorin <encukou@gmail.com>
This is a small fix as suggested in #135427 - using
PyErr_WriteUnraisable()instead ofPyErr_Clear().The fix includes a test which checks that the
DeprecationWarningis provided when usingforkorforkptywithin a thread, also when running with-Werror.@encukou