Skip to content

gh-64192: Make imap()/imap_unordered() in multiprocessing.pool actually lazy #136871

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open

gh-64192: Make imap()/imap_unordered() in multiprocessing.pool actually lazy #136871

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
d8e8a02
draft: impl lazy input consumption in mp.Pool.imap(_unordered)
obaltian Jul 20, 2025
f6f423c
Use semaphore to synchronize threads
obaltian Jul 20, 2025
937862d
Update buffersize behavior to match concurrent.futures.Executor behavior
obaltian Jul 21, 2025
b6f6caa
Release all `buffersize_lock` obj from the parent thread when terminate
obaltian Jul 21, 2025
3bafd5d
Add 2 basic `ThreadPool.imap()` tests w/ and w/o buffersize
obaltian Jul 21, 2025
e43232b
Fix accidental swap in imports
obaltian Jul 21, 2025
dd416e0
clear Pool._taskqueue_buffersize_semaphores safely
obaltian Jul 21, 2025
99f5a8c
Slightly optimize Pool._taskqueue_buffersize_semaphores terminate
obaltian Jul 21, 2025
2a53398
Rename `Pool.imap()` buffersize-related tests
obaltian Jul 21, 2025
f8878eb
Fix typo in `IMapIterator.__init__()`
obaltian Jul 22, 2025
2ca51e3
Add tests for buffersize combinations with other kwargs
obaltian Jul 22, 2025
bf27d5d
Remove if-branch in `_terminate_pool`
obaltian Jul 27, 2025
508c765
Add more edge-case tests for `imap` and `imap_unodered`
obaltian Jul 27, 2025
dff1167
Split inf iterable test for `imap` and `imap_unordered`
obaltian Jul 27, 2025
94cc0b9
Add doc for `buffersize` argument of `imap` and `imap_unordered`
obaltian Jul 27, 2025
816fb6c
add *versionadded* for `imap_unordered`
obaltian Jul 28, 2025
88cc10a
Remove ambiguity in `buffersize` description.
obaltian Jul 28, 2025
05e3b24
Set *versionadded* as next in docs
obaltian Jul 28, 2025
503982f
Add whatsnew entry
obaltian Jul 28, 2025
b92cad9
Fix aggreed comments on code formatting/minor refactoring
obaltian Jul 28, 2025
02ebc6a
Remove `imap` and `imap_unordered` body code duplication
obaltian Jul 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Slightly optimize Pool._taskqueue_buffersize_semaphores terminate
  • Loading branch information
@obaltian
obaltian committed Jul 27, 2025
commit 99f5a8c74c899c1f4a859598966780c9875d246a
7 changes: 4 additions & 3 deletions Lib/multiprocessing/pool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -726,9 +726,10 @@ def _terminate_pool(cls, taskqueue, inqueue, outqueue, pool, change_notifier,

task_handler._state = TERMINATE
# Release all semaphores to wake up task_handler to stop.
for job_id, sema in tuple(taskqueue_buffersize_semaphores.items()):
taskqueue_buffersize_semaphores.pop(job_id, None)
sema.release()
for job_id in tuple(taskqueue_buffersize_semaphores.keys()):
sema = taskqueue_buffersize_semaphores.pop(job_id, None)
if sema is not None:
sema.release()

util.debug('helping task handler/workers to finish')
cls._help_stuff_finish(inqueue, task_handler, len(pool))
Expand Down
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy