-
-
Notifications
You must be signed in to change notification settings - Fork 32.5k
gh-137026: Add an explainer guide for asyncio #137215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will review further once the lines are wrapped, as now it will invalidate suggestions.
- Start sentences on new lines to minimize disruption of diffs.
…Which concurrency do I want" section.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm going to commit to reviewing this if nobody else has the time. I think is generally an improvement to the documentation, but definitely needs some work still. Thanks for working on this!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are still many minor infringements of the Style guide (e.g. simple English) which could be addressed in one pass.
You can also use Sphinx cross references when discussing specific keywords, functions etc.
Yep! Was just taking a little break. All references to queue besides the one in that first analogy are now gone. |
@ZeroIntensity, @willingc: It should be noted, I just added a blurb about cpython/Doc/howto/a-conceptual-overview-of-asyncio.rst Lines 207 to 228 in 9d3828d
|
That example highlights how using only ``await coroutine`` could | ||
unintentionally hog control from other tasks and effectively stall the event | ||
loop. | ||
:func:`asyncio.run` can help you detect such occurences with the ``debug=True`` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To clarify, the way I'd expect most developers to use this is not by modifying the code, but by running python in dev mode locally (-X dev
or PYTHONDEVMODE=1
). A link should take users to: https://docs.python.org/3/library/asyncio-dev.html#asyncio-debug-mode
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added the link 👍
Hm. I agree but that's effectively a shortcut. Pedagogically, I think it's clearer to mention the debug=True
flag.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm. I agree but that's effectively a shortcut. Pedagogically, I think it's clearer to mention the
debug=True
flag.
If you say so. I didn't even know that parameter existed and can't imagine ever using it. All I need to know as a user is that this feature exists and is enabled as part of dev mode..
Once it pauses or completes, it returns control to the event loop. | ||
The event loop will then select another job from its pool and invoke it. | ||
You can *roughly* think of the collection of jobs as a queue: jobs are added and | ||
then processed one at a time, generally (but not always) in order. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
generally (but not always) in order.
I still find this bit awkward as it's again suggesting that order of execution is unpredictable. Can we say that the jobs are processed in the order that they were scheduled or something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm. If there's phrasing you have in mind, that the folks who are wary of the queue analogy would also be on board with, I'm open to it!
Because the coroutine ``main()`` exits before awaiting the task and no other | ||
references to the task are made, the task object ``hello_task`` *might* be |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This may not be the best example, as the application is expected to close at the end of main(). So the task gets cancelled anyway. i.e. Even if I keep a reference, I can get the task to not complete:
>>> async def hello():
... await asyncio.sleep(1)
... print("BAR")
...
>>> async def main():
... global t
... t = asyncio.create_task(hello())
...
>>> asyncio.run(main())
>>> t
<Task cancelled name='Task-10' coro=<hello() done, defined at <stdin>:1>>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you did:
async def hello():
await asyncio.sleep(1)
print("hello!")
async def main():
asyncio.create_task(hello())
await asyncio.sleep(2)
Or similar, then this probably better shows the risk. This works in practice today, but is not guaranteed to work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Neat first example!
Hm. I don't see how the second better illustrates the risk. The risk that the current example is meant to emphasize is the problem with losing references (before await
ing). As opposed to the program exiting before all registered (but unawait
ed) tasks finish.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The risk that the current example is meant to emphasize is the problem with losing references (before
await
ing).
My point is that it doesn't show that, because main() exits regardless and the other task is not expected to continue running. As I showed above, it doesn't run past the initial iteration if we do hold references to it, nevermind if we don't.
The second example has a sleep so that main() outlives the task. So, in this case the lack of a reference could cause an issue. With a reference, the task is meant to run until completion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, yeah it doesn't show the issue of GC killing off references. But none of these examples shown so far do. It's somewhat difficult to create a simple, real example. #117379
Ah, I think I see what you're saying of how folks could confuse the termination of main()
with the danger of GC'd tasks. I've reworked a decent bit to clarify.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, that looks better to me.
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
-------------------------------------------- | ||
A conceptual overview part 1: the high-level | ||
-------------------------------------------- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes little sense to me, since the title of the second part does not say it is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, good catch. I tweaked the latter section title to include part 2.
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Explainer guide for asyncio
gh-137026: HOWTO article for asyncio, and a reference to it from the main page of the asyncio docs.
Hi!
I've used Python's
asyncio
a couple times now, but never really felt confident in my mental model of how it fundamentally works and therefore how I can best leverage it. The official docs provide good documentation for each specific function in the package, but, in my opinion, are missing a cohesive overview of the systems design and architecture. Something that could help the user understand the why and how behind the recommended patterns. And a way to help the user make informed decisions about which tool in the asyncio toolkit they ought to grab, or to recognize when asyncio is the entirely wrong toolkit. I thought I'd take a stab at filling that gap and contributing back to a community that's given so much!📚 Documentation preview 📚: https://cpython-previews--137215.org.readthedocs.build/en/137215/howto/a-conceptual-overview-of-asyncio.html