-
-
Notifications
You must be signed in to change notification settings - Fork 32.5k
gh-137026: Add an explainer guide for asyncio #137215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will review further once the lines are wrapped, as now it will invalidate suggestions.
- Start sentences on new lines to minimize disruption of diffs.
…Which concurrency do I want" section.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm going to commit to reviewing this if nobody else has the time. I think is generally an improvement to the documentation, but definitely needs some work still. Thanks for working on this!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are still many minor infringements of the Style guide (e.g. simple English) which could be addressed in one pass.
You can also use Sphinx cross references when discussing specific keywords, functions etc.
|
||
.. seealso:: | ||
|
||
The `guide <https://github.com/anordin95/a-conceptual-overview-of-asyncio/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's also add two more references:
- import asyncio: Learn Python's AsyncIO Rename README to README.rst and enhance formatting #2 - The Event Loop presented by core team member, Łukasz Langa https://www.youtube.com/watch?v=E7Yn5biBZ58
- 500 Lines or Less: A Web Crawler With asyncio Coroutines by A. Jesse Jiryu Davis and Guido van Rossum https://aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done 👍
Since your suggestions include the author names, it felt odd not to have it for the guide, so I added my name in. Happy to also drop all the names too! Just looks a bit odd if it's not consistent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Names for all works fine to me.
A task also maintains a list of callback functions whose importance will become | ||
clear in a moment when we discuss ``await``. | ||
The recommended way to create tasks is via :func:`asyncio.create_task`. | ||
Creating a task automatically adds it to the event loop's queue of tasks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@anordin95 Yes, I understand that you find the concept of a queue a simplified mental model for an event loop. But, in practice, a queue overly simplifies the purpose of an event loop as it leaves out the important concept of an event loop, selecing ready coroutines. An event loop manages which coroutines are "ready" to run and schedules a "ready" coroutine to run. A queue would sequentially run items.
Instead, the event loop at its most fundamental is a "while" loop that is responsible for managing and scheduling coroutines. asyncio's event loop abstracts away the low-level implementation. It's essentially the "traffic director" that selects which "ready" callback of all "scheduled and ready callbacks" should run.
While it is true that asyncio can execute tasks in a predictable order, it does so based on selecting among "ready" callbacks.
I've added suggestions for 2 additional see also resources (one is a video) that further explain.
Hi @ambv, If you have the time to review this PR, I think your perspective would be helpful. You did a great job on the 8-part asyncio youtube playlist on clearly explaining an event loop. I'm hoping your fresh eyes will help us find wording that strikes the right tone for explanation while also being technically accurate. |
- Discuss why await was designed to not yield for coros.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me. I've left a few grammar fixes as suggestions, but I'm comfortable enough with this to put the first approval on here.
I haven't read too deep into some of the technical discussions about asyncio internals, but I'm personally fine with keeping terms like "queue" in here. I really think we should focus on creating a HOWTO that's clear and easy to understand for beginners, not something that documents the exact inner workings of asyncio in detail.
This is based on the concept of lies-to-children ("a simplified, and often technically incorrect, explanation of technical or complex subjects employed as a teaching method"). This tutorial is targeted towards beginners, so it should be totally OK to use some information that's technically incorrect in the pursuit of initial comprehension.
For example, most schools initially teach you the concept of planetary electron shells, despite the fact that there are more correct (and complicated) models in quantum mechanics. We should not teach the quantum mechanics of asyncio here. Instead, our goal should be to spark curiosity in readers, even if the information is not as technically correct.
Thanks for all your hard work @anordin95!
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Thanks @ZeroIntensity and @anordin95. HOWTO guides are not strictly targeted to beginners rather they are a deeper explanation of something. The name HOWTO comes from early Linux days as a deep dive into a particular area. OK, we are down to one key decision: the best mental model for an event loop. We've been discussing a) the queue and b) the execution of scheduled work.
From the event loop docs, the above defines work run by an event loop. I'm going to suggest that instead of sticking with "queue" which doesn't sit well with multiple asyncio maintainers that we explore some other terms: "to do list", "work checklist", "collection of work". Working from the conductor metaphor: Like a symphony conductor, the event loop orchestrates multiple performers (tasks). It knows when each section should play, pauses some while others perform, and coordinates the overall harmony of concurrent operations. |
This is a beautiful idea! But, I'm a bit wary of using it, since in a symphony many sections play truly concurrently. The trumpets, the drums & the flutes could all be going at once. I reworked the event loop paragraph a decent bit (see below). The idea of a queue is mentioned only once now and is explicitly called out as a rough analogy twice in the sentence . Otherwise, I refer to the event loop's jobs with a variety of similar but different terms to help reinforce and contextualize the idea. Alongside this change, my idea also entails removing any other references to a queue in the broader article. In more technical terms, the event loop contains a collection of jobs to be run. Some jobs are added directly by you, and some indirectly by |
Cool @anordin95! I like the last commit you pushed. 🎉 Can we continue throughout the doc to deemphasize queue? |
Yep! Was just taking a little break. All references to queue besides the one in that first analogy are now gone. |
@ZeroIntensity, @willingc: It should be noted, I just added a blurb about cpython/Doc/howto/a-conceptual-overview-of-asyncio.rst Lines 207 to 228 in 9d3828d
|
Explainer guide for asyncio
gh-137026: HOWTO article for asyncio, and a reference to it from the main page of the asyncio docs.
Hi!
I've used Python's
asyncio
a couple times now, but never really felt confident in my mental model of how it fundamentally works and therefore how I can best leverage it. The official docs provide good documentation for each specific function in the package, but, in my opinion, are missing a cohesive overview of the systems design and architecture. Something that could help the user understand the why and how behind the recommended patterns. And a way to help the user make informed decisions about which tool in the asyncio toolkit they ought to grab, or to recognize when asyncio is the entirely wrong toolkit. I thought I'd take a stab at filling that gap and contributing back to a community that's given so much!📚 Documentation preview 📚: https://cpython-previews--137215.org.readthedocs.build/en/137215/howto/a-conceptual-overview-of-asyncio.html