py/objtype: Add support for __set_name__. (no-self-modification version) #17693
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Eliminate the potential iterate-while-modifying hazard in the calls to user __set_name__ functions by marking the locals_dict as fixed during the __set_name__ loop. Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #17693 +/- ##
=======================================
Coverage 98.44% 98.44%
=======================================
Files 171 171
Lines 22192 22219 +27
=======================================
+ Hits 21847 21874 +27
Misses 345 345 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Code size report: |
|
I like this approach, it's nice and simple, and safe.
Do you think we'd be severely limiting this feature by disallowing locals dict updates? |
The use cases aren't overwhelming, so there's definitely merit to just disallowing locals dict updates; or maybe having a feature flag for this. Just to describe a few, though: Some descriptors need to coordinate with other descriptor instances on the same class (e.g. a serialization or ORM libraries); in these cases, it's common to have the first descriptor create an attribute on the class to hold a data structure that they all register in. (One of my own libraries use owner class attrs like this for the tables it uses to work out what registers are contiguous, for I2C transaction pooling.) class Registered:
def __set_name__(self, owner, name):
try:
owner.__registry[name] = self
except AttributeError:
owner.__registry = {name: self}In CPython, a library writer can also do this using a mixin class, with an It's also potentially a performance optimization in the case where a descriptor can replace itself with a concrete value at class-creation time, to avoid the indirection of dispatching access through a getter: class Configured:
def __set_name__(self, owner, name):
concrete_value = query_config_database(name)
setattr(owner, name, concrete_value)
class MyApp:
TIME_BETWEEN_WIDGETS: int = Configured() |
|
Combined into #16806. |
AJMansfield
added a commit
to AJMansfield/micropython
that referenced
this pull request
Jul 21, 2025
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in setname_list_t, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
AJMansfield
added a commit
to AJMansfield/micropython
that referenced
this pull request
Jul 21, 2025
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in `setname_list_t`, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
AJMansfield
added a commit
to AJMansfield/micropython
that referenced
this pull request
Jul 23, 2025
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in `setname_list_t`, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements the feature described in #15501, adding support for the
__set_name__data model method.In particular, this implementation leverages the existing
is_fixedflag bit on micropython's flag type to avoid the potential modify-while-iterating hazard in the simplest possible way, i.e. by preventing__set_name__from modifying the owner class.Testing
This PR includes and passes the unit test originally submitted in #15500 to verify the feature's absence, as well as additional tests to specifically verify that attempting to mutate the descriptor's owner triggers an error.
Trade-offs and Alternatives
This is the least powerful of the four ways I've tried implementing this (just allowing the hazard in #15503, or copying a list of calls to make in #16806, or making a private copy of the dict in #16816), as the ability for a descriptor to add siblings or transparently delete or replace itself isn't an especially-rarely-used functionality in descriptor-heavy code.
The functionality in question can, though, for the most part be replaced by inheriting the owner class from a suitable parent class with an
__init_subclass__method that handles this on behalf of its descriptors. This will need to be documented as a cpydiff workaround, though that can only really be considered a workaround once we have actual__init_subclass__(#15511) or at least documented workarounds to that method's nonfunctionality (#16786).As an alternative, maybe this PR could be combined with either #16806 or #16816 with a feature flag that defaults to this minimal 'disallow modifying the owner' implementation, but can be set for platforms/applications where the full cost of making the extra copies is tolerable?