-
-
Notifications
You must be signed in to change notification settings - Fork 8.3k
py/objtype: Add support for __set_name__. (no-self-modification version) #17693
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Eliminate the potential iterate-while-modifying hazard in the calls to user __set_name__ functions by marking the locals_dict as fixed during the __set_name__ loop. Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #17693 +/- ##
=======================================
Coverage 98.44% 98.44%
=======================================
Files 171 171
Lines 22192 22219 +27
=======================================
+ Hits 21847 21874 +27
Misses 345 345 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Code size report:
|
I like this approach, it's nice and simple, and safe.
Do you think we'd be severely limiting this feature by disallowing locals dict updates? |
The use cases aren't overwhelming, so there's definitely merit to just disallowing locals dict updates; or maybe having a feature flag for this. Just to describe a few, though: Some descriptors need to coordinate with other descriptor instances on the same class (e.g. a serialization or ORM libraries); in these cases, it's common to have the first descriptor create an attribute on the class to hold a data structure that they all register in. (One of my own libraries use owner class attrs like this for the tables it uses to work out what registers are contiguous, for I2C transaction pooling.) class Registered:
def __set_name__(self, owner, name):
try:
owner.__registry[name] = self
except AttributeError:
owner.__registry = {name: self} In CPython, a library writer can also do this using a mixin class, with an It's also potentially a performance optimization in the case where a descriptor can replace itself with a concrete value at class-creation time, to avoid the indirection of dispatching access through a getter: class Configured:
def __set_name__(self, owner, name):
concrete_value = query_config_database(name)
setattr(owner, name, concrete_value)
class MyApp:
TIME_BETWEEN_WIDGETS: int = Configured() |
Combined into #16806. |
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in setname_list_t, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in `setname_list_t`, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Summary
This PR implements the feature described in #15501, adding support for the
__set_name__
data model method.In particular, this implementation leverages the existing
is_fixed
flag bit on micropython's flag type to avoid the potential modify-while-iterating hazard in the simplest possible way, i.e. by preventing__set_name__
from modifying the owner class.Testing
This PR includes and passes the unit test originally submitted in #15500 to verify the feature's absence, as well as additional tests to specifically verify that attempting to mutate the descriptor's owner triggers an error.
Trade-offs and Alternatives
This is the least powerful of the four ways I've tried implementing this (just allowing the hazard in #15503, or copying a list of calls to make in #16806, or making a private copy of the dict in #16816), as the ability for a descriptor to add siblings or transparently delete or replace itself isn't an especially-rarely-used functionality in descriptor-heavy code.
The functionality in question can, though, for the most part be replaced by inheriting the owner class from a suitable parent class with an
__init_subclass__
method that handles this on behalf of its descriptors. This will need to be documented as a cpydiff workaround, though that can only really be considered a workaround once we have actual__init_subclass__
(#15511) or at least documented workarounds to that method's nonfunctionality (#16786).As an alternative, maybe this PR could be combined with either #16806 or #16816 with a feature flag that defaults to this minimal 'disallow modifying the owner' implementation, but can be set for platforms/applications where the full cost of making the extra copies is tolerable?