py/objtype: Add support for __set_name__. (dict-copy version) #16816
Conversation
f48eadc to
14fa945
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #16816 +/- ##
=======================================
Coverage 98.44% 98.44%
=======================================
Files 171 171
Lines 22192 22223 +31
=======================================
+ Hits 21847 21878 +31
Misses 345 345 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Code size report: |
ba1d031 to
003d4ff
Compare
|
I've done some benchmarking using a new suite of internalbench class creation benchmarks (see PR #16825) and have compared the benchmark times from the base branch to this branch. There's a lot of noise in this (I'll see if I can run these on real hardware at some point), but overall this patch makes processing classes take about 30% longer -- though with the amount of noise it's a coin flip whether the code was actually faster than #16806 or just luckier. There were also two other benchmark tests that got concerningly slower: None of the other tests in their families seemed to be affected, but my tests for #15503 and #16806 both showed |
003d4ff to
1c0b2ff
Compare
|
Just updated this to instead make its own copy of @dpgeorge Does anything extra need to be done to guarantee that objects referenced in the table I'm allocating on the stack can't be garbage collected? The If a descriptor's Outside of this The only thing kinda close is in |
|
Actually, for that matter... if what I'm worrying about is the case, the class being initialized could itself end up being collected before its returned, even without this patch. The only pointers to the class object until EDIT: After some experimentation I've determined that this (the class weirdness at least) is at least, much harder to trigger than I originally imagined, if it's possible; but I did uncover another very weird behavioral difference. On master: class A:
def __get__(self, instance, owner=None):
print("A")
class B:
__get__ = A()
print("C")
class C:
x = B()(I have no idea how it is that |
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
02d9895 to
6720e77
Compare
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
|
This is my least favourite of the implementations. It's stack heavy and unconditionally allocates. I think #16806 could be made to be simpler than this version. |
|
Closed in favor of #16806. |
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in setname_list_t, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in `setname_list_t`, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in `setname_list_t`, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Summary
This implements the alternative "flat dict copy" logic discussed in #16806 as another way to eliminate the modify-while-iterating hazard in the original
__set_name__implementation (#15503).Testing
This feature was developed test-first. All automated tests pass, including additional tests added to verify the additional behavior around whether
__set_name__is called even on descriptors that another__set_name__removes beforehand.Trade-offs and Alternatives
This version has significantly lower code complexity than #16806, but at the cost of a larger runtime memory footprint. It's still a bit more complex than #15503, though.