py/objtype: Add support for PEP487 __set_name__. #16806
Conversation
|
Code size report: |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #16806 +/- ##
==========================================
- Coverage 98.41% 98.39% -0.02%
==========================================
Files 171 171
Lines 22210 22228 +18
==========================================
+ Hits 21857 21871 +14
- Misses 353 357 +4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
5d73521 to
c6bd944
Compare
|
I've done some benchmarking using a new suite of internalbench class creation benchmarks (see PR #16825) and have compared the benchmark times from the base branch to this branch. There's a lot of noise in this (I'll see if I can run these on real hardware at some point), but overall this patch makes processing classes take about 32% longer: There were also three other benchmark tests that got concerningly slower: None of the other tests in their families seemed to be affected, but my tests for #15503 and #16816 both showed |
7a169aa to
0dc3ccb
Compare
|
Ended up switching out |
d710e5a to
b69bfbd
Compare
|
Also, since it's only binding rather than calling
|
c0934f9 to
96e0b67
Compare
|
Spurious coverage change for 8179697, along with three different spurious failures on that same I've also manually tested the path gated behind |
|
Would be good to see the change in benchmark results with the latest version here. |
|
I ran an updated version of the benchmarks in #16825 on my Pico2 board to compare the performance of master vs this PR with
Raw Test Results(Note, these tests runs were terminated at the start of the master: Feature Not ImplementedRun from v1.25.0-390-g270b00215 on the rp2 port, on my overclocked Pico2. is_fixed:
|
|
Coverage change for 03b37bd is spurious. |
|
Thanks for running the benchmarks, that's quite comprehensive and certainly helps to evaluate this PR. (Note that I think you have the sign the wrong way around in the final column "vs is_fixed".)
So, both of those are faster with the list version, compared to is-fixed. That makes sense to me because the list version only needs to iterate once through the locals dict, whereas the is-fixed version iterates twice.
Yes, that's good. And it's worth explicitly pointing out that this is about class creation, not instance (of a class) creation. The former is done once, the latter many times. So making class creation a little bit slower in order to add this feature is acceptable (and it won't affect instance creation performance).
Yes, indeed. I tested esp8266 and the list version is only +40 bytes. And on PYBV10 it's only +24 bytes. And from a quick static code analysis, I can't fully rule out that My conclusion: I think we should go with this approach to |
Indeed, fixed it.
Not a hazard I realized might be at play.
I was surprised at how close the two are too; but yeah I agree, since they're so close we might as well leave out the less-functional version. |
|
This is looks really good now. I will do a final review, and might have a little play to see if there is any way to reduce code size. |
Including the stochastic tests needed to guarantee sensitivity to the potential iterate-while-modifying hazard a naive implementation might have. Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in `setname_list_t`, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
One thought: are there any allocator modes that would make it safe to unconditionally call It might also be ok to just omit the call to |
Summary
This PR adds support for the
__set_name__data model method. This includes support for methods that mutate the owner class, and avoids the modify-while-iterating hazard on the class locals dictionary encountered by a more naive implementation like #15503. It's also able to do this faster than the naive implementation, thanks to clever data layout in the linked list it uses and the way this approach allows the capture step to run during an existing step that already iterates the class dict.Testing
This feature was developed test-first with comprehensive tests of the feature (originally cpydiffs proposed in #15500). It also includes additional tests that were needed to guarantee test sensitivity to the modify-while-iterating hazard.
I've run the test suite against both Unix and RP2 targets. I've verified that the tests are sensitive to the feature's presence, and that this implementation passes the full test suite.
Trade-offs and Alternatives
Performing
__set_name__in a way that allows both owner-class mutation and avoids the modify-while-iterating hazard, requires allocating some kind of additional memory for a structure that can be iterated; but within that there are several ways this can be done.During the implementation of this, it was discovered that some versions of CPython also have this naieve modify-while-iterating bug, and this issue was raised with them. This was resolved in bpo-28797; their approach is now to make a complete flat copy of the class's dict and iterate on this copy. This is a decision that doesn't make very much sense on a microcontroller, but makes all the sense on a desktop platform where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and in their case it's also motivated by error-tracing considerations.
This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here.
The simplest implementation is to just not worry about it and just let the user face the consequences if they mutate the owner class. It's not a very friendly behavior, though, and it's not actually much more performant than this PR on time or code size.
Another alternative is to do the same as py/objtype: Add support for __set_name__. (hazard version) #15503 but leverage Micropython's existing
is_fixedfield in its dict type to convert attempted mutations of the owner dict intoAttributeErrors. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance gain is small enough that it isn't worth missing support for those cases.Also considered was combining #17693 and this PR, using a new feature define
MICROPY_PY_METACLASSES_LITEto control whether this PR's algorithm or the naive version that blocks owner mutations is used. But, based on the very limited performance margin the naive version has (which in some cases even goes against it), it was determined to remove that and only support a full implementation.