Skip to content

py/objtype: Add support for __set_name__. (list version) #16806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

py/objtype: Add support for __set_name__. (list version) #16806

wants to merge 7 commits into from
+429 −21

Conversation

AJMansfield
Copy link
Contributor

@AJMansfield AJMansfield commented Feb 24, 2025 •
edited
Loading

Summary

This modifies #15503 to add the additional logic required to eliminate the modify-while-iterating hazard in the original implementation.

Testing

This feature was developed test-first as a cpydiff. All automated tests pass, including the original cpydiff that this patch converts into a mandatory test.

Trade-offs and Alternatives

Performing __set_name__ in a hazard-free way requires about double the code complexity, including the need to allocate temporary storage to capture a frozen list of __set_name__ members into that can be iterated safely while calls into user code potentially mutate the class.

The way the list is frozen currently involves generating a list of the names that the code is able to successfully do a __set_name__ method lookup on, and then looping over the list looking up and calling those names. Potentially, it would be possible to skip that second name lookup if we store both the name and the method binding that was retrieved in the first step. This alternate approach would need three times the memory in its list allocation. It would also change one very slight detail about the observable behavior, for whether __set_name__ still gets called even if a descriptor was removed by another descriptor before we reached it in the iteration order (currently: no, cpython: yes).

Alternatively, we could do a complete flat memcpy on locals_dict and iterate that copy. This seems to be the approach that CPython takes in their implementation (see python/cpython#72983). (Copying bulk memory is cheap there, though, and conditionals expensive, while on microcontrollers it's usually the opposite situation.)

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 24, 2025 •
edited
Loading

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:  +232 +0.027% standard[incl +32(data)]
      stm32:  +132 +0.034% PYBV10
     mimxrt:  +136 +0.036% TEENSY40
        rp2:  +136 +0.015% RPI_PICO_W
       samd:  +152 +0.057% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:  +145 +0.032% VIRT_RV32

@codecov Codecov
Copy link

codecov bot commented Feb 24, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.42%. Comparing base (df05cae) to head (8179697).
Report is 22 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master   #16806      +/-   ##
==========================================
- Coverage   98.44%   98.42%   -0.02%     
==========================================
  Files         171      171              
  Lines       22192    22226      +34     
==========================================
+ Hits        21847    21877      +30     
- Misses        345      349       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@AJMansfield AJMansfield force-pushed the set-name-2 branch 4 times, most recently from 5d73521 to c6bd944 Compare February 24, 2025 22:36
@AJMansfield AJMansfield mentioned this pull request Feb 25, 2025
Open
@AJMansfield AJMansfield changed the title py/objtype: Add hazard-free support for __set_name__. py/objtype: Add hazard-free support for __set_name__. (list version) Feb 25, 2025
@AJMansfield AJMansfield changed the title py/objtype: Add hazard-free support for __set_name__. (list version) py/objtype: Add support for __set_name__. (list version) Feb 26, 2025
This was referenced Feb 26, 2025
Open
Open
@AJMansfield
Copy link
Contributor Author

AJMansfield commented Feb 26, 2025 •
edited
Loading

I've done some benchmarking using a new suite of internalbench class creation benchmarks (see PR #16825) and have compared the benchmark times from the base branch to this branch.

There's a lot of noise in this (I'll see if I can run these on real hardware at some point), but overall this patch makes processing classes take about 32% longer:

internal_bench/class_create:
    0.349 -> 0.450 (+29%) internal_bench/class_create-0-empty.py
    0.476 -> 0.630 (+33%) internal_bench/class_create-1-slots.py
    0.490 -> 0.622 (+27%) internal_bench/class_create-1.1-slots5.py
    0.432 -> 0.551 (+28%) internal_bench/class_create-2-classattr.py
    0.776 -> 0.970 (+25%) internal_bench/class_create-2.1-classattr5.py
    0.461 -> 0.594 (+29%) internal_bench/class_create-3-instancemethod.py
    0.477 -> 0.634 (+33%) internal_bench/class_create-4-classmethod.py
    0.452 -> 0.619 (+37%) internal_bench/class_create-4.1-classmethod_implicit.py
    0.494 -> 0.651 (+32%) internal_bench/class_create-5-staticmethod.py
    0.458 -> 0.599 (+31%) internal_bench/class_create-6-getattribute.py
    0.474 -> 0.572 (+21%) internal_bench/class_create-6.1-getattr.py
    0.386 -> 0.518 (+34%) internal_bench/class_create-6.2-descriptor.py
    0.517 -> 0.722 (+39%) internal_bench/class_create-6.3-descriptor_setname.py
    0.419 -> 0.573 (+37%) internal_bench/class_create-6.4-property.py
    0.363 -> 0.477 (+31%) internal_bench/class_create-7-inherit.py
    0.368 -> 0.491 (+33%) internal_bench/class_create-7.1-inherit_initsubclass.py

There were also three other benchmark tests that got concerningly slower:

internal_bench/arrayop:
    0.174 -> 0.196 (+13%) internal_bench/arrayop-3-bytearray_inplace.py
internal_bench/func_builtin:
    0.209 -> 0.232 (+11%) internal_bench/func_builtin-2-enum_kw.py
internal_bench/loop_count:
    0.268 -> 0.340 (+27%) internal_bench/loop_count-2-range_iter.py

None of the other tests in their families seemed to be affected, but my tests for #15503 and #16816 both showed loop_count-2 slowed by near-enough the exact same margins as class creation.

@dpgeorge dpgeorge added the py-core Relates to py/ directory in source label Mar 13, 2025
@AJMansfield
tests/basics/class_descriptor: Test for __set_name__.
3173fa1 
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
@AJMansfield
py/objtype: Implement __set_name__.
5da9c94 
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
@AJMansfield AJMansfield force-pushed the set-name-2 branch 3 times, most recently from 7a169aa to 0dc3ccb Compare July 16, 2025 17:41
@AJMansfield AJMansfield mentioned this pull request Jul 16, 2025
Open
dpgeorge
dpgeorge reviewed Jul 18, 2025
View reviewed changes
@AJMansfield
Copy link
Contributor Author

Ended up switching out mp_obj_list_t for a bespoke setname_list_t linked list structure, in order to be able to capture all of the per-attr values needed for each __set_name__ call into a struct with the same memory layout as the eventual call list. (Using the position for the owner argument for the linked list pointer.)

@AJMansfield AJMansfield force-pushed the set-name-2 branch 2 times, most recently from d710e5a to b69bfbd Compare July 18, 2025 17:57
@AJMansfield
Copy link
Contributor Author

AJMansfield commented Jul 18, 2025 •
edited
Loading

Also, since it's only binding rather than calling __set_name__ during the locals dict loop, it was possible to combine that with the loop that does the special accessors check, per #15503 (comment)

There's already check_for_special_accessors() that's run on class creation in a loop over all members. Maybe that can be reused instead of essentially duplicating the work?

@AJMansfield
py/mpconfig: Add METACLASSES_LITE feature flag for PEP847.
ebcea98 
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
@AJMansfield
tests/basics/class_setname_hazard: Test for __set_name__ hazard.
69e622f 
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
@AJMansfield
py/objtype: Eliminate __set_name__ hazard.
09c4ee3 
Captures all __set_name__ calls with their per-attr args into a linked
list during the special accessors check, to iterate safely later.

Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
@AJMansfield
py/objtype: Rework special accessor check's early-termination.
96e0b67 
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
@AJMansfield
py/objtype: Added alternate no-mutate __set_name__ for descriptor use.
8179697 
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
@AJMansfield
Copy link
Contributor Author

Spurious coverage change for 8179697, along with three different spurious failures on that same extmod/select_poll_eintr.py test.

I've also manually tested the path gated behind MICROPY_PY_DESCRIPTORS && !MICROPY_PY_METACLASSES_LITE to verify it passes the tests from #17693, but I'm not sure there's actually any way to write a SKIP condition for them that isn't just checking the very thing the tests are meant to discriminate.

@dpgeorge
Copy link
Member

Would be good to see the change in benchmark results with the latest version here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dpgeorge dpgeorge dpgeorge left review comments

Assignees
No one assigned
Labels
py-core Relates to py/ directory in source
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@AJMansfield @dpgeorge
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy