Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.44%. Comparing base (df05cae) to head (4ba892a).
Report is 9 commits behind head on master.

@@           Coverage Diff           @@
##           master   #15503   +/-   ##
=======================================
  Coverage   98.44%   98.44%           
=======================================
  Files         171      171           
  Lines       22192    22217   +25     
=======================================
+ Hits        21847    21872   +25     
  Misses        345      345           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:  +168 +0.020% standard[incl +32(data)]
      stm32:   +92 +0.023% PYBV10
     mimxrt:   +96 +0.026% TEENSY40
        rp2:  +104 +0.011% RPI_PICO_W
       samd:  +120 +0.045% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:  +101 +0.022% VIRT_RV32

Thanks for the contribution.

I think this is a good addition. It's important to support this __set_name__ special-method to properly make use of descriptors (which MicroPython does support, at least with get/set/delete).

I'm just wondering about the performance hit. It means that each class that's defined will need to iterate through all of its members. I wonder if there's a way to make use of MP_TYPE_FLAG_HAS_SPECIAL_ACCESSORS to decide whether to do this search or not. There's already check_for_special_accessors() that's run on class creation in a loop over all members. Maybe that can be reused instead of essentially duplicating the work?

I'm just wondering about the performance hit.

I do think the extra loop is necessary, for reasons described below -- but also, this code only runs at class-creation time, something that in most cases should only happen a small fixed number of times during setup. (At least, outside of intractable runtime dynamic metaclassing scenarios for which performance is already a lost cause...)

I wonder if there's a way to make use of MP_TYPE_FLAG_HAS_SPECIAL_ACCESSORS to decide whether to do this search or not.

That could be done if we are only interested in addressing instance-attribute-like descriptors, but that would miss classattr- and classmethod-like descriptors that only use __set_name__ to bind to the containing class. We could try to special-case that out too, but it's much simpler to just check the thing itself and look up MP_QSTR___set_name__ directly.

There's already check_for_special_accessors() that's run on class creation in a loop over all members. Maybe that can be reused instead of essentially duplicating the work?

A check for a __set_name__ sub-member could be added as part of that loop, but there could be a sequencing hazard with invoking user code anywhere except right at the end -- and even then, invoking before attribute access is fully ready (incl. having the special accessor flag correctly set) would become a sequencing hazard if/when metaclasses become supported.

My opinion, is that it's better to just keep the loop separate. It does mean chasing down member pointers a second time, but the expensive part of the loop is looking up MP_QSTR___set_name__ on each member, which needs to be done whether or not it's part of the same loop with check_for_special_accessors.

Update: I'm currently investigating some corner-case behavior I recently found around what happens (and what should happen) when a __set_name__ function inserts or removes elements from the class being initialized.

At the moment I'm still waiting on python/cpython#122381 to define what the exact semantics should be in that case -- but this PR is still valid for a merge.

Whatever the correct semantics are, they'll need at least a small code-size increase. And with how rarely that gets used, I feel like it'll probably be better gate them behind the MICROPY_PY_METACLASSES flag from #15511, and make that a separate PR.

@dpgeorge should I spend the time adding a cpydiff to this PR about what the exact mismatch is?

should I spend the time adding a cpydiff to this PR about what the exact mismatch is?

Yes please. Then at least all this work/knowledge is encoded in a test and can be improved later on.

Finally found some time to step back in to finish what I started here.

It looks like python/cpython#122381 is probably going nowhere and will just leave that behavior unspecified for now, or at best will just officially document CPython's current behavior.

Yes please. Then at least all this work/knowledge is encoded in a test and can be improved later on.

Unfortunately, I've not been able to find a way to actually cpydiff this :(.
Certainly, I've found test cases that fail, but I've simply found no way to make a test that fails because of the sequence hazard, rather than just due to differences in CPython vs Micropython iteration order.

I'll see if I can at least add a cpydiff that'll at least remain failing even when the iteration order happens to be the same, though, and other tests that at least verify that hazard-free class namespace editing scenarios work while I'm rebasing this to the current master.

Not exactly proud of needing to resort to a stochastic test to reliably show off the bug, but I have a reliable cpydiff now for the specific sequence hazard I was worried about.

Fundamentally, it's a modify-while-iterating bug, of exactly the same kind that creates this diff:

d = {'a':1, 'b':2, 'c':3}
for k,v in d.items():
    d[k+k]=v+v
print(d)

(CPython errors with RuntimeError: dictionary changed size during iteration, while micropython just blithely proceeds and produces {'c': 3, 'b': 2, 'bbbbbbbb': 16, 'aaaa': 4, 'a': 1, 'aa': 2, 'bb': 4, 'bbbb': 8})

CPython avoids this bug in its __set_name__ implementation because it effectively iterates on dict.keys(), which creates a copy of the current set of keys instead of just a view. However, my proposed implementation for Micropython iterates through the raw allocated slots in locals_map (the same way the other steps that iterate its namespace do).

As a result, in my cpydiff/core_class_setname_hazard test case, involving a class with 26 descriptors with names a through z that each insert 10 additional attributes into the class at random, CPython ends up calling __set_name__ exactly once on only these original descriptors -- but my proposed micropython code calls some of them multiple times, and some not at all, as the order of the dict entries being iterated on shifts in response to the new insertions:

+-------------+-------------+
| CPy output: | uPy output: |
+-------------+-------------+
|     a 1     |     a 2     |
|     b 1     |     b 1     |
|     c 1     |     c 2     |
|     d 1     |     d 3     |
|     e 1     |     e 4     |
|     f 1     |     f 3     |
|     g 1     |     g 4     |
|     h 1     |     h 2     |
|     i 1     |     i 1     |
|     j 1     |     j 3     |
|     k 1     |     k 2     |
|     l 1     |     l 1     |
|     m 1     |     m 1     |
|     n 1     |     n 1     |
|     o 1     |     o 1     |
|     p 1     |     p 0     |
|     q 1     |     q 1     |
|     r 1     |     r 0     |
|     s 1     |     s 0     |
|     t 1     |     t 1     |
|     u 1     |     u 3     |
|     v 1     |     v 0     |
|     w 1     |     w 1     |
|     x 1     |     x 0     |
|     y 1     |     y 0     |
|     z 1     |     z 0     |
+-------------+-------------+

A similar effect can happen when __set_name__ removes instead of adds; but the worst case is still just descriptors having their __set_name__ missed or called multiple times.

There's no hazard in cases where there are no class-mutating descriptors, and this might not be a use-case it's worthwhile to support. I'll see if I can create an alternative pull-request for a version that exactly matches CPython, though, so we can compare the performance cost of creating that copy on more than just speculation.

As one more potential alternative to leaving it an undefined behavior... we could temporarily set locals_map->is_fixed = 1 before the calls to __set_name__ to trigger an error if the user's code tries to mutate the class, and clear it after. Though of the four, this version is the least functional/useful, but also the simplest that avoids the potential for confusion created by the iteration hazard.

I've done some benchmarking using a new suite of internalbench class creation benchmarks (see PR #16825) and have compared the benchmark times from the base branch to this branch.

There's a lot of noise in this (I'll see if I can run these on real hardware at some point), but overall this patch makes processing classes take about 12% longer:

internal_bench/class_create:
    0.349 -> 0.391 (+12%) internal_bench/class_create-0-empty.py
    0.476 -> 0.544 (+14%) internal_bench/class_create-1-slots.py
    0.490 -> 0.553 (+13%) internal_bench/class_create-1.1-slots5.py
    0.432 -> 0.483 (+12%) internal_bench/class_create-2-classattr.py
    0.776 -> 0.868 (+12%) internal_bench/class_create-2.1-classattr5.py
    0.461 -> 0.533 (+16%) internal_bench/class_create-3-instancemethod.py
    0.477 -> 0.535 (+12%) internal_bench/class_create-4-classmethod.py
    0.452 -> 0.511 (+13%) internal_bench/class_create-4.1-classmethod_implicit.py
    0.494 -> 0.537 (+09%) internal_bench/class_create-5-staticmethod.py
    0.458 -> 0.512 (+12%) internal_bench/class_create-6-getattribute.py
    0.474 -> 0.518 (+09%) internal_bench/class_create-6.1-getattr.py
    0.386 -> 0.447 (+16%) internal_bench/class_create-6.2-descriptor.py
    0.517 -> 0.625 (+21%) internal_bench/class_create-6.3-descriptor_setname.py
    0.419 -> 0.480 (+15%) internal_bench/class_create-6.4-property.py
    0.363 -> 0.420 (+16%) internal_bench/class_create-7-inherit.py
    0.368 -> 0.399 (+09%) internal_bench/class_create-7.1-inherit_initsubclass.py

There were also two other benchmark tests that got concerningly slower:

internal_bench/arrayop:
    0.174 -> 0.194 (+12%) internal_bench/arrayop-3-bytearray_inplace.py
internal_bench/loop_count:
    0.268 -> 0.310 (+16%) internal_bench/loop_count-2-range_iter.py

None of the other tests in their families seemed to be affected, but my tests for #16806 and #16816 both showed loop_count-2 slowed by near-enough the exact same margins as class creation.