Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.44%. Comparing base (df05cae) to head (87cb9a0).
Report is 9 commits behind head on master.

@@           Coverage Diff           @@
##           master   #16816   +/-   ##
=======================================
  Coverage   98.44%   98.44%           
=======================================
  Files         171      171           
  Lines       22192    22223   +31     
=======================================
+ Hits        21847    21878   +31     
  Misses        345      345           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:  +376 +0.044% standard[incl +32(data)]
      stm32:  +132 +0.034% PYBV10
     mimxrt:  +136 +0.036% TEENSY40
        rp2:  +152 +0.017% RPI_PICO_W
       samd:  +160 +0.060% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:  +187 +0.041% VIRT_RV32

I've done some benchmarking using a new suite of internalbench class creation benchmarks (see PR #16825) and have compared the benchmark times from the base branch to this branch.

There's a lot of noise in this (I'll see if I can run these on real hardware at some point), but overall this patch makes processing classes take about 30% longer -- though with the amount of noise it's a coin flip whether the code was actually faster than #16806 or just luckier.

internal_bench/class_create:
    0.349 -> 0.464 (+33%) internal_bench/class_create-0-empty.py
    0.476 -> 0.627 (+32%) internal_bench/class_create-1-slots.py
    0.490 -> 0.609 (+24%) internal_bench/class_create-1.1-slots5.py
    0.432 -> 0.556 (+29%) internal_bench/class_create-2-classattr.py
    0.776 -> 0.936 (+21%) internal_bench/class_create-2.1-classattr5.py
    0.461 -> 0.577 (+25%) internal_bench/class_create-3-instancemethod.py
    0.477 -> 0.628 (+32%) internal_bench/class_create-4-classmethod.py
    0.452 -> 0.588 (+30%) internal_bench/class_create-4.1-classmethod_implicit.py
    0.494 -> 0.627 (+27%) internal_bench/class_create-5-staticmethod.py
    0.458 -> 0.597 (+30%) internal_bench/class_create-6-getattribute.py
    0.474 -> 0.573 (+21%) internal_bench/class_create-6.1-getattr.py
    0.386 -> 0.534 (+39%) internal_bench/class_create-6.2-descriptor.py
    0.517 -> 0.701 (+36%) internal_bench/class_create-6.3-descriptor_setname.py
    0.419 -> 0.574 (+37%) internal_bench/class_create-6.4-property.py
    0.363 -> 0.479 (+32%) internal_bench/class_create-7-inherit.py
    0.368 -> 0.477 (+30%) internal_bench/class_create-7.1-inherit_initsubclass.py

There were also two other benchmark tests that got concerningly slower:

internal_bench/func_builtin:
    0.209 -> 0.246 (+18%) internal_bench/func_builtin-2-enum_kw.py
internal_bench/loop_count:
    0.268 -> 0.340 (+27%) internal_bench/loop_count-2-range_iter.py

None of the other tests in their families seemed to be affected, but my tests for #15503 and #16806 both showed loop_count-2 slowed by near-enough the exact same margins as class creation.

Just updated this to instead make its own copy of locals_map into a mp_local_alloc in order to avoid the need to set a NLR trap handler on non-MICROPY_ENABLE_PYSTACK ports, but I have a small anxiety about a potential use-after-free related to garbage collection and I can't say I know enough about how gc works to be confident it can't be a problem.

@dpgeorge Does anything extra need to be done to guarantee that objects referenced in the table I'm allocating on the stack can't be garbage collected?

The __set_name__ call returns control back to user code and the bytecode executor during mp_call_method_n_kw(2, 0, set_name_method). Theoretically, it's therefore possible for gc.collect() to run in the middle that, either explicitly or as a result of memory pressure.

If a descriptor's __set_name__ method were to remove an attribute from its owner, that copy of the map sitting on the stack in run_set_name_hooks could be the only remaining live reference to that attribute's value and/or name. If that's not enough to prevent an object from being collectable, the garbage collector could collect them, and if it happens to be later in the iteration order trying to load and call __set_name__ on it would become a use-after-free.

Outside of this __set_name__ implementation, I have been able to find zero examples of code needing to allocate an object and then call to user code without already needing to do something else that would indirectly give it root-pointer provenance.

The only thing kinda close is in mp_obj_instance_make_new, where you have o = mp_obj_new_instance followed by mp_call_method_n_kw -- but that object o is being passed as the self-parameter and could be picking up provenance there. Everywhere else, the allocated object is returned back to the bytecode interpreter, which grants provenance with whatever variable it assigns it to, or it's a MP_OBJ_NEW_SMALL_INT and MP_OBJ_NEW_QSTR that could never be garbage-collected anyway.

Actually, for that matter... if what I'm worrying about is the case, the class being initialized could itself end up being collected before its returned, even without this patch. The only pointers to the class object until mp_obj_new_type returns are also on the stack, and the mp_load_method_protected call in check_for_special_accessors could potentially itself call to user code if the descriptor defines __getattr__ or its __get__ attribute is itself a descriptor.

EDIT: After some experimentation I've determined that this (the class weirdness at least) is at least, much harder to trigger than I originally imagined, if it's possible; but I did uncover another very weird behavioral difference. On master:

class A:
    def __get__(self, instance, owner=None):
        print("A")

class B:
    __get__ = A()

print("C")
class C:
    x = B()

FAILURE /home/anson/mpy/micropython/tests/results/basics_class_descriptor_descriptor.py
--- /home/anson/mpy/micropython/tests/results/basics_class_descriptor_descriptor.py.exp 2025-02-27 15:21:01.414746630 -0500
+++ /home/anson/mpy/micropython/tests/results/basics_class_descriptor_descriptor.py.out 2025-02-27 15:21:01.414746630 -0500
@@ -1 +1,2 @@
 C
+A

(I have no idea how it is that check_for_special_accessors into mp_load_method_protected manages to recurse into an attempted meta-descriptor's __get__ method (instead of its __call__, the way CPython does), despite looking like it loads from the type's locals_dict directly instead of going through attribute access -- but that's definitely what's happening here, and you can nest it even further and still get the first descriptor's __get__ being called.)

This is my least favourite of the implementations. It's stack heavy and unconditionally allocates.

I think #16806 could be made to be simpler than this version.