Skip to content

tests/internal_bench: Benchmarks for descriptor-related features. #16825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

AJMansfield
Copy link
Contributor

@AJMansfield AJMansfield commented Feb 26, 2025

Summary

This includes new internal_bench benchmarks for how quickly micropython can process class definitions, created in order to gauge the performance penalty for implementing __set_name__ in the different ways proposed in #15503, #16806, and #16816.

It also includes an update to run-internalbench.py that adds in the same ability to run the benchmarks on remote micropython instances (e.g. actual microcontroller hardware) that the main run-tests.py has.

Testing

I've tested these benchmarks using the unix port inside WSL on my development machine, with the following results:

 . . .
internal_bench/class_create:
    0.351s (+00.00%) internal_bench/class_create-0-empty.py
    0.476s (+35.91%) internal_bench/class_create-1-slots.py
    0.484s (+38.01%) internal_bench/class_create-1.1-slots5.py
    0.427s (+21.87%) internal_bench/class_create-2-classattr.py
    0.773s (+120.54%) internal_bench/class_create-2.1-classattr5.py
    0.458s (+30.62%) internal_bench/class_create-3-instancemethod.py
    0.500s (+42.76%) internal_bench/class_create-4-classmethod.py
    0.462s (+31.65%) internal_bench/class_create-4.1-classmethod_implicit.py
    0.498s (+41.93%) internal_bench/class_create-5-staticmethod.py
    0.456s (+29.97%) internal_bench/class_create-6-getattribute.py
    0.467s (+33.13%) internal_bench/class_create-6.1-getattr.py
    0.388s (+10.58%) internal_bench/class_create-6.2-descriptor.py
    0.529s (+50.89%) internal_bench/class_create-6.3-descriptor_setname.py
    0.425s (+21.22%) internal_bench/class_create-6.4-property.py
    0.368s (+05.02%) internal_bench/class_create-7-inherit.py
    0.359s (+02.35%) internal_bench/class_create-7.1-inherit_initsubclass.py
 . . .
internal_bench/var:
    0.459s (+00.00%) internal_bench/var-1-constant.py
    0.620s (+35.05%) internal_bench/var-2-global.py
    0.443s (-03.42%) internal_bench/var-3-local.py
    0.441s (-03.96%) internal_bench/var-4-arg.py
    1.131s (+146.39%) internal_bench/var-5-class-attr.py
    0.539s (+17.34%) internal_bench/var-6-instance-attr.py
    0.555s (+20.85%) internal_bench/var-6.1-instance-attr-5.py
    0.537s (+17.04%) internal_bench/var-6.2-instance-speciallookup.py
    2.436s (+430.54%) internal_bench/var-6.3-instance-property.py
    3.102s (+575.63%) internal_bench/var-6.4-instance-descriptor.py
    3.144s (+584.85%) internal_bench/var-6.5-instance-getattr.py
    2.422s (+427.62%) internal_bench/var-7-instance-meth.py
    0.865s (+88.46%) internal_bench/var-8-namedtuple-1st.py
    0.924s (+101.30%) internal_bench/var-8.1-namedtuple-5th.py
 . . .

Trade-offs and Alternatives

Due to the overall slow speed of class creation in general, the class_create tests reduce the number of iterations from 20,000,000 to just 500,000; but this should still be wide enough to give the results adequate statistical significance.

For microcontroller remotes it uses 200,000 as the base default (which in the class_create case gets divided down to 5,000 runs).

Copy link

codecov bot commented Feb 26, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.41%. Comparing base (17fbc5a) to head (f391368).
Report is 6 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #16825      +/-   ##
==========================================
- Coverage   98.44%   98.41%   -0.04%     
==========================================
  Files         171      171              
  Lines       22208    22210       +2     
==========================================
- Hits        21863    21857       -6     
- Misses        345      353       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@AJMansfield
Copy link
Contributor Author

It looks like the unix port / settrace_stackless test failure is spurious? It failed in thread/thread_gc1.py, which should have nothing to do with anything this PR does.

@AJMansfield AJMansfield changed the title tests/internal_bench: Benchmarks for descriptor-related things. tests/internal_bench: Benchmarks for descriptor-related features. Feb 26, 2025
@dpgeorge dpgeorge added the tests Relates to tests/ directory in source label Mar 13, 2025
@AJMansfield AJMansfield force-pushed the benchmarks branch 2 times, most recently from e7ba4d7 to 833837a Compare July 20, 2025 17:47
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Copy link
Member

@dpgeorge dpgeorge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, it's a nice addition, quite a comprehensive set of tests. Also great to be able to now run it on a remote target.

I noticed that tests like internal_bench/var-1-constant.py fail on remote targets because they take too long, about 1 minute. I'd be OK to just leave that for now, unless you want to try and fix it.

@AJMansfield
Copy link
Contributor Author

I noticed that tests like internal_bench/var-1-constant.py fail on remote targets because they take too long, about 1 minute.

Yeah, I figured it might just be a timeout --- the root cause here is actually just that the test functions in the var- tests can't just use the num parameter since the feature under test is actually the way that their loop bound is represented.

Fixing this probably means needing to rewrite them to generate the code under test with a string interpolation and then eval-ing the resulting string.

@AJMansfield
Copy link
Contributor Author

AJMansfield commented Jul 21, 2025

Ok actually I found a much dumber way of making the var tests run --- I can just have the runner string-replace "20000000" in the benchmark code with the actual number of iterations.

Probably an even worse syntactic sin than trying to parse HTML with regex --- but if you think about it, you can just define a sub-language with a grammar that only accepts the specific benchmarks that are actually in the folder as valid, and at least over that sub-language the literal string replacement is a correct parse.

Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
AJMansfield added a commit to AJMansfield/micropython that referenced this pull request Jul 21, 2025
This PR adds support for the `__set_name__` data model method
specified by PEP487 - Simpler customisation of class creation.

This includes support for methods that mutate the owner class,
and avoids the naive modify-while-iterating hazard possible in a naive
implementation like micropython#15503.

Note that based on the benchmarks in micropython#16825, this
is also as fast or faster than the naive implementation, thanks to
clever data layout in setname_list_t, and the way this allows the
capture step to run during an existing loop through the class dict.

Other rejected approaches for dealing with the hazard include:

- python/cpython#72983
During the implementation of this feature for MicroPython, it was
discovered that some versions of CPython also have this naive hazard.
CPython resolved this bug in BPO-28797 and now makes a complete flat
copy of the class's dict to iterate. This design decision doesn't make
much sense for a microcontroller though, even if it's perfectly
reasonable in the desktop world where memcpy might actually be cheaper
than a hard-to-branch-predict conditional; and it's also motivated in
their case by error-tracing considerations.

- micropython#16816
This is an equivalent implementation to CPython's approach that places
this copy directly on the stack; however it is both slower and has
larger code size than the approach taken here.

- micropython#15503
The simplest implementation is to just not worry about it and let the
user face the consequences if they mutate the owner class.
That's not a very friendly behavior, though, and it's not actually much
more performant than this implementation on either time or code size.

- micropython#17693
Another alternative is to do the same as micropython#15503 but leverage
MicroPython's existing `is_fixed` field in its dict type to convert
attempted mutations of the owner dict into `AttributeError`s.
This is safer than just leaving the open hazard, but there's still
important use-cases for owner-mutating descriptors, and the performance
ain is small enough that it isn't worth missing support for those cases.

- combined micropython#17693 with this
Another version of this feature used a new feature define,
`MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the
naive version is used. This was rejected in favor of simplicity, based
on the very limited performance margin the naive version has (which in
some cases even goes _against_ it).

Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
AJMansfield added a commit to AJMansfield/micropython that referenced this pull request Jul 21, 2025
This PR adds support for the `__set_name__` data model method
specified by PEP487 - Simpler customisation of class creation.

This includes support for methods that mutate the owner class,
and avoids the naive modify-while-iterating hazard possible in a naive
implementation like micropython#15503.

Note that based on the benchmarks in micropython#16825, this
is also as fast or faster than the naive implementation, thanks to
clever data layout in `setname_list_t`, and the way this allows the
capture step to run during an existing loop through the class dict.

Other rejected approaches for dealing with the hazard include:

- python/cpython#72983
During the implementation of this feature for MicroPython, it was
discovered that some versions of CPython also have this naive hazard.
CPython resolved this bug in BPO-28797 and now makes a complete flat
copy of the class's dict to iterate. This design decision doesn't make
much sense for a microcontroller though, even if it's perfectly
reasonable in the desktop world where memcpy might actually be cheaper
than a hard-to-branch-predict conditional; and it's also motivated in
their case by error-tracing considerations.

- micropython#16816
This is an equivalent implementation to CPython's approach that places
this copy directly on the stack; however it is both slower and has
larger code size than the approach taken here.

- micropython#15503
The simplest implementation is to just not worry about it and let the
user face the consequences if they mutate the owner class.
That's not a very friendly behavior, though, and it's not actually much
more performant than this implementation on either time or code size.

- micropython#17693
Another alternative is to do the same as micropython#15503 but leverage
MicroPython's existing `is_fixed` field in its dict type to convert
attempted mutations of the owner dict into `AttributeError`s.
This is safer than just leaving the open hazard, but there's still
important use-cases for owner-mutating descriptors, and the performance
ain is small enough that it isn't worth missing support for those cases.

- combined micropython#17693 with this
Another version of this feature used a new feature define,
`MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the
naive version is used. This was rejected in favor of simplicity, based
on the very limited performance margin the naive version has (which in
some cases even goes _against_ it).

Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
AJMansfield added a commit to AJMansfield/micropython that referenced this pull request Jul 23, 2025
This PR adds support for the `__set_name__` data model method
specified by PEP487 - Simpler customisation of class creation.

This includes support for methods that mutate the owner class,
and avoids the naive modify-while-iterating hazard possible in a naive
implementation like micropython#15503.

Note that based on the benchmarks in micropython#16825, this
is also as fast or faster than the naive implementation, thanks to
clever data layout in `setname_list_t`, and the way this allows the
capture step to run during an existing loop through the class dict.

Other rejected approaches for dealing with the hazard include:

- python/cpython#72983
During the implementation of this feature for MicroPython, it was
discovered that some versions of CPython also have this naive hazard.
CPython resolved this bug in BPO-28797 and now makes a complete flat
copy of the class's dict to iterate. This design decision doesn't make
much sense for a microcontroller though, even if it's perfectly
reasonable in the desktop world where memcpy might actually be cheaper
than a hard-to-branch-predict conditional; and it's also motivated in
their case by error-tracing considerations.

- micropython#16816
This is an equivalent implementation to CPython's approach that places
this copy directly on the stack; however it is both slower and has
larger code size than the approach taken here.

- micropython#15503
The simplest implementation is to just not worry about it and let the
user face the consequences if they mutate the owner class.
That's not a very friendly behavior, though, and it's not actually much
more performant than this implementation on either time or code size.

- micropython#17693
Another alternative is to do the same as micropython#15503 but leverage
MicroPython's existing `is_fixed` field in its dict type to convert
attempted mutations of the owner dict into `AttributeError`s.
This is safer than just leaving the open hazard, but there's still
important use-cases for owner-mutating descriptors, and the performance
ain is small enough that it isn't worth missing support for those cases.

- combined micropython#17693 with this
Another version of this feature used a new feature define,
`MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the
naive version is used. This was rejected in favor of simplicity, based
on the very limited performance margin the naive version has (which in
some cases even goes _against_ it).

Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Relates to tests/ directory in source
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy