-
-
Notifications
You must be signed in to change notification settings - Fork 8.3k
tests/internal_bench: Benchmarks for descriptor-related features. #16825
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #16825 +/- ##
==========================================
- Coverage 98.44% 98.41% -0.04%
==========================================
Files 171 171
Lines 22208 22210 +2
==========================================
- Hits 21863 21857 -6
- Misses 345 353 +8 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
04d3702
to
2409a66
Compare
It looks like the unix port / settrace_stackless test failure is spurious? It failed in |
e7ba4d7
to
833837a
Compare
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this, it's a nice addition, quite a comprehensive set of tests. Also great to be able to now run it on a remote target.
I noticed that tests like internal_bench/var-1-constant.py
fail on remote targets because they take too long, about 1 minute. I'd be OK to just leave that for now, unless you want to try and fix it.
Yeah, I figured it might just be a timeout --- the root cause here is actually just that the test functions in the Fixing this probably means needing to rewrite them to generate the code under test with a string interpolation and then |
Ok actually I found a much dumber way of making the var tests run --- I can just have the runner string-replace Probably an even worse syntactic sin than trying to parse HTML with regex --- but if you think about it, you can just define a sub-language with a grammar that only accepts the specific benchmarks that are actually in the folder as valid, and at least over that sub-language the literal string replacement is a correct parse. |
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in setname_list_t, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in `setname_list_t`, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
This PR adds support for the `__set_name__` data model method specified by PEP487 - Simpler customisation of class creation. This includes support for methods that mutate the owner class, and avoids the naive modify-while-iterating hazard possible in a naive implementation like micropython#15503. Note that based on the benchmarks in micropython#16825, this is also as fast or faster than the naive implementation, thanks to clever data layout in `setname_list_t`, and the way this allows the capture step to run during an existing loop through the class dict. Other rejected approaches for dealing with the hazard include: - python/cpython#72983 During the implementation of this feature for MicroPython, it was discovered that some versions of CPython also have this naive hazard. CPython resolved this bug in BPO-28797 and now makes a complete flat copy of the class's dict to iterate. This design decision doesn't make much sense for a microcontroller though, even if it's perfectly reasonable in the desktop world where memcpy might actually be cheaper than a hard-to-branch-predict conditional; and it's also motivated in their case by error-tracing considerations. - micropython#16816 This is an equivalent implementation to CPython's approach that places this copy directly on the stack; however it is both slower and has larger code size than the approach taken here. - micropython#15503 The simplest implementation is to just not worry about it and let the user face the consequences if they mutate the owner class. That's not a very friendly behavior, though, and it's not actually much more performant than this implementation on either time or code size. - micropython#17693 Another alternative is to do the same as micropython#15503 but leverage MicroPython's existing `is_fixed` field in its dict type to convert attempted mutations of the owner dict into `AttributeError`s. This is safer than just leaving the open hazard, but there's still important use-cases for owner-mutating descriptors, and the performance ain is small enough that it isn't worth missing support for those cases. - combined micropython#17693 with this Another version of this feature used a new feature define, `MICROPY_PY_METACLASSES_LITE`, to control whether this algorithm or the naive version is used. This was rejected in favor of simplicity, based on the very limited performance margin the naive version has (which in some cases even goes _against_ it). Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Summary
This includes new
internal_bench
benchmarks for how quickly micropython can process class definitions, created in order to gauge the performance penalty for implementing__set_name__
in the different ways proposed in #15503, #16806, and #16816.It also includes an update to
run-internalbench.py
that adds in the same ability to run the benchmarks on remote micropython instances (e.g. actual microcontroller hardware) that the mainrun-tests.py
has.Testing
I've tested these benchmarks using the unix port inside WSL on my development machine, with the following results:
Trade-offs and Alternatives
Due to the overall slow speed of class creation in general, the
class_create
tests reduce the number of iterations from 20,000,000 to just 500,000; but this should still be wide enough to give the results adequate statistical significance.For microcontroller remotes it uses 200,000 as the base default (which in the class_create case gets divided down to 5,000 runs).