Summary

Some tests are just too big for targets that don't have much heap memory. Eg tests/extmod/vfs_rom.py. Other tests are too large because the target doesn't have enough IRAM for native code, eg tests/micropython/ viper_args.py.

Previously, such tests were explicitly skipped on targets known to have little memory, eg esp8266. But this doesn't scale to multiple targets, nor to more and more tests which are too large.

This PR addresses that by adding logic to the test runner so it can automatically skip tests when they don't fit in the target's memory. It does this by prepending a print('START TEST') to every test, and if a MemoryError occurs before that line is printed then the test was too big. This works for standard tests, tests that go via .mpy files, and tests that run in native emitter mode via .mpy files.

For tests that are too big, it prints lrge <test name> on the output, and adds them to a separate list of skipped tests so they can be distinguished. They are also written to the _result.json file in a separate entry.

Testing

Tested on esp8266:

$ ./run-tests.py -t u0
platform=esp8266 arch=xtensa inlineasm=xtensa
pass  basics/0prelim.py 
pass  basics/andor.py 
...
pass  micropython/stack_use.py 
pass  micropython/viper_addr.py 
lrge  micropython/viper_args.py
lrge  micropython/viper_binop_arith.py
pass  micropython/viper_binop_arith_uint.py 
pass  micropython/viper_binop_bitwise_uint.py 
...
pass  misc/non_compliant_lexer.py 
pass  misc/print_exception.py 
skip  misc/rge_sm.py
skip  misc/sys_atexit.py
skip  misc/sys_exc_info.py
skip  misc/sys_settrace_features.py
skip  misc/sys_settrace_generator.py
skip  misc/sys_settrace_loop.py
785 tests performed (22822 individual testcases)
785 tests passed
115 tests skipped: binascii_crc32 builtin_compile <long line truncated>
3 tests skipped because they are too large: vfs_rom viper_args viper_binop_arith

Also ran --via-mpy tests on esp8266: it skipped the two tests viper_args and viper_binop_arith because they were too large but was able to run vfs_rom because that can fit when compiled as .mpy (although it still skipped due to the target not supporting VfsRom but at least it shows that the auto-skipping due to OOM is working well).

Also ran --via-mpy --emit native on esp8266: prior to this PR running that would have 343 tests fail, but now with this PR it's 348 tests skipped because they are too large and 10 tests failed which is a very big improvement.

Also ran tests on RPI_PICO2_W. They run as usual and nothing is skipped.

Trade-offs and Alternatives

• This makes each test slightly larger due to the added print statement. As a consequence a test that may have fit previously might not fit now. But such tests were already on the edge of having enough RAM to run anyway.
• Tests that rely on source-code line numbers in their output must start with a comment, so that the print can be inserted before the comment and not change the line numbering.
• It cannot detect tests that fail due to out-of-memory part way through. Only those that fail upfront to even parse/compile (which accounts for the vast majority of tests anyway).
• Alternative would be to continue to manually maintain a list of the tests which are large, and detect the amount of heap memory on a target, and skip tests if the target is lower than a set threshold. But this is time consuming and error prone.