Port, board and/or hardware

Any

MicroPython version

MicroPython v1.26.0-preview.386.g17fbc5abd.dirty

Reproduction

1. Clean build your port of choice
2. Run the test suite, notice the absence of failures
3. Add #define MICROPY_ERROR_REPORTING (MICROPY_ERROR_REPORTING_TERSE) to your port's mpconfigport.h file
4. Clean build the port
5. Run the test suite again.

Expected behaviour

All tests that were passing before changing the error reporting level should still pass.

Observed behaviour

I've seen the following tests fail (for example on QEMU/RV32):

• micropython/heapalloc_exc_compressed.py
• micropython/heapalloc_exc_compressed_emg_exc.py
• micropython/native_with.py
• micropython/opt_level_lineno.py
• micropython/viper_with.py
• misc/print_exception.py

Probably there are more of these but are port-specific and haven't encountered them yet.

Additional Information

The issue is that those tests perform an exact string match on the exception message being raised by the interpreter. With the error reporting level changing, the exception message may change, and thus fail perfectly working tests.

I see this issue on a custom port I'm doing where, for space reasons, I've got to keep error reporting verboseness down. This means I cannot get a single clean test run because with the extra space taken by the error messages (and their interpolation) I have to cut out other features to fit things in the allotted flash space.

A similar issue also exists, as in it's not possible to have a clean test run with the port's ROM level set to MICROPY_CONFIG_ROM_LEVEL_MINIMUM as some tests (even in basic) depend on features cut out by that definition.

I don't mind cleaning up those tests to have a more comprehensive pass/fail check and eventually migrate them to unittest if that's required (and putting in the work to make it testable with the minimum feature set, whilst I'm here...), but before starting this I wonder if this is a PR (or multiple PRs, who knows) that will be at least considered for inclusion.

Code of Conduct

Yes, I agree