unix/main: Use standard pyexec/repl for unix port. #12802

andrewleech · 2023-10-25T23:40:29Z

This improves repl usage consistency across ports.
Only enabled when MICROPY_USE_READLINE == 1 (default).

In particular I noticed that aiorepl didn't work properly on the unix port currently, it would echo the "just entered line" on a newline each time you hit enter, eg.

>>> import aiorepl, asyncio;asyncio.run(aiorepl.task())
Starting asyncio REPL...
--> print("hello")
print("hello")
hello

I feel there were also a few other inconsistencies with how I'm used to the repl working on the stm port / via mpremote which are alsos hopefully resolved now.

I've got an optional couple of lines to restore/reset stdio_raw mode when running code - however if raw mode is disabled the aiorepl issue above remains. I think for consistency with other ports it's better to keep stdio in raw mode, but not sure if this is likely to have other side effects?

This PR consilidates the repl functionality and fixes the above aiorepl glitch.

dpgeorge · 2023-10-25T23:51:56Z

Thanks for this, it's a good clean up.

I actually attempted it before, see #6008. The approach here is much simpler, so maybe it's a good first step to make.

andrewleech · 2023-10-26T01:34:26Z

Ah interesting, I did see some of the return handling and kind of assumed it shouldn't matter - if someone is dealing with the repl (live coding) they're unlikely to care about the exit code?

I didn't consider the SystemExit interrupt at all, haven't looked at how that behaves but probably should!

If running a python file directly then certainly the exit code is important but that shouldn't be impacted by this.

andrewleech · 2023-10-26T01:43:03Z

Ah, I might need to read through the other PR discussion a bit more... but in the mean time this is a little off:

$ ~/micropython/ports/unix/build-standard/micropython
MicroPython v1.22.0-preview.32.g2c1c7d8b30 on 2023-10-26; linux [GCC 10.2.1] version
Type "help()" for more information.
>>> raise ValueError()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError:
>>> raise SystemExit()
>>>

Edit:
Actually that behaviour above is consistent with running a script on Unix port in that SystemExit will drop out of the running script back to console without printing anything, other exceptions drop out while printing the traceback.

jimmo · 2023-10-26T07:36:27Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.44%. Comparing base (8f8f853) to head (dbda928).

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #12802   +/-   ##
=======================================
  Coverage   98.44%   98.44%           
=======================================
  Files         171      171           
  Lines       22192    22192           
=======================================
  Hits        21847    21847           
  Misses        345      345

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dpgeorge · 2024-07-05T06:24:34Z

Regarding the comment above about raise SystemExit: actually this PR changes the behaviour, now if you raise SystemExit from the REPL it does nothing. Currently on master it will exit MicroPython back to the shell.

We need to retain the existing behaviour, so we don't break existing usage, and because that's how CPython works.

Then it's a separate question how to handle this on bare-metal targets. But for now we shouldn't change the unix behaviour on this.

github-actions · 2024-07-09T07:06:44Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:   +61 +0.032% 
   unix x64: +1704 +0.199% standard
      stm32:   +52 +0.013% PYBV10
     mimxrt:   +56 +0.015% TEENSY40
        rp2:   +56 +0.006% RPI_PICO_W
       samd:   +52 +0.019% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:   +54 +0.012% VIRT_RV32

andrewleech · 2024-07-09T07:07:09Z

Regarding the comment above about raise SystemExit: actually this PR changes the behaviour, now if you raise SystemExit from the REPL it does nothing. Currently on master it will exit MicroPython back to the shell.

We need to retain the existing behaviour, so we don't break existing usage, and because that's how CPython works.

Thanks, yes I hadn't realised this behavior had been changed inadvertantly. I've now fixed it in the latest push.
There was an associates variable here pyexec_system_exit which I did a global search for - it was never actually being used (anymore) so I've stripped that out as well.
Any variable passed to SystemExit is now returned for unix port.

anl@STEP: ~ $ micropython
MicroPython v1.24.0-preview.102.g03e992d59d.dirty on 2024-07-09; linux [GCC 9.4.0] version
Type "help()" for more information.
>>> raise SystemExit
anl@STEP: ~ $ echo $?
0
anl@STEP: ~ $ micropython
MicroPython v1.24.0-preview.102.g03e992d59d.dirty on 2024-07-09; linux [GCC 9.4.0] version
Type "help()" for more information.
>>> raise SystemExit(-1)
anl@STEP: ~ $ echo $?
255

... which matches cpython correctly:

anl@ANL2-LAP:~$ python
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> raise SystemExit(-1)
anl@ANL2-LAP:~$ echo $?
255

Then it's a separate question how to handle this on bare-metal targets. But for now we shouldn't change the unix behaviour on this.

Well it looks to me like this should be treated as a soft reset.

micropython/extmod/modmachine.c

Line 66 in 358e501

mp_raise_type(&mp_type_SystemExit);

The current push also works this way:

anl@STEP: ~/micropython $ mpremote
Connected to MicroPython at /dev/ttyACM0
Use Ctrl-] or Ctrl-x to exit this shell
MicroPython v1.24.0-preview.104.g161a9c3097.dirty on 2024-07-09; WEACTV20_F411 with STM32F411CE
Type "help()" for more information.
>>> raise SystemExit
MPY: sync filesystems
MPY: soft reboot
MicroPython v1.24.0-preview.104.g161a9c3097.dirty on 2024-07-09; WEACTV20_F411 with STM32F411CE
Type "help()" for more information.
>>>

dpgeorge · 2024-07-18T04:03:19Z

There was an associates variable here pyexec_system_exit which I did a global search for - it was never actually being used (anymore) so I've stripped that out as well.

Right, it would be a nice clean up to get rid of that variable, and also change the behaviour of SystemExit on bare-metal ports to match unix/CPython.

Because this is a change in user-facing behaviour, I've split that change out in #15486.

andrewleech · 2024-08-13T12:46:15Z

This has been rebased to master, taking into account the SystemExit stuff already merged.

andrewleech · 2025-06-02T05:15:51Z

@dpgeorge this should be ready for re-review.

This PR does not include the unhandled exception hooks that you had in #6008 but I feel that should be a separate feature / pr anyway if it's still needed by others?

There's a minor compatibility update to aiorepl in micropython/micropython-lib#1016 which has been tested with this PR and seems to behave as expected. Without that change aiorepl needs crtl-enter to execute / send a line.

shared/runtime/pyexec.c

dpgeorge · 2025-07-02T04:50:15Z

@andrewleech can you please check the CI failures? Not sure what's going on there.

dpgeorge · 2025-07-02T04:57:52Z

Note: the code size increase across most of the ports is due to improved support for MICROPY_PY___FILE__: the variable __file__ is now set whenever a script is executed via pyexec. So now we have:

__file__ is already set on import of scripts (eg import foo, then __file__ would be foo.py), that doesn't change with this PR
when executing boot.py/main.py from the filesystem, __file__ is set to either "boot.py" or "main.py" respectively (this is not done for frozen boot.py/main.py though)
doing mpremote run test.py now sets __file__ to "<stdin>"
at the REPL, __file__ now exists and is the name of the most recently executed file

I think that's a good improvement, makes things more consistent.

shared/runtime/pyexec.c

ports/unix/Makefile

ports/unix/main.c

ports/unix/mphalport.h

ports/unix/main.c

This improves repl usage consistency across ports. Only enabled when MICROPY_USE_READLINE == 1 (default). Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

When MICROPY_PY___FILE__ is enabled and parsing file input, set the global __file__ variable to the source filename. This matches the behavior of the unix port and provides the current filename to the executing script. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

When MICROPY_PYEXEC_COMPILE_ONLY is enabled and global mp_compile_only is True code is compiled but not executed. Also add comprehensive tests for compile-only functionality covering both successful compilation and syntax error detection. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

Provides support for command line -X compile-only option on unix port. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

Add a general normalize_newlines() function that handles newline variations (\\r\\r\\n, \\r\\n) to \\n while preserving literal \\r characters that are part of test content. This provides a robust solution for cross-platform test compatibility, particularly addressing PTY double-newline issues that can occur with some terminal implementations. The function is applied to all test output before comparison, eliminating platform-specific newline issues. Includes a unit test to verify the normalization behavior. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

andrewleech · 2025-07-15T23:20:06Z

@dpgeorge I've addressed your comments and added a couple of basic tests for systemexit - the outputs aren't very interesting but basically show a number in systemexit is shown as a failed (crash) test. Zero or no arg is a successful result.

dpgeorge · 2025-07-16T05:01:40Z

shared/runtime/pyexec.c

@@ -141,8 +160,13 @@ static int parse_compile_execute(const void *source, mp_parse_input_kind_t input

        // check for SystemExit
        if (mp_obj_is_subclass_fast(MP_OBJ_FROM_PTR(((mp_obj_base_t *)nlr.ret_val)->type), MP_OBJ_FROM_PTR(&mp_type_SystemExit))) {
-            // at the moment, the value of SystemExit is unused
-            ret = PYEXEC_FORCED_EXIT;
+            // Extract SystemExit value


It'd be worth copying the comment over from the unix port's implementation of this:

// None is an exit value of 0; an int is its value; anything else is 1

dpgeorge · 2025-07-16T06:27:19Z

ports/unix/main.c

+        return ret; // SystemExit with exit value in lower 8 bits
+    } else {
+        return 1; // exception
+    }
 }

 static int do_str(const char *str) {


It would be possible to replace the execute_from_lexer() below with pyexec_vstr() (just make a temporary vstr on the C stack and populate it with str and strlen(str)).

Might be worth doing that as part of this PR?

Josverl · 2025-07-17T20:49:14Z

Is there a specific reason for

this is not done for frozen boot.py/main.py though

andrewleech · 2025-07-17T22:47:36Z

Is there a specific reason for

this is not done for frozen boot.py/main.py though

I'm assuming the __FILE__ setting section is in a different pathway to the "run from frozen" code path. This wasn't intentional certainly. It might not be that easy to fix in a generic way though as I presume frozen code will be referenced by python path rather than the typical file path that's used here.

dpgeorge · 2025-07-18T06:20:46Z

I'm assuming the __FILE__ setting section is in a different pathway to the "run from frozen" code path. This wasn't intentional certainly. It might not be that easy to fix in a generic way though as I presume frozen code will be referenced by python path rather than the typical file path that's used here.

Yes, that's correct. There would need to be extra code added to set __FILE__ for execution of frozen boot/main.

andrewleech · 2025-07-18T06:30:10Z

Yes, that's correct. There would need to be extra code added to set FILE for execution of frozen boot/main.

I'm thinking that would be worth doing too for consistency at runtime, would it make sense to just report these frozen modules as <module name>.py ?
Or would it be better as frozen:<module name>.py or similar so users can tell if a frozen version is being used?

dpgeorge · 2025-07-18T06:49:52Z

I'm thinking that would be worth doing too for consistency at runtime, would it make sense to just report these frozen modules as <module name>.py ?

There's code in py/builtinimport.c that does it already for import of frozen modules. Following that logic would be best.

Josverl · 2025-07-18T10:22:21Z

as a side note; if all frozen modules can indeed be identified using the pattern <module name>.py that would also allow them to be excluded by default from sys.settrace tracing,

dpgeorge added the port-unix label Oct 25, 2023

andrewleech force-pushed the unix_pyexec branch 2 times, most recently from e4cc4f5 to 2c1c7d8 Compare October 26, 2023 01:38

andrewleech force-pushed the unix_pyexec branch from 2c1c7d8 to 55032e9 Compare October 26, 2023 01:44

jimmo mentioned this pull request Oct 26, 2023

ports/unix: Add embedded mode. #12807

Open

andrewleech force-pushed the unix_pyexec branch 2 times, most recently from 12f5770 to 7a0825f Compare November 16, 2023 09:37

stinos reviewed Nov 16, 2023

View reviewed changes

andrewleech force-pushed the unix_pyexec branch from 7a0825f to ad27be1 Compare November 16, 2023 10:24

andrewleech force-pushed the unix_pyexec branch from ad27be1 to 02fb334 Compare November 16, 2023 21:53

mattytrentini added the maintainer-queue label Feb 27, 2024

dpgeorge added this to the release-1.24.0 milestone Mar 26, 2024

andrewleech force-pushed the unix_pyexec branch 3 times, most recently from d08fe6e to 554ba06 Compare March 26, 2024 01:01

andrewleech force-pushed the unix_pyexec branch from 554ba06 to 03e992d Compare July 9, 2024 06:58

dpgeorge mentioned this pull request Jul 18, 2024

Make raising SystemExit do a soft reset on bare-metal targets #15486

Merged

andrewleech force-pushed the unix_pyexec branch from 03e992d to 95bce29 Compare August 13, 2024 12:26

andrewleech force-pushed the unix_pyexec branch from 82a1c84 to 7e084a5 Compare May 31, 2025 23:51

andrewleech force-pushed the unix_pyexec branch from 7e084a5 to 701609f Compare June 3, 2025 02:53

dpgeorge reviewed Jun 27, 2025

View reviewed changes

shared/runtime/pyexec.c Outdated Show resolved Hide resolved

andrewleech force-pushed the unix_pyexec branch from 701609f to b0bd43e Compare July 1, 2025 11:50