Running Micropython in a CPython subprocess on Unix #12821
Replies: 16 comments 3 replies
-
|
Hi @kdschlosser this certainly can work, in the past I used a custom cpython unit test runner script that would execute a unix build of micropython with subprocess. These days I don't bother with cpython though, I just write my tests using the micropython-lib version of unittest and run everything on micropython. I'm not sure I understand your full issue though, if you run your compiled copy of micropython from a terminal shell, do you get the repl and can you run code Magnitsky from there? |
Beta Was this translation helpful? Give feedback.
-
|
Using Now you need to run 'Hello.py' in the same directory as 'hello.py', unless you have included the absolute path to the file. Send us any error messages to help us understand what went wrong. |
Beta Was this translation helpful? Give feedback.
-
|
Now If I am not mistaken in those tests you are instructing MicroPython to run a script. code is not being passed to the repl using stdin of the subprocess. The repl works fine if I launch MicroPython from the shell. I get a prompt and it prints as it should. I do not however get the repl when I launch it from a subprocess. This is what I am doing to run MicroPython from a subprocess import subprocess
import sys
import os
ctrl_c = b'\x03' # 2 times to exit any running code
ctrl_d = b'\x04' # exit paste mode committing what has been pasted
ctrl_e = b'\x05' # enter paste mode
paste_mode_prompt = b'==='
repl_prompt = b'>>>'
code = b'''\
for i in range(1000):
print(i)
'''
p = subprocess.Popen(
[os.path.expandvars('$HOME/lv_micropython/ports/unix/build-standard/micropython')],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
)
entered_repl = False
entered_paste_mode = False
while p.poll() is None:
for line in iter(p.stdout.readline, b''):
line = line.strip()
if line:
sys.stdout.write(line.decode('utf-8') + '\n')
if line == repl_prompt:
if not entered_repl:
p.stdin.write(ctrl_e)
p.stdin.flush()
entered_repl = True
else:
p.stdin.close()
elif line == paste_mode_prompt:
if not entered_paste_mode:
p.stdin.write(code)
p.stdin.flush()
entered_paste_mode = True
else:
p.stdin.write(ctrl_d)
p.stdin.flush()
if not p.stdout.closed:
p.stdout.close()
if not p.stderr.closed:
p.stderr.close()
sys.stdout.flush()I get zero output from the subprocess. I should be seeing which I am not getting at all when I run it as a subprocess. I am not sure why. |
Beta Was this translation helpful? Give feedback.
-
|
It seems like I am not able to interact with MicroPython due to it behaving in a different manner then when it is run from the shell. |
Beta Was this translation helpful? Give feedback.
-
|
Looks like Unix port checks that it's running with a tty before entering repl Line 732 in 2fda94c Maybe if you run subprocess with shell=True it'll behave like you want.
|
Beta Was this translation helpful? Give feedback.
-
|
I tried using the shell argument and it still didn't behave. |
Beta Was this translation helpful? Give feedback.
-
|
That really stinks that it is made like that. I wanted to be able to feed in code via the repl and realtime monitor things like memory while performing the unit tests. IDK if you have heard of LVGL or not. LVGL is a graphics framework and there is a binding that is made so it will run in MicroPython. I wanted to do unit tests in a manner that would kind of simulate an MCU. I was going to have it dump the drame buffer output to stdout so that it could be read using subprocess without having a large amountof additional overhead like adding some other means of transferring the information between the 2 processes. I will make a feature request and see if that is something that could be turned off using a build argument this way it makes it a bit more flexible. It might get added, ya never know. There is an applicable use case for it. |
Beta Was this translation helpful? Give feedback.
-
|
OH! it looks like there is a way to do it. Have to set MICROPYINSPECT to 1 in the environment. |
Beta Was this translation helpful? Give feedback.
-
|
yessir that did the trick. Awesome! I am happy with that. |
Beta Was this translation helpful? Give feedback.
-
|
bonjour , |
Beta Was this translation helpful? Give feedback.
-
|
May I suggest two easier alternatives. Please set MICROPYINSPECT before running the script. Thanks Kevin! - for figuring this out. The first of these uses subprocess.communicate(). The second is a modification based on an idea from Kevin's test script. The two scripts are as follows.
#
# my_test_3.py
from subprocess import Popen, PIPE
p = Popen(['micropython'], stdin=PIPE, stdout=PIPE, stderr=PIPE, text=True)
# List of cmd input strings
inputs = [
"x=2*3\n",
"print(x)\n",
"print(x**4)\n",
'''
for i in range(10):
print(i)
''',
]
# Sending each cmd input string one after the other
for input_str in inputs:
p.stdin.write(input_str)
p.stdin.flush() # Flush the input buffer
# Read the output from the subprocess
res, err = p.communicate()
print("Output:-")
for r in res.split('\n'):
print(r)
#
# my_test_1.py
from subprocess import Popen, PIPE
import sys
cmd = [
"x=23\n",
"print(x)\n",
"print(x**24)\n",
'''
for i in range(10):
print(i)
''',
'\n', # need extra blank line to terminate
]
with Popen(["micropython"],
stdout=PIPE,
stderr=PIPE,
stdin=PIPE,
text=True) as proc:
inx=0
ww = ''
while 1:
w = proc.stdout.read(1)
if w != '\n':
ww+=w
if ww.find(">>>") >= 0:
print(ww, end='')
ww = ''
if inx < len(cmd):
proc.stdin.write(cmd[inx])
proc.stdin.flush()
inx+=1
else:
print('DONE')
break
else:
print(ww)
ww = ''I hope they are of some use to someone. |
Beta Was this translation helpful? Give feedback.
-
|
no worries m8, Thanks goes to @andrewleech for providing the link to the code that was blocking the repl because of using isatty. I clicked the link and say there was another condition and scrolled up a little bit to locate what that condition exactly was. once I saw it was a check for an environment variable i knew it was going to be good to go from that point. So TY @andrewleech for tracking down the code that was blocking it from working. |
Beta Was this translation helpful? Give feedback.
-
|
thank's for reply. |
Beta Was this translation helpful? Give feedback.
-
|
Give me a few to locate the test code I used. I will pass it on over to you. I know it works 100% by just running the script from CPython. The only thing you will have to change in the script is the path to MicroPython |
Beta Was this translation helpful? Give feedback.
-
|
found it. no need to set any environment variables or anything like that. change the path and run the script. If your build of micropython is good then it should work. import subprocess
import sys
import os
MICROPYTHON_PATH = '$HOME/lv_micropython/ports/unix/build-standard/micropython'
ctrl_c = b'\x03' # 2 times to exit any running code
ctrl_d = b'\x04' # exit paste mode committing and running what has been pasted
ctrl_e = b'\x05' # enter paste mode
paste_mode_prompt = b'==='
repl_prompt = b'>>>'
code = b'''\
for i in range(1000):
print(i)
'''
os.environ['MICROPYINSPECT'] = '1'
p = subprocess.Popen(
[os.path.expandvars(MICROPYTHON_PATH)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
env=os.environ
)
entered_repl = False
entered_paste_mode = False
output = b''
while p.poll() is None:
char = p.stdout.read(1)
if char != b'':
output += char
if b'999' in output:
sys.stdout.write(output.decode('utf-8'))
sys.stdout.flush()
p.stdin.close()
elif repl_prompt in output:
sys.stdout.write(output.decode('utf-8'))
sys.stdout.flush()
output = output.split(repl_prompt, 1)[-1]
if not entered_repl:
p.stdin.write(ctrl_e)
p.stdin.flush()
entered_repl = True
elif paste_mode_prompt in output:
sys.stdout.write(output.decode('utf-8'))
sys.stdout.flush()
output = output.split(paste_mode_prompt, 1)[-1]
if not entered_paste_mode:
p.stdin.write(code)
p.stdin.write(ctrl_d)
p.stdin.flush()
entered_paste_mode = True
if not p.stdout.closed:
p.stdout.close()
if not p.stderr.closed:
p.stderr.close()
sys.stdout.flush() |
Beta Was this translation helpful? Give feedback.
-
|
thank's |
Beta Was this translation helpful? Give feedback.
-
|
Yeah that's because the prompts don't end in a new line. They actually end with a space. It's just best to dump the output to a buffer 1 character at a time and then check the buffer for what you need and split the buffer a single time shaving off the front of it before the data that you looked for. To be able to tell where the code I pasted in ends and any output begins I made it so that # end is added to the end of the code that I past in. once the ctrl code get sent and the code runs there is not going to be an easy way to know what is what unless you put "=== " at the beginning of each line of the code you pasted in and then you can tell where the codes end in the output. so by adding the # end on the last line of code that is sent when you read the output from MicroPython that is what you look for. when it is gotten then you can send the ctrl code to exit paste mode which will run the code you just pasted in.. that way you know any output that happens from that point until you get another repl code prompt is going to be from the code you pasted in. |
Beta Was this translation helpful? Give feedback.
-
|
I have already hammered out a rough framework to run unit tests using CPython and passing data between MicroPython and CPython for the tests. The reason why I wanted to go this route is it is an easy way to send GUI frame buffer data and be able to save the invalid data to an output png file using PIL. This way the error image can be a build artifact in a CI and visually inspected for the issues with it. using the stdout and stdin built into MicroPython is already set up and how it is written makes it fast and not too memory intensive. It simplifies the entire process which makes things easier from a maintenance standpoint. I also already have a plethora of examples that are written that can now be used to run the tests with and they don't have to be modified in any way to be able to do that. I run each example as a test case and a subprocess is launched for each test case, this ensure a clean sandbox environment for each example to run in. because the subprocess is persistent between each test in a test case I am able to write the code needed to check input and output from the different objects. 1/2 the work is now already done. |
Beta Was this translation helpful? Give feedback.
-
|
I am going to build a more complete framework for using CPython to run unit tests on code that is written for MicroPython. This can be a handy thing to use. I am also going to build a framework specifically made for passing data back and forth this way as well. MicroPython can be packaged with other software if wanting to provide realtime scripting support for whatever reason. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
OK so this is the skinny on what I am wanting to do but I seem to be having an issue with it working.
I am building MicroPython with a user module and I want to be able to text if the user module is functioning properly. How I wanted to do this was by creating a CPython unittest script that would open a subprocess to MicroPython and I could send over the code to execute in the subprocess. For whatever reason I am not getting any output when I run Micropython. I have trued running a shell and passing the command to execute micropython and I have tried directly sending the micropython command to subprocess. It doesn't want to run. I am not sure if there is some kind of mechanics in MicroPython that mucks it up from being able to be used in this manner.
Any information would be helpful.
Thanks.
--Kevin
Beta Was this translation helpful? Give feedback.
All reactions