Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.38%. Comparing base (e993f53) to head (c7182e4).
Report is 59 commits behind head on master.

@@            Coverage Diff             @@
##           master   #17089      +/-   ##
==========================================
- Coverage   98.41%   98.38%   -0.03%     
==========================================
  Files         171      171              
  Lines       22210    22239      +29     
==========================================
+ Hits        21857    21880      +23     
- Misses        353      359       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Windows - Git Bash

Some test cases pass :

./test_eval_exec_run.sh: OK
./test_recursive_cp.sh: OK
./test_resume.sh: OK

all others fail

Tests fail when

• uses {$TMP} or assumes a host path convention

• comparison of a sha256sum is slightly different with an additional asterix.
  The default mode is to print a line with checksum, a space, a character indicating input mode ('*' for binary, ' ' for text or where binary is insignificant), and name for each FILE`

  can be mitigated with 'sha256sum -t'

• the ramdisk creation in test_mip_local_install.sh fails while the one in test_filesystem.sh works.
  it is trying to create file in a location where it should not attempt write at all.

     > cp ${TMP}/package :
     ./test_mip_local_install.sh: FAIL
     2,3c2,11
     < mkdir :C:/Program Files/Git/__ramdisk/lib
     < mpremote: mkdir: C:/Program Files/Git/__ramdisk/lib: No such file or directory.
  

  I would suggest ramdisk.py that should not be created on the fly, but be part of the testsuite and referenced from both scripts.

@Josverl could you test the latest push? I've now majorly rewritten most of the test runner script with far more paranoid/maximalist quoting, and updated it to use a separate results directory to match the behavior of the main test suite (with a .gitignore line matching the way the main test suite gets its results ignored).

This also reduces some of the jank by inverting the way the tests are compared -- that is, instead of trying to use sed to replace instances of the temporary directory path with the string ${TMP} in the .out file, it instead uses envsubst to generate a copy of the .exp file with TMP expanded.

Also updated, it uses a diff compare mode that ignores carriage returns, and outputs the failure diffs at the end in unified-diff format the same way the main test suite does.

I've also fixed the tests that were individually broken due to insufficient quoting, and added --text to the sha256sum invocations in test_filesystem.sh to make them more windows compatible.

initial results : recursive now also FAILS

./test_eval_exec_run.sh: OK
./test_filesystem.sh: FAIL
./test_mip_local_install.sh: CRASH
./test_mount.sh: FAIL
./test_recursive_cp.sh: FAIL
./test_resume.sh: OK

common issues :
There is a common patter to setup a ramdisk. This should be a common fixture.
Instead it is setup in multiple slightly different ways , and only one (the filesystem test ) works on Windows.

I think the bash syntax: ${target@Q} does not work or work the same. Simpler to just use the correct delimiters instead.

I think the diffing on errors would be better to understand when $DIFF "${result}.exp" "${result}.out"
That shows the extra's as + and the missing lines as - , which makes more sense to my simple Dutch brain.

main problems in filesystem test

• Windows path <> Posix Path
• the "editor" test via sed fails / stops the entire script

Details : https://gist.github.com/Josverl/32de060bfa072f19ea55ad326aca6403

🪲While debugging I found this bug 🦗
Git bash + mpremote result in a transport error on mpremote resume mkdir /ramdisk/lib ( no leading :)
#17093

The same command works just fine in Windows powershell or even an old style command prompt.
soo, just testing on Git bash - does not appear to cover enough ( id say that about 2% of windows users use git bash for mpremote)
Even with a perfect bash script , that would cover only about 20% of the end user computing scenarios.

Personally I think that test are best written using a test framework, such as pytest.
And that can handle .exp files, and has much more flexibility and parametrization and fuzzing capabilities, code coverage etc etc that will take so much more time to try to accomplish using bash, or any other shell language for that matter.

https://github.com/Josverl/micropython/blob/mpr/pytest/tools/mpremote/tests/test_filesystem.py

I really need to spend some more time getting my Windows build environment working so we're not just playing telephone on this.

There is a common patter to setup a ramdisk. This should be a common fixture. Instead it is setup in multiple slightly different ways , and only one (the filesystem test ) works on Windows.

I'll try to see about refactoring this.

I think the bash syntax: ${target@Q} does not work or work the same. Simpler to just use the correct delimiters instead.

You're right that that that doesn't address all possible escaping problems, but there is one remaining issue I also just spotted even with that, in that it's actually the missing r prefix to get python to not interpret backslashes as escapes.

I think the diffing on errors would be better to understand when $DIFF "${result}.exp" "${result}.out" That shows the extra's as + and the missing lines as - , which makes more sense to my simple Dutch brain.

Good catch, the other test runners also diff their .exp and .out in the order you suggest, so that'll make it more consistent.

🪲While debugging I found this bug 🦗 Git bash + mpremote result in a transport error on mpremote resume mkdir /ramdisk/lib ( no leading :) #17093

Could you run that command with set -x enabled? That'll print a trace as commands run that includes the final interpolated value of everything.

just testing on Git bash - does not appear to cover enough ( id say that about 2% of windows users use git bash for mpremote) Even with a perfect bash script , that would cover only about 20% of the end user computing scenarios.

Agreed, this should also at least support .ps1 test cases, and probably .cmd test cases too.
This might also make it easier to test windows-y behaviors, too, since there's a port of powershell for linux it could use.

Personally I think that test are best written using a test framework, such as pytest. And that can handle .exp files, and has much more flexibility and parametrization and fuzzing capabilities, code coverage etc etc that will take so much more time to try to accomplish using bash, or any other shell language for that matter.

Almost definitely. There's a saying that, if you're using arrays in bash, it's time to rewrite it in a different language... and the rewrite technically does, even if just barely.

As a shell tool, it makes sense to have the test case files for mpremote in shell languages, but I do think it'll be useful to rewrite the test runner in python so it can reuse and integrate with parts of the other test runners.

I want to get to a working 'good' config first before refactoring further, though -- it's not just the runner that's broken, many of the actual tests are too.

Is this ready for another test run?

Is this ready for another test run?

Not really, no; just rebased this in order to more easily experiment with #12802 --- since its patch theoretically enables raw repl mode on the unix port, and might make it possible to run the mpremote tests as part of CI.

@Josverl Just rewrote the test driver to use more-or-less the same test code as the main suite's run-tests.py.

I've tested it against my own boards and can confirm it works correctly on unix; and since it's now just doing exactly what the main test suite does it should be compatible with windows too. Theoretically, hopefully you can confirm this.

It now also looks for and tries to run test_*.bat and test_*.ps1 scripts in addition to test_*.sh scripts, though I've not been able to test that just yet.

Still need to test the merge with #12802 or #16141 to confirm that the micropython socket server it sets up actually works though, and add the relevant CI jobs for it.

On a separate note, does the Windows port of micropython support raw repl mode? If it's possible to add an mpremote CI job for windows that'd be pretty useful to have too, given that it's been one of the tool's main pain points.