Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.54%. Comparing base (037f2da) to head (ef8282c).
Report is 3 commits behind head on master.

@@           Coverage Diff           @@
##           master   #16994   +/-   ##
=======================================
  Coverage   98.54%   98.54%           
=======================================
  Files         169      169           
  Lines       21890    21890           
=======================================
  Hits        21571    21571           
  Misses        319      319           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

can someone give this a testrun on a Host OS + physical device just to make sure ?

• Windows using Pytest
• Windows + WSL2 - using ./run-mpremote-tests.sh
• Linux
• Mac

I'll test run this in my environment tomorrow -- it's WSL that I've mounted my devices into with usbipd, but that's still linux enough to count lol.

I needed to add a special case to avoid confusing errors:

• rm :/ - should not attempt to remove the root
• rm :/ramdisk - mounted file systems such as 'ramdisk' used in the test cannot be removed , but they are currently undiscoverable.

split in multiple commits ( code / tests / docs )
tested in wsl2 - tests show success
tested in Git Bash - Success but tests show a False Negative due to platform differences

One other detail; with the --verbose flag this should output a rm :/file line for each file and rm -r :/dir for each directory as it's being removed.

rm shouldn't mysteriously fail to remove it just because it has a name it considers special

magic is never nice. A Try / Except - continue is somewhat better, but has a risk hide other special cases.

I do not want the CLI to terminate on hitting a mountpoint , as that requires knowledge that a user cannot have with current firmwares. Raising a Exception is useful in an API , not to send information to a user in a CLI.
I can log a info or verbose message.

so that the command can still remove directories that come after a mountpoint in their parent directory

That already works en is part of the tests

with the --verbose flag this should output a rm :/file line for each file and rm -r :/dir for each directory

that would not be consistent with cp -r that does not do that either, and I do not think that is useful for rm -r either.

$ mpremote cp -r --verbose ports/zephyr/boards/ : 
cp ports/zephyr/boards/ :
# copied multiple files

I do not want the CLI to terminate on hitting a mountpoint , as that requires knowledge that a user cannot have with current firmwares. Raising a Exception is useful in an API , not to send information to a user in a CLI. I can log a info or verbose message.

Sure; the main point is that the way to detect this EPERM error is to add it to the code that detects the rest of the device-side errors, and raise the appropriate exception that follows the pattern.

Then in do_filesystem_recursive_rm you can try/catch for a PermissionError around the state.transport.fs_rmdir(path) call in order to output the appropriate message and continue.

that would not be consistent with cp -r that does not do that either, and I do not think that is useful for rm -r either.

Oh, indeed? IMO it'd be good to make both subcommands' behavior more closely resemble GNU coreutils, rsync, etc -- but you're right that that's out of scope for this PR; that can be the subject of a separate PR and discussion later.

Regarding EINVAL vs EPERM, per the vfs source code EPERM is the error that theoretically should be happening for invalid operations on or between mountpoints (including /), e.g:

https://github.com/micropython/micropython/blob/master/extmod/vfs.c#L120-L121
https://github.com/micropython/micropython/blob/master/extmod/vfs.c#L266-L267
https://github.com/micropython/micropython/blob/master/extmod/vfs.c#L490-L491

EINVAL does happen, but only for trying to unmount a path that's not a mountpoint:

https://github.com/micropython/micropython/blob/master/extmod/vfs.c#L303-L305

So I'm quite curious to hear what port/board you're using. I know some ports include their own filesystem components, perhaps there's another issue here to be fixed later.

In the mean time though, for portability it seems reasonable to just generically catch OSErrors of all types when removing the directories.

Tested on 5 different ports, all raise the same error.
It can't hurt to catch the other error, but would like a way to trigger that error IRL, if only to verify.
Perhaps with an SD card, rather than a ramdisk ?

Tested on 5 different ports, all raise the same error. It can't hurt to catch the other error, but would like a way to trigger that error IRL, if only to verify. Perhaps with an SD card, rather than a ramdisk ?

It seems you're correct; I've misunderstood the error code here. The EPERM happens when attempting to remove the filesystem root /, a case that your code already filters out; removing other mounts does indeed produce EINVAL.

Thanks for confirming.

That clears up all review comments AFAIKT.

The CI error on Mac appears to be a flakey build that I can't re-run

That clears up all review comments AFAIKT.

One thing I'd consider still mandatory to fix, is the dead code introduced in the latest push at tools/mpremote/mpremote/transport.py:68, per this comment.

(There's also the choice of python exception type mapping for EINVAL, per these comments; but that's a more minor nitpick.)

For regularity sake, I also think that there should be an error message on failing to remove the root directory, even when the path filtering on the host side avoids actually executing the command on the device.

i.e. rm -r :/ in those tests should output:

rm -r: could not remove :/ramdisk
rm -r: could not remove :/

And should have an error exit code, not 0.

You have a point on OSError.
Code and messages adjusted

As I indicated before : I disagree that trying to remove all files and folder down from root should result in an error, as that would reduce usability too much.

@peterhinch , @mattytrentini,
Do you have a preference whether or not an error should be raised on mpremote -rm -r :/ ?

I would prefer no error: clearing the filesystem is a normal thing to do, and users of rm -r should be aware that (as on Unix) it has the potential to be drastic.

The point of reporting that error isn't to error in a way that fails to remove the contents of /, just to report that the directory / itself still exists despite the ostensible request that the directory be removed. (i.e. consistent with a failure to remove any other directory the user might name). The command should still wipe the contents of the directory before it errors; I am not suggesting kneecapping the functionality to prevent this use case; this is purely a detail about the command's exit status and error output.

What I'm describing is the standard behavior for rm -r /; that is, the behavior specified by the applicable POSIX standard and implemented by both GNU coreutils and BusyBox (see below for demonstrations). If rm exits 0, that means the path, whatever it is, should no longer exist. If it still does -- even if the reason is that that path is / -- then rm needs to log an error and eventually exit >0.

GNU coreutils 9.1

user@host:~$ podman run -it debian
root@ecdfda5df5ad:/# mkdir /jail
root@ecdfda5df5ad:/# cp -r /lib /jail/lib
root@ecdfda5df5ad:/# cp -r /lib64 /jail/lib64
root@ecdfda5df5ad:/# cp -r /bin /jail/bin
root@ecdfda5df5ad:/# cp -r /usr /jail/usr
root@ecdfda5df5ad:/# ls /jail
bin  lib  lib64  usr
root@ecdfda5df5ad:/# chroot /jail rm -r --no-preserve-root /
rm: cannot remove '/': Device or resource busy
root@ecdfda5df5ad:/# echo $?
1
root@ecdfda5df5ad:/# ls /jail
root@ecdfda5df5ad:/# 

BusyBox v1.37.0

user@host:~$ podman run -it alpine
/ # mkdir /jail
/ # cp -r /lib /jail/lib
/ # cp -r /bin /jail/bin
/ # ls /jail
bin  lib
/ # chroot /jail rm -r /
rm: can't remove '/': Resource busy
/ # echo $?
1
/ # ls /jail
/ #

POSIX: IEEE Std 1003.1-2024, Shells & Utilities, Part 3: rm — remove directory entries

For each file the following steps shall be taken:
....
4. rm shall perform actions equivalent to the remove() function defined in the System Interfaces volume of POSIX.1-2024 called with a pathname of the current file used as the path argument. .... If the actions fail for any reason, rm shall write a diagnostic message to standard error, do nothing more with the current file, and go on to any remaining files.

EXIT STATUS
The following exit values shall be returned:
0 : All requested directory entries ... were successfully deleted.

[Regarding the -f option,] it is less clear that error messages regarding files that cannot be unlinked (removed) should be suppressed. Although this is historical practice, this volume of POSIX.1-2024 does not permit the -f option to suppress such messages.

Do you have a preference whether or not an error should be raised on mpremote -rm -r :/ ?

I think @AJMansfield presents a pretty compelling argument why there should be an error - and I think it also satisfies @peterhinch's expectation that the folders contents will be removed (just not the directory itself).

Personally I'm not particularly invested, as long as I can delete the contents!

Regarding errors when trying to remove the root: some boards/ports have the internal filesystem mounted at /flash (eg stm32). On such boards if you do mpremote rm -r :/ then it'll go through all mounted partitions and delete everything from them and then try to delete the mount points. That's perhaps not what you want.

Maybe one way to support boards with / and /flash as the main filesystem is instead to allow:

mpremote rm -r :

which would use the current directory as the target and remove everything from it, but crucially it would not attempt to remove the directory itself. So that's kind of like rm -rf * on unix. Implementing that is pretty easy and would give a generic way to remove all files in the main working directory, without issuing any errors.

Then it would still support mpremote rm -r :/ but in that case try to remove / and print an error about it.

Maybe one way to support boards with / and /flash as the main filesystem is instead to allow:
mpremote rm -r :
which would use the current directory as the target and remove everything from it, but crucially it would not attempt to remove the directory itself. So that's kind of like rm -rf * on unix. Implementing that is pretty easy and would give a generic way to remove all files in the main working directory, without issuing any errors

I like this approach as it will allow users to do cleanup in a single command without throwing errors.
Let me try to make this work.

WRT raising error on root deletion: on ubuntu in docker (docker container run -it ubuntu bash) the following command does not even try to delete the root ⚠️ rm -rfv --no-preserve-root /. ⚠️don't try this outside a throwaway docker / vm

I now find a desire for a mpremote fs tree command , let's land this one first.

WRT raising error on root deletion: on ubuntu in docker (docker container run -it ubuntu bash) the following command does not even try to delete the root ⚠️ rm -rfv --no-preserve-root /. ⚠️don't try this outside a throwaway docker / vm

Interesting! It looks like coreutils rm skips trying to remove a parent directory if it errored trying to remove any of its children. This happens with any directory, not just /.

user@host:~$ podman run -it --mount=type=tmpfs,dst=/dir/fs --mount=type=tmpfs,dst=/dir/subdir/fs ubuntu bash
root@9c63b3159882:/# rm -rfv /dir
rm: cannot remove '/dir/fs': Device or resource busy
rm: cannot remove '/dir/subdir/fs': Device or resource busy
root@9c63b3159882:/# 

Note that there was never an attempt to remove /dir/ or /dir/subdir/; attempting to remove / never tries to delete it as a directory because by then it has already failed to remove other sysfs items normally mounted there e.g. /proc, /dev.

Using a chroot without any mounted filesystems to avoid this error (as in my demonstrations), does result in an attempt and error with removing /.

Note that this seems to be implementation-specific. BusyBox's behavior is somewhat different:

user@host:~$ podman run -it --mount=type=tmpfs,dst=/dir/fs --mount=type=tmpfs,dst=/dir/subdir/fs alpine
/ # rm -rfv /dir
rm: can't remove '/dir/fs': Resource busy
rm: can't remove '/dir/subdir/fs': Resource busy
removed directory: '/dir/subdir'
removed directory: '/dir'
/ # echo $?
1
/ # ls /dir
fs      subdir
/ # 

It still exits nonzero -- but curiously, it log a success at removing the parent directories even though they still exist afterward.

I've reported this potential issue to BusyBox here: https://bugs.busybox.net/show_bug.cgi?id=16330

Current iteration works :

• without specifying a path , effectively using the MCU's cwd
• with a relative path to a folder
• with an absolute path to a subfolder
• with an absolute path to a mount point (not a terminating error, -v shows "skipped: '/ramdisk' (vfs mountpoint)")
• with a relative path to a mount point (not a terminating error, -v shows "skipped: '/ramdisk' (vfs mountpoint)")
• with the absolute root (Raises error) and exitcode = 1

$ mpremote rm -rv : 
rm -r :
removed: './a.py'
removed: './b.py'
removed: './package/subpackage/__init__.py'
removed: './package/subpackage/y.py'
removed directory: './package/subpackage'
removed: './package/__init__.py'
removed: './package/x.py'
removed directory: './package'

$ mpremote rm -rv : 
rm -r :/
removed: '/ramdisk/a.py'
removed: '/ramdisk/b.py'
removed: '/ramdisk/package/subpackage/__init__.py'
removed: '/ramdisk/package/subpackage/y.py'
removed directory: '/ramdisk/package/subpackage'
removed: '/ramdisk/package/__init__.py'
removed: '/ramdisk/package/x.py'
removed directory: '/ramdisk/package'
removed: '/ramdisk/package2/subpackage/__init__.py'
removed: '/ramdisk/package2/subpackage/y.py'
removed directory: '/ramdisk/package2/subpackage'
removed: '/ramdisk/package2/__init__.py'
removed: '/ramdisk/package2/x.py'
removed directory: '/ramdisk/package2'
skipped: '/ramdisk' (vfs mountpoint)
mpremote: rm -r: cannnot remove :/ Operation not permitted

Also supports -v , and exit = 1 on attempt to remove root.
tested on stm32 with / without SDcard , ESP32 with & without SD card , and SAMD with several versions of MicroPython

(not yet pushed, waiting for feedback as I want to limit churn )

I have an alternative implementation of do_filesystem_recursive_rm I've been playing with as well, that combined with #17090 would make the way errors are handled much more consistent:

def do_filesystem_recursive_rm(state, path, args):
    if state.transport.fs_isdir(path):
        for entry in state.transport.fs_listdir(path):
            try:
                do_filesystem_recursive_rm(state, _remote_path_join(path, entry.name), args)
            except OSError as er:
                from .main import _PROG

                # Make sure existing stdout appears before the error message on stderr.
                sys.stdout.flush()
                print(
                    "{}: rm: {}: {}.".format(_PROG, er.strerror, os.strerror(er.errno)),
                    file=sys.stderr,
                )
                sys.stderr.flush()
        state.transport.fs_rmdir(path)
    else:
        state.transport.fs_rmfile(path)

    if args.verbose:
        from .main import _PROG

        print("{}: rm: {}".format(_PROG, path))

This restructures it to specifically only catch the OSError generated by the recursive child calls, and intentionally allows any error generated by state.transport.fs_rmdir(path) or state.transport.fs_rmfile(path) to propagate up so that the base exception handlers in do_filesystem still see the final OSError(errno.ENOTEMPTY, ...) to trigger an overall CommandError and nonzero exit code. And the output message code is duplicated from mpremote's main.py (interpolated over the OSError -> CommandError behavior of #17090), in order to produce output that precisely matches the output that would be produced by that final CommandError.

This doesn't implement the rm -r : behavior @dpgeorge suggested, but IMO it's just unnecessary to have a way of clearing a mountpoint without triggering an error output -- you can just run it on the mountpoint's path, and if you need to suppress the stderr message or nonzero exit you just 2>/dev/null || true.

I have an alternative implementation

indeed, it has it merits, and can stand on its own.
i see no need to include that PR in this one , I think it will only diffuse matters, and do not plan to pull it in.

This doesn't implement the rm -r : behavior @dpgeorge suggested

I'll await Damien's comments.

you can just run it on the mountpoint's path,

mpremote does not (yet) CLI to expose these mountpoints to users without writing code to a new and therefore mostly unknown API. Thus I see very clear merits.

I now find a desire for a mpremote fs tree command , let's land this one first.

Yes, I was also thinking that tree would be very useful. That can be the next PR 😄

This doesn't implement the rm -r : behavior @dpgeorge suggested, but IMO it's just unnecessary to have a way of clearing a mountpoint without triggering an error output -- you can just run it on the mountpoint's path, and if you need to suppress the stderr message or nonzero exit you just 2>/dev/null || true.

Clearing out an entire mount point / filesystem is probably one of the main uses of rm -r, and having that always error out in some way is not very user friendly. Similarly, asking users to redirect stderr is just going to add extra friction and require documentation and more support. mpremote is supposed to be a very user friendly front-end to a device.

Since mpremote doesn't support wildcards you can't do mpremote rm -r :\* to clear out the current directory, so mpremote rm -r : seems like a simple way to implement that feature. And it leaves mpremote rm -r :/ to have behaviour like unix and give an error if it can't remove /.

The failing CI test is unrelated.

It's been a worthwhile effort getting this into a shape we can all be glad to see land!