Looks good for starters. Would need to refactor on top of #3069 of course.

I fixed the mkdir method, and used a value of 0777 for the mode (this is anyway how current unix uos module has it, so it's at least not a regression).

I rebased against the ilistdir() PR #3069 and changed VfsPosix.listdir() to VfsPosix.ilistdir().

There's still some testing that needs to be done. And micropython-lib/os needs to be reworked because it currently bypasses the vfs stuff, going straight to syscalls.

And micropython-lib/os needs to be reworked because it currently bypasses the vfs stuff, going straight to syscalls.

Sure. What timeframe do we have to merge this? This is pretty big change in unix port to target for 1.9 I guess. On the other hand, I'm trying, trying to synchronize change in upy-lib with releases, for example, I'm waiting for ETA on 1.9 to push ipoll change ;-). From that PoV, trying to push everything into 1.9 makes it "easier".

What timeframe do we have to merge this? This is pretty big change in unix port to target for 1.9 I guess.

I didn't intend for it to go in 1.9, but that's an option to consider. It requires a fair bit of testing, but I don't see any technical blockers.

I cleaned up the commits for this and it's getting closer to being mergeable. One main thing to decide is whether to keep the old behaviour (ie no VFS) as a build option, or to just support the VFS build.

I don't know if this will be ready for 1.9 because there are lots of subtleties to consider. Eg, with a standard FS you can do something like:

>>> os.chdir("some/dir")
>>> os.system("./my_executable")

and the current directory set by chdir is seen by the system command. This is no longer true with VFS enabled (or at least it's not guaranteed).

whether to keep the old behaviour (ie no VFS) as a build option, or to just support the VFS build.

I'd say it's too much variability to maintain. If we have good VFS architecture, it should be the only choice. But would be nice to know how much size we lose (i.e. grow) with it, test enough for such a big change, etc. So, I think it shouldn't be hasted into 1.9, current time better invest into esp8266 testing ;-).

and the current directory set by chdir is seen by the system command. This is no longer true with VFS enabled (or at least it's not guaranteed).

Hmm, I thought we call thru to the underlying FS chdir(), etc. Yeah, the code above definitely should work to switch to VFS impl.

I think it would be good to merge this now (with some minor clean up) and then find any bugs during usage over the coming weeks, with enough time before the next release to fix bugs.

One main thing to decide about: I wrote the VFS POSIX class so that it could mount any particular subdir of the host machine, not just root. Eg:

vfs = VfsPosix() # has access to the whole host filesystem, starting at root
vfs = VfsPosix('/') # same as above
vfs = VfsPosix('/home/username') # has access only to my home directory

Is this a feature to keep or discard? It adds minor complexity to the code.

I think it would be good to merge this now (with some minor clean up) and then find any bugs during usage over the coming weeks, with enough time before the next release to fix bugs.

Sounds good.

I wrote the VFS POSIX class so that it could mount any particular subdir of the host machine, not just root.

Given that it's already written, I'd say, keep it, unless there're noticeable savings from removing it.

Trying to merge this, another issue I come up against is that the unix built-in open function allows (as CPython does) to pass an integer as the first argument which is interpreted as the fileno to open, instead of the filename. Is this a necessary feature to have? To make it work would require a config option and a hook in mp_vfs_open.

To make [fileno] work would require a config option and a hook in mp_vfs_open.

Actually, support for fileno instead of filename as the first arg to open() is easy to support: it can be put in the VfsPosix code, the method that deals with opening files. Since the implementation is based around the POSIX file functions, they have the concept of fileno and so this feature should work on all POSIX systems.

Yes, I'd say definitely such feature is needed. The "hook" apparently should be a config define, the function name to call.

Ok, this is now blocked on getting it to work with the windows port. I don't really see how it will work because windows has the concetp of a "drive" and that doesn't work with the VFS.

I would have thought following the cygwin model would work. MicroPython has a pure tree and a mount point may correspond to a tree within a drive.

So you might mount /data in upy world to C:/SomeDir/SomeOtherDir. If the path passed in is missing a drive letter, then the current drive would be assumed.

Will .mount("/d", "d:/") work? If yes, it works as intended. If not, I'm still +1 to merge it as is (and deal with Windows issues later). A compromise may be to leave Windows port as is.

For reference, how Win10 "WSL" deals with it: https://blogs.msdn.microsoft.com/wsl/2016/06/15/wsl-file-system-support/ , which gives us even more indulgence to not care about drive letters.

Will .mount("/d", "d:/") work?

I don't know. I need to try it all out on a windows machine, which likely will take some time.

A compromise may be to leave Windows port as is.

That would require copying across the current unix filesystem support, and uos module, to windows (since it's all going to be replaced) and having it all as independent code under windows. Not really a good solution.

For reference, how Win10 "WSL" deals with it: https://blogs.msdn.microsoft.com/wsl/2016/06/15/wsl-file-system-support/ , which gives us even more indulgence to not care about drive letters.

But WSL is more linux than windows, and the point of the windows port is to run under windows. The unix port should run fine under WSL.

I don't know. I need to try it all out on a windows machine, which likely will take some time.

Algorithmically, is there a reason why it wouldn't work? I wouldn't think so, but I'm too overloaded/out of context, that's why I'm asking. Btw, testing can also be done on Wine.

That would require copying across the current unix filesystem support, and uos module, to windows (since it's all going to be replaced) and having it all as independent code under windows. Not really a good solution.

Well, that means that unix port should retain configurability for that part, inherited by windows port. Definitely an additional chore given that we were morally ready for complete switchover, but maybe not that problematic.

But WSL is more linux than windows, and the point of the windows port is to run under windows. The unix port should run fine under WSL.

WSL is integral part of Win10. So, if Microsoft thinks it's ok if parts of Windows access drives as /mnd/c, /mnt/d, that it should be good for other software which has POSIX FS design.

I only now figured VFS also affects the Windows port; I still have no clear idea of what it is exactly nor why it is used nor what should be tested to verify operation on windows. Clarification on that last point should allow me to get it running, if possible.
For now I just tried to build it and there are relatively few errors. After resolving these (some in a hackish way, but again i have no idea of how it's supposed to behave) the resulting uPy exe runs and can open files, uos.ilistdir seems to work properly. The whole mounting code in main.c seems a no-op though, but that is to be expected I guess since vfs_posix_mount doesn't do anything?
Patch (statvfs excluded completely, don't know what the counterpart is):

diff --git a/extmod/vfs.c b/extmod/vfs.c
index 4deba8f..f4b3246 100644
--- a/extmod/vfs.c
+++ b/extmod/vfs.c
@@ -109,7 +109,7 @@ STATIC mp_obj_t mp_vfs_proxy_call(mp_vfs_mount_t *vfs, qstr meth_name, size_t n_
         // can't do operation on root dir
         mp_raise_OSError(MP_EPERM);
     }
-    mp_obj_t meth[n_args + 2];
+    mp_obj_t *meth = alloca((n_args + 2) * sizeof(*args ));
     mp_load_method(vfs->obj, meth_name, meth);
     if (args != NULL) {
         memcpy(meth + 2, args, n_args * sizeof(*args));
diff --git a/extmod/vfs_posix.c b/extmod/vfs_posix.c
index 6442020..8bb2cb6 100644
--- a/extmod/vfs_posix.c
+++ b/extmod/vfs_posix.c
@@ -36,6 +36,14 @@
 #include "py/mperrno.h"
 #include "extmod/vfs.h"
 #include "extmod/vfs_posix.h"
+#ifdef _MSC_VER
+#define __STDC__ 1
+#include <direct.h>
+#define chdir _chdir
+#define rmdir _rmdir
+#define getcwd _getcwd
+#define mkdir(a, b) mkdir(a)
+#endif
 
 typedef struct _mp_vfs_posix_obj_t {
     mp_obj_base_t base;
@@ -293,6 +301,7 @@ STATIC MP_DEFINE_CONST_FUN_OBJ_2(vfs_posix_stat_obj, vfs_posix_stat);
 #define USE_STATFS 1
 #endif
 
+#ifndef _MSC_VER
 #if USE_STATFS
 #include <sys/vfs.h>
 #define STRUCT_STATVFS struct statfs
@@ -331,6 +340,7 @@ STATIC mp_obj_t vfs_posix_statvfs(mp_obj_t self_in, mp_obj_t path_in) {
     return MP_OBJ_FROM_PTR(t);
 }
 STATIC MP_DEFINE_CONST_FUN_OBJ_2(vfs_posix_statvfs_obj, vfs_posix_statvfs);
+#endif
 
 STATIC const mp_rom_map_elem_t vfs_posix_locals_dict_table[] = {
     { MP_ROM_QSTR(MP_QSTR_mount), MP_ROM_PTR(&vfs_posix_mount_obj) },
@@ -345,7 +355,9 @@ STATIC const mp_rom_map_elem_t vfs_posix_locals_dict_table[] = {
     { MP_ROM_QSTR(MP_QSTR_rename), MP_ROM_PTR(&vfs_posix_rename_obj) },
     { MP_ROM_QSTR(MP_QSTR_rmdir), MP_ROM_PTR(&vfs_posix_rmdir_obj) },
     { MP_ROM_QSTR(MP_QSTR_stat), MP_ROM_PTR(&vfs_posix_stat_obj) },
+#ifndef _MSC_VER
     { MP_ROM_QSTR(MP_QSTR_statvfs), MP_ROM_PTR(&vfs_posix_statvfs_obj) },
+#endif
 };
 STATIC MP_DEFINE_CONST_DICT(vfs_posix_locals_dict, vfs_posix_locals_dict_table);
 
diff --git a/windows/msvc/sources.props b/windows/msvc/sources.props
index 857f996..7f2bc22 100644
--- a/windows/msvc/sources.props
+++ b/windows/msvc/sources.props
@@ -15,6 +15,7 @@
     <ClCompile Include="$(PyBaseDir)extmod\vfs.c"/>
     <ClCompile Include="$(PyBaseDir)extmod\vfs_posix.c"/>
     <ClCompile Include="$(PyBaseDir)extmod\vfs_posix_file.c"/>
+    <ClCompile Include="$(PyBaseDir)extmod\vfs_reader.c"/>
     <ClCompile Include="$(PyBaseDir)extmod\machine_mem.c" />
     <ClCompile Include="$(PyBaseDir)extmod\machine_pinbase.c" />
     <ClCompile Include="$(PyBaseDir)extmod\machine_pulse.c" />
```

@stinos: Thanks for looking into that. Long story short, one of big problems we face with uPy development/maintenance is separate code subsets used by different ports and complexity of testing and maintaining that. With VFS code in particular, with all the effort went into it and generality requirements put into it, we'd like to use VFS by default with Unix port, just the same as with bare-metals. So, "uos" module will work with work only with VFS subsystem, and access to POSIX FS is implemented as a VFS module, just like any other FS like FatFs.

Ok, this was reworked to compile with latest master and the addition of a VfsPosix class/component merged in 8d82b0e. Follow up work was done up to commit a8b9e71 to switch unix coverage build fully to VFS (in particular see 1d40f12).

The standard unix build, along with windows, are unchanged and still use the traditional "uos" module. They can be converted to use VFS eventually.