micropython · dpgeorge · Dec 1, 2020 · Oct 6, 2020 · Oct 8, 2020 · Nov 30, 2020
diff --git a/docs/reference/repl.rst b/docs/reference/repl.rst
@@ -196,17 +196,105 @@ So you can use the underscore to save the result in a variable. For example:
     15
     >>>
 
-Raw mode
---------
+Raw mode and raw-paste mode
+---------------------------
 
-Raw mode is not something that a person would normally use. It is intended for
-programmatic use. It essentially behaves like paste mode with echo turned off.
+Raw mode (also called raw REPL) is not something that a person would normally use.
+It is intended for programmatic use and essentially behaves like paste mode with
+echo turned off, and with optional flow control.
 
 Raw mode is entered using Ctrl-A. You then send your python code, followed by
 a Ctrl-D. The Ctrl-D will be acknowledged by 'OK' and then the python code will
 be compiled and executed. Any output (or errors) will be sent back. Entering
 Ctrl-B will leave raw mode and return the the regular (aka friendly) REPL.
 
-The ``tools/pyboard.py`` program uses the raw REPL to execute python files on the
-MicroPython board.
+Raw-paste mode is an additional mode within the raw REPL that includes flow control,
+and which compiles code as it receives it. This makes it more robust for high-speed
+transfer of code into the device, and it also uses less RAM when receiving because
+it does not need to store a verbatim copy of the code before compiling (unlike
+standard raw mode).
 
+Raw-paste mode uses the following protocol:
+
+#. Enter raw REPL as usual via ctrl-A.
+
+#. Write 3 bytes: ``b"\x05A\x01"`` (ie ctrl-E then "A" then ctrl-A).
+
+#. Read 2 bytes to determine if the device entered raw-paste mode:
+
+   * If the result is ``b"R\x00"`` then the device understands the command but
+     doesn't support raw paste.
+
+   * If the result is ``b"R\x01"`` then the device does support raw paste and
+     has entered this mode.
+
+   * Otherwise the result should be ``b"ra"`` and the device doesn't support raw
+     paste and the string ``b"w REPL; CTRL-B to exit\r\n>"`` should be read and
+     discarded.
+
+#. If the device is in raw-paste mode then continue, otherwise fallback to
+   standard raw mode.
+
+#. Read 2 bytes, this is the flow control window-size-increment (in bytes)
+   stored as a 16-bit unsigned little endian integer.  The initial value for the
+   remaining-window-size variable should be set to this number.
+
+#. Write out the code to the device:
+
+   * While there are bytes to send, write up to the remaining-window-size worth
+     of bytes, and decrease the remaining-window-size by the number of bytes
+     written.
+
+   * If the remaining-window-size is 0, or there is a byte waiting to read, read
+     1 byte.  If this byte is ``b"\x01"`` then increase the remaining-window-size
+     by the window-size-increment from step 5.  If this byte is ``b"\x04"`` then
+     the device wants to end the data reception, and ``b"\x04"`` should be
+     written to the device and no more code sent after that.  (Note: if there is
+     a byte waiting to be read from the device then it does not need to be read
+     and acted upon immediately, the device will continue to consume incoming
+     bytes as long as reamining-window-size is greater than 0.)
+
+#. When all code has been written to the device, write ``b"\x04"`` to indicate
+   end-of-data.
+
+#. Read from the device until ``b"\x04"`` is received.  At this point the device
+   has received and compiled all of the code that was sent and is executing it.
+
+#. The device outputs any characters produced by the executing code.  When (if)
+   the code finishes ``b"\x04"`` will be output, followed by any exception that
+   was uncaught, followed again by ``b"\x04"``.  It then goes back to the
+   standard raw REPL and outputs ``b">"``.
+
+For example, starting at a new line at the normal (friendly) REPL, if you write::
+
+    b"\x01\x05A\x01print(123)\x04"
+
+Then the device will respond with something like::
+
+    b"\r\nraw REPL; CTRL-B to exit\r\n>R\x01\x80\x00\x01\x04123\r\n\x04\x04>"
+
+Broken down over time this looks like::
+
+    # Step 1: enter raw REPL
+    write: b"\x01"
+    read: b"\r\nraw REPL; CTRL-B to exit\r\n>"
+
+    # Step 2-5: enter raw-paste mode
+    write: b"\x05A\x01"
+    read: b"R\x01\x80\x00\x01"
+
+    # Step 6-8: write out code
+    write: b"print(123)\x04"
+    read: b"\x04"
+
+    # Step 9: code executes and result is read
+    read: b"123\r\n\x04\x04>"
+
+In this case the flow control window-size-increment is 128 and there are two
+windows worth of data immediately available at the start, one from the initial
+window-size-increment value and one from the explicit ``b"\x01"`` value that
+is sent.  So this means up to 256 bytes can be written to begin with before
+waiting or checking for more incoming flow-control characters.
+
+The ``tools/pyboard.py`` program uses the raw REPL, including raw-paste mode, to
+execute Python code on a MicroPython-enabled board.
diff --git a/lib/utils/pyexec.c b/lib/utils/pyexec.c
@@ -56,6 +56,7 @@ STATIC bool repl_display_debugging_info = 0;
 #define EXEC_FLAG_SOURCE_IS_RAW_CODE (8)
 #define EXEC_FLAG_SOURCE_IS_VSTR (16)
 #define EXEC_FLAG_SOURCE_IS_FILENAME (32)
+#define EXEC_FLAG_SOURCE_IS_READER (64)
 
 // parses, compiles and executes the code in the lexer
 // frees the lexer before returning
@@ -91,6 +92,8 @@ STATIC int parse_compile_execute(const void *source, mp_parse_input_kind_t input
             if (exec_flags & EXEC_FLAG_SOURCE_IS_VSTR) {
                 const vstr_t *vstr = source;
                 lex = mp_lexer_new_from_str_len(MP_QSTR__lt_stdin_gt_, vstr->buf, vstr->len, 0);
+            } else if (exec_flags & EXEC_FLAG_SOURCE_IS_READER) {
+                lex = mp_lexer_new(MP_QSTR__lt_stdin_gt_, *(mp_reader_t *)source);
             } else if (exec_flags & EXEC_FLAG_SOURCE_IS_FILENAME) {
                 lex = mp_lexer_new_from_file(source);
             } else {
@@ -122,6 +125,12 @@ STATIC int parse_compile_execute(const void *source, mp_parse_input_kind_t input
         // uncaught exception
         mp_hal_set_interrupt_char(-1); // disable interrupt
         mp_handle_pending(false); // clear any pending exceptions (and run any callbacks)
+
+        if (exec_flags & EXEC_FLAG_SOURCE_IS_READER) {
+            const mp_reader_t *reader = source;
+            reader->close(reader->data);
+        }
+
         // print EOF after normal output
         if (exec_flags & EXEC_FLAG_PRINT_EOF) {
             mp_hal_stdout_tx_strn("\x04", 1);
@@ -170,6 +179,99 @@ STATIC int parse_compile_execute(const void *source, mp_parse_input_kind_t input
 }
 
 #if MICROPY_ENABLE_COMPILER
+
+// This can be configured by a port (and even configured to a function to be
+// computed dynamically) to indicate the maximum number of bytes that can be
+// held in the stdin buffer.
+#ifndef MICROPY_REPL_STDIN_BUFFER_MAX
+#define MICROPY_REPL_STDIN_BUFFER_MAX (256)
+#endif
+
+typedef struct _mp_reader_stdin_t {
+    bool eof;
+    uint16_t window_max;
+    uint16_t window_remain;
+} mp_reader_stdin_t;
+
+STATIC mp_uint_t mp_reader_stdin_readbyte(void *data) {
+    mp_reader_stdin_t *reader = (mp_reader_stdin_t *)data;
+
+    if (reader->eof) {
+        return MP_READER_EOF;
+    }
+
+    int c = mp_hal_stdin_rx_chr();
+
+    if (c == CHAR_CTRL_C || c == CHAR_CTRL_D) {
+        reader->eof = true;
+        mp_hal_stdout_tx_strn("\x04", 1); // indicate end to host
+        if (c == CHAR_CTRL_C) {
+            #if MICROPY_KBD_EXCEPTION
+            MP_STATE_VM(mp_kbd_exception).traceback_data = NULL;
+            nlr_raise(MP_OBJ_FROM_PTR(&MP_STATE_VM(mp_kbd_exception)));
+            #else
+            mp_raise_type(&mp_type_KeyboardInterrupt);
+            #endif
+        } else {
+            return MP_READER_EOF;
+        }
+    }
+
+    if (--reader->window_remain == 0) {
+        mp_hal_stdout_tx_strn("\x01", 1); // indicate window available to host
+        reader->window_remain = reader->window_max;
+    }
+
+    return c;
+}
+
+STATIC void mp_reader_stdin_close(void *data) {
+    mp_reader_stdin_t *reader = (mp_reader_stdin_t *)data;
+    if (!reader->eof) {
+        reader->eof = true;
+        mp_hal_stdout_tx_strn("\x04", 1); // indicate end to host
+        for (;;) {
+            int c = mp_hal_stdin_rx_chr();
+            if (c == CHAR_CTRL_C || c == CHAR_CTRL_D) {
+                break;
+            }
+        }
+    }
+}
+
+STATIC void mp_reader_new_stdin(mp_reader_t *reader, mp_reader_stdin_t *reader_stdin, uint16_t buf_max) {
+    // Make flow-control window half the buffer size, and indicate to the host that 2x windows are
+    // free (sending the window size implicitly indicates that a window is free, and then the 0x01
+    // indicates that another window is free).
+    size_t window = buf_max / 2;
+    char reply[3] = { window & 0xff, window >> 8, 0x01 };
+    mp_hal_stdout_tx_strn(reply, sizeof(reply));
+
+    reader_stdin->eof = false;
+    reader_stdin->window_max = window;
+    reader_stdin->window_remain = window;
+    reader->data = reader_stdin;
+    reader->readbyte = mp_reader_stdin_readbyte;
+    reader->close = mp_reader_stdin_close;
+}
+
+STATIC int do_reader_stdin(int c) {
+    if (c != 'A') {
+        // Unsupported command.
+        mp_hal_stdout_tx_strn("R\x00", 2);
+        return 0;
+    }
+
+    // Indicate reception of command.
+    mp_hal_stdout_tx_strn("R\x01", 2);
+
+    mp_reader_t reader;
+    mp_reader_stdin_t reader_stdin;
+    mp_reader_new_stdin(&reader, &reader_stdin, MICROPY_REPL_STDIN_BUFFER_MAX);
+    int exec_flags = EXEC_FLAG_PRINT_EOF | EXEC_FLAG_SOURCE_IS_READER;
+    return parse_compile_execute(&reader, MP_PARSE_FILE_INPUT, exec_flags);
+}
+
 #if MICROPY_REPL_EVENT_DRIVEN
 
 typedef struct _repl_t {
@@ -203,6 +305,13 @@ void pyexec_event_repl_init(void) {
 STATIC int pyexec_raw_repl_process_char(int c) {
     if (c == CHAR_CTRL_A) {
         // reset raw REPL
+        if (vstr_len(MP_STATE_VM(repl_line)) == 2 && vstr_str(MP_STATE_VM(repl_line))[0] == CHAR_CTRL_E) {
+            int ret = do_reader_stdin(vstr_str(MP_STATE_VM(repl_line))[1]);
+            if (ret & PYEXEC_FORCED_EXIT) {
+                return ret;
+            }
+            goto reset;
+        }
         mp_hal_stdout_tx_str("raw REPL; CTRL-B to exit\r\n");
         goto reset;
     } else if (c == CHAR_CTRL_B) {
@@ -388,6 +497,15 @@ int pyexec_raw_repl(void) {
             int c = mp_hal_stdin_rx_chr();
             if (c == CHAR_CTRL_A) {
                 // reset raw REPL
+                if (vstr_len(&line) == 2 && vstr_str(&line)[0] == CHAR_CTRL_E) {
+                    int ret = do_reader_stdin(vstr_str(&line)[1]);
+                    if (ret & PYEXEC_FORCED_EXIT) {
+                        return ret;
+                    }
+                    vstr_reset(&line);
+                    mp_hal_stdout_tx_str(">");
+                    continue;
+                }
                 goto raw_repl_reset;
             } else if (c == CHAR_CTRL_B) {
                 // change to friendly REPL

diff --git a/tools/pyboard.py b/tools/pyboard.py
@@ -253,6 +253,7 @@ def inWaiting(self):
 
 class Pyboard:
     def __init__(self, device, baudrate=115200, user="micro", password="python", wait=0):
+        self.use_raw_paste = True
         if device.startswith("exec:"):
             self.serial = ProcessToSerial(device[len("exec:") :])
         elif device.startswith("execpty:"):
@@ -359,6 +360,41 @@ def follow(self, timeout, data_consumer=None):
         # return normal and error output
         return data, data_err
 
+    def raw_paste_write(self, command_bytes):
+        # Read initial header, with window size.
+        data = self.serial.read(2)
+        window_size = data[0] | data[1] << 8
+        window_remain = window_size
+
+        # Write out the command_bytes data.
+        i = 0
+        while i < len(command_bytes):
+            while window_remain == 0 or self.serial.inWaiting():
+                data = self.serial.read(1)
+                if data == b"\x01":
+                    # Device indicated that a new window of data can be sent.
+                    window_remain += window_size
+                elif data == b"\x04":
+                    # Device indicated abrupt end.  Acknowledge it and finish.
+                    self.serial.write(b"\x04")
+                    return
+                else:
+                    # Unexpected data from device.
+                    raise PyboardError("unexpected read during raw paste: {}".format(data))
+            # Send out as much data as possible that fits within the allowed window.
+            b = command_bytes[i : min(i + window_remain, len(command_bytes))]
+            self.serial.write(b)
+            window_remain -= len(b)
+            i += len(b)
+
+        # Indicate end of data.
+        self.serial.write(b"\x04")
+
+        # Wait for device to acknowledge end of data.
+        data = self.read_until(1, b"\x04")
+        if not data.endswith(b"\x04"):
+            raise PyboardError("could not complete raw paste: {}".format(data))
+
     def exec_raw_no_follow(self, command):
         if isinstance(command, bytes):
             command_bytes = command
@@ -370,7 +406,26 @@ def exec_raw_no_follow(self, command):
         if not data.endswith(b">"):
             raise PyboardError("could not enter raw repl")
 
-        # write command
+        if self.use_raw_paste:
+            # Try to enter raw-paste mode.
+            self.serial.write(b"\x05A\x01")
+            data = self.serial.read(2)
+            if data == b"R\x00":
+                # Device understood raw-paste command but doesn't support it.
+                pass
+            elif data == b"R\x01":
+                # Device supports raw-paste mode, write out the command using this mode.
+                return self.raw_paste_write(command_bytes)
+            else:
+                # Device doesn't support raw-paste, fall back to normal raw REPL.
+                data = self.read_until(1, b"w REPL; CTRL-B to exit\r\n>")
+                if not data.endswith(b"w REPL; CTRL-B to exit\r\n>"):
+                    print(data)
+                    raise PyboardError("could not enter raw repl")
+            # Don't try to use raw-paste mode again for this connection.
+            self.use_raw_paste = False
+
+        # Write command using standard raw REPL, 256 bytes every 10ms.
         for i in range(0, len(command_bytes), 256):
             self.serial.write(command_bytes[i : min(i + 256, len(command_bytes))])
             time.sleep(0.01)