Binary over USB/serial #11889

stephanelsmith · 2023-06-27T21:21:36Z

stephanelsmith
Jun 27, 2023
Sponsor

On ESP32S3 (not sure if the port matters), I'm transfering binary over USB to my application. I'm having two issues which I think should have simple configuration issues which I can't seem to find. My goal is to update firmware, ota partition update, without going into bootloader. I already have this working over wifi, but want USB update option as well. Should be staight forward, right?

I'm sending binary data over USB and reading in the bytes via streamreader. Something like:

sreader = asyncio.StreamReader(sys.stdin)
sreader.readinto(mv)

I have kbd_intr(-1) set to prevent KeyboardInterrupt.

As a test, transferring 256 bytes (eg. bytes([x for x in range(256)])) in a go, in I receive the bytes sent with two artifacts:

Value 13 is being automatically converted to 10 (auto CR->LF conversion). I tried sending 13 on both Windows and Linux machine, results the same, so I'm fairly confident it's on embedded side. Where would one find the configuration that's manipulating the CR->LF? This is very annoying, I'd like to avoid base conversion, it shouldn't be necessary.
I've found the following ESP-IDF settings added to sdkconfig, which has no effect.

CONFIG_NEWLIB_STDIN_LINE_ENDING_CR=n
CONFIG_NEWLIB_STDIN_LINE_ENDING_CRLF=n
CONFIG_NEWLIB_STDIN_LINE_ENDING_LF=y
CONFIG_NEWLIB_STDOUT_LINE_ENDING_CR=n
CONFIG_NEWLIB_STDOUT_LINE_ENDING_CRLF=n
CONFIG_NEWLIB_STDOUT_LINE_ENDING_LF=y
CONFIG_NEWLIB_STDIN_LINE_ENDING_CR=n

I also think there's some RX buffer sensitivity. In a single trasfer, I get 251 bytes sent correctly. If I add pauses, or chunk, etc... I get all my bytes, presumably giving the RX side enough time to drain the buffer. I found the following setting which has no effect.
CONFIG_USB_CDC_RX_BUFSIZE=512
Is there an RX buffer I can expand?

I'm hoping someone has run into this as it's driving me a bit nuts!

Answered by stephanelsmith

Jun 28, 2023

First, thank you for the response. First yes, it works. Now that I've found shared/runtime/sys_stdio_mphal.c I can see it.

I'm using asyncio, this seems to do the trick:
asyncio.StreamReader(sys.stdin.buffer)

From there on out, my readinto's are working as expected.

Still feels quite quirky as I would expect readintos to straight up copy byte for byte, but this will do.

View full answer

stephanelsmith · 2023-06-28T16:39:32Z

stephanelsmith
Jun 28, 2023
Author Sponsor

For anyone else that expects to read the same bytes that were sent over stdio:

Here's the culprit for item 1. In shared/runtime/sys_stdio_mphal.c, read method. Commenting out the \r to \n conversion will fix it.
#11898

STATIC mp_uint_t stdio_read(mp_obj_t self_in, void *buf, mp_uint_t size, int *errcode) {
    sys_stdio_obj_t *self = MP_OBJ_TO_PTR(self_in);
    if (self->fd == STDIO_FD_IN) {
        for (uint i = 0; i < size; i++) {
            int c = mp_hal_stdin_rx_chr();
            if (c == '\r') {
                c = '\n';
            }
            ((byte *)buf)[i] = c;
        }
        return size;
    } else {
        *errcode = MP_EPERM;
        return MP_STREAM_ERROR;
    }
}

4 replies

robert-hh Jun 28, 2023
Collaborator

Did you try sys.stdin.buffer.read() and sys.stdout.buffer.write()?

stephanelsmith Jun 28, 2023
Author Sponsor

First, thank you for the response. First yes, it works. Now that I've found shared/runtime/sys_stdio_mphal.c I can see it.

I'm using asyncio, this seems to do the trick:
asyncio.StreamReader(sys.stdin.buffer)

From there on out, my readinto's are working as expected.

Still feels quite quirky as I would expect readintos to straight up copy byte for byte, but this will do.

Answer selected by stephanelsmith

jimmo Jun 30, 2023
Maintainer

@stephanelsmith readinto/read is just about whether it reads into a buffer or returns the bytes.

The key thing here is that sys.stdin.buffer is the underlying stream, whereas sys.stdin does line ending conversion.

stephanelsmith Jun 30, 2023
Author Sponsor

Thank you for the kind response. Mentally I was thinking stdin was the underlying stream which is why I got snared. Learn something new everyday. Again, thank you for the response!

stephanelsmith · 2023-06-28T21:03:23Z

stephanelsmith
Jun 28, 2023
Author Sponsor

Part two answer, the stdin buffer size on esp32 is set here:
ports/esp32/mphalport.c
STATIC uint8_t stdin_ringbuf_array[260];

Increasing the size works as expected. Should this be a #define option?

0 replies

Baker68 · 2025-02-16T10:01:50Z

Baker68
Feb 16, 2025

@stephanelsmith ; Are you willing to share your solution (Python/MPY scripts)?

Thanks in advance

0 replies

MicroPython

Binary over USB/serial #11889

Uh oh!

Uh oh!

stephanelsmith Jun 27, 2023 Sponsor

Replies: 3 comments · 4 replies

Uh oh!

stephanelsmith Jun 28, 2023 Author Sponsor

Uh oh!

robert-hh Jun 28, 2023 Collaborator

Uh oh!

Uh oh!

stephanelsmith Jun 28, 2023 Author Sponsor

Uh oh!

jimmo Jun 30, 2023 Maintainer

Uh oh!

stephanelsmith Jun 30, 2023 Author Sponsor

Uh oh!

stephanelsmith Jun 28, 2023 Author Sponsor

Uh oh!

Baker68 Feb 16, 2025

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

stephanelsmith
Jun 27, 2023
Sponsor

Replies: 3 comments 4 replies

stephanelsmith
Jun 28, 2023
Author Sponsor

robert-hh Jun 28, 2023
Collaborator

stephanelsmith Jun 28, 2023
Author Sponsor

jimmo Jun 30, 2023
Maintainer

stephanelsmith Jun 30, 2023
Author Sponsor

stephanelsmith
Jun 28, 2023
Author Sponsor

Baker68
Feb 16, 2025