-
Notifications
You must be signed in to change notification settings - Fork 932
Fix PIO I2C race condition #617
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Thanks for the analysis. I'll see if I can reproduce the problem with a BH1750. |
cc @Wren6991 for thoughts on the underlying issue |
Just thinking through this part on the potential race condition:
It's my belief that if you write to It's more likely that the input shift counter is nonzero at the point autopush is enabled. The only way I could see that happening is if this function were somehow called at the wrong time: void pio_i2c_resume_after_error(PIO pio, uint sm) {
pio_sm_drain_tx_fifo(pio, sm);
pio_sm_exec(pio, sm, (pio->sm[sm].execctrl & PIO_SM0_EXECCTRL_WRAP_BOTTOM_BITS) >> PIO_SM0_EXECCTRL_WRAP_BOTTOM_LSB);
pio_interrupt_clear(pio, sm);
} ...since this is the only point that we directly touch the SM's program counter. The workaround given here makes sense under that theory (I would still like to understand how the shift counter ends up nonzero!) and the only change I would make would be to move the counter clear into the diff --git a/pio/i2c/pio_i2c.c b/pio/i2c/pio_i2c.c
index a3a19f5..7028fce 100644
--- a/pio/i2c/pio_i2c.c
+++ b/pio/i2c/pio_i2c.c
@@ -67,9 +67,10 @@ uint8_t pio_i2c_get(PIO pio, uint sm) {
}
void pio_i2c_start(PIO pio, uint sm) {
- pio_i2c_put_or_err(pio, sm, 1u << PIO_I2C_ICOUNT_LSB); // Escape code for 2 instruction sequence
+ pio_i2c_put_or_err(pio, sm, 2u << PIO_I2C_ICOUNT_LSB); // Escape code for 3 instruction sequence
pio_i2c_put_or_err(pio, sm, set_scl_sda_program_instructions[I2C_SC1_SD0]); // We are already in idle state, just pull SDA low
pio_i2c_put_or_err(pio, sm, set_scl_sda_program_instructions[I2C_SC0_SD0]); // Also pull clock low so we can present data
+ pio_i2c_put_or_err(pio, sm, pio_encode_mov(pio_isr, pio_null)); // Ensure ISR counter is clear
}
void pio_i2c_stop(PIO pio, uint sm) {
@@ -80,11 +81,12 @@ void pio_i2c_stop(PIO pio, uint sm) {
};
void pio_i2c_repstart(PIO pio, uint sm) {
- pio_i2c_put_or_err(pio, sm, 3u << PIO_I2C_ICOUNT_LSB);
+ pio_i2c_put_or_err(pio, sm, 4u << PIO_I2C_ICOUNT_LSB);
pio_i2c_put_or_err(pio, sm, set_scl_sda_program_instructions[I2C_SC0_SD1]);
pio_i2c_put_or_err(pio, sm, set_scl_sda_program_instructions[I2C_SC1_SD1]);
pio_i2c_put_or_err(pio, sm, set_scl_sda_program_instructions[I2C_SC1_SD0]);
pio_i2c_put_or_err(pio, sm, set_scl_sda_program_instructions[I2C_SC0_SD0]);
+ pio_i2c_put_or_err(pio, sm, pio_encode_mov(pio_isr, pio_null));
}
static void pio_i2c_wait_idle(PIO pio, uint sm) { |
Ah, no, it's just because the ISR counter increments during the write (even though autopush is disabled) and is never cleared. The next |
This avoids a spurious shift on a write followed by a read. See discussion on #617
This avoids a spurious shift on a write followed by a read. See discussion on #617
Closing in favour of #676 which is the same thing but without using a PIO instruction memory slot |
This avoids a spurious shift on a write followed by a read. See discussion on #617
This is a (hacky) fix for a race condition bug in the PIO I2C code that breaks I2C reads.
In many cases when reading data over the I2C bus via PIO, the correct data is received over the wire (as seen on a logic analyzer) but is incorrectly reported by the PIO state machine, with the result that I2C reads don't work. This problem has already been observed in the following threads and issue:
[1] https://forums.raspberrypi.com/viewtopic.php?t=356883
[2] https://forums.raspberrypi.com/viewtopic.php?t=340111
[3] #168
Testing suggests the (incorrect) bytes reported by the PIO are the expected I2C byte stream but shifted by 7 (!) bits. For example, the thread [2] reported the following examples of incorrectly read byte sequences. The I2C bus sees an 8 bytes for the address + read bit, then 32 bytes of data:
This makes the following sequences in binary form, where we can see the PIO data is offset 7 bits from what is visible on the I2C bus:
I have replicated all this with a BH1750 light sensor and FDC2112 capacitance sensor on both a RP2040 and RP2350, and I can further confirm the mysterious first 7 bits reported by the PIO appear to be the last 7 bits of the (address + read bit) byte sent over the I2C bus prior to the expected data. The PIO is correctly reading the bus but has somehow shifted the result by 7 bits.
My best explanation for this is as follows:
First, I suspect the PIO program itself works as expected. Every time it reads or writes a byte on the bus, it puts 8 bits in the ISR (note this happens for both reads and writes, which the PIO treats identically). Autopush should be enabled, moving 8 bits at a time into the RX FIFO. To write a byte over I2C, the main core should put a byte into the state machine TX FIFO, wait for the state machine, then read and discard a byte from the RX FIFO. To read a byte, the main core should but 0xff into the TX FIFO, then read the result from the RX FIFO.
However, the C program running on the main core messes with this. In
pio_i2c.c
, the functionpio_i2c_write_blocking()
first callspio_i2c_rx_enable()
, which disables the PIO state machine autopush via direct register manipulation. This is done so that the PIO doesn't put anything on the RX FIFO, so bytes don't have to be read then discarded. The functionpio_i2c_write_blocking()
re-enables autopush.Crucially, enabling/disabling autopush is not synced to the PIO state machine clock.
I haven't been able to figure out exactly when things break (I don't know why the shift seems is reliably 7 bits instead of a random number depending on the setup), but the following is a possibility:
Even though the exact race condition hasn't been pinned down, there's enough evidence to create solutions that seem to work:
pio_i2c_rx_enable()
and havepio_i2c_write_blocking()
read and discard the resulting RX FIFO dummy bytes. I confirmed this seemed to work in at least one case. This is similar to the solution noted in [2].I chose to implement solution B by adding a
mov isr, null
instruction to the PIO assembly since this protects against all race conditions. Since this is a patch rather than fixing the ultimate root issue, it is possible there exist some setups where there is still edge case broken behavior, but I was not able to find any. Further implementing solution A might be wise in the future because it probably fixes whatever the root issue is. Solution C might be helpful in the long term as there are some other suspicious bits of code (e.g., emptying the RX FIFO buffer viawhile (!pio_sm_is_rx_fifo_empty(pio, sm)){(void)pio_i2c_get(pio, sm);}
at the start ofpio_i2c_read_blocking()
seems like it should be unnecessary).