[Messenger][Doctrine] Deadlock when deleting message #54103
Description
Symfony version(s) affected
6.4
Description
We started getting this deadlock recently. It has happened only twice so far under high load.
SQLSTATE[40P01]: Deadlock detected: 7 ERROR: deadlock detected
DETAIL: Process 221664 waits for ShareLock on transaction 59539641; blocked by process 221671.
Process 221671 waits for AccessExclusiveLock on tuple (77,27) of relation 16455 of database 16385; blocked by process 221605.
Process 221605 waits for ShareLock on transaction 59539646; blocked by process 221606.
Process 221606 waits for AccessExclusiveLock on tuple (69,16) of relation 16455 of database 16385; blocked by process 221664.
HINT: See server log for query details.
CONTEXT: while deleting tuple (69,16) in relation "messenger_messages"
Process 221664 waits for ShareLock on transaction 59539641;
blocked by process 221671.
Here are the queries for each process:
Process 221664 waits for ShareLock on transaction 59539641;
blocked by process 221671.
221671
SELECT m.* FROM messenger_messages m WHERE (m.queue_name = $1) AND (m.delivered_at is null OR m.delivered_at < $2) AND (m.available_at <= $3) ORDER BY available_at ASC LIMIT 1 FOR UPDATE
221605
SELECT m.* FROM messenger_messages m WHERE (m.queue_name = $1) AND (m.delivered_at is null OR m.delivered_at < $2) AND (m.available_at <= $3) ORDER BY available_at ASC LIMIT 1 FOR UPDATE
221606
SELECT m.* FROM messenger_messages m WHERE (m.queue_name = $1) AND (m.delivered_at is null OR m.delivered_at < $2) AND (m.available_at <= $3) ORDER BY available_at ASC LIMIT 1 FOR UPDATE
221664
DELETE FROM messenger_messages WHERE id = $1
Is something we can can improve in the Doctrine messenger implementation to detect when a deadlock exception occurs and retry the delete? Or is it possible to eliminate the potential for deadlocks entirely?
How to reproduce
It is hard to reproduce this because it only happens with high volume.
Possible Solution
Catch the deadlock exception and retry the delete. I am happy to make this PR if we think it is the right way to address the issue.
Additional Context
I started to look at the existing code, and we handle deadlocks on get() but not on reject() or ack().