I encountered a deadlock situation today where multiple workflows became stuck in an infinite loop. The symptoms were:

• Workflow jobs were being dispatched for short durations then released back to the queue repeatedly
• Database records showed status = "running" even when no job was actively processing
• WithoutOverlappingMiddleware was not blocking the jobs
• The workflows could not progress or recover

Environment Context

• Platform: Kubernetes cluster with frequent container scaling
• Queue System: Laravel Horizon with auto-scaling workers
• Database: Multi-master MySQL cluster with read-only replicas (eventual consistency)
• Caching: Redis for atomic locks and real-time data

While Horizon containers are frequently replaced due to auto-scaling, they normally shut down cleanly. However, Kubernetes can occasionally evict containers, forcing unclean shutdowns that we need to account for in system design.

Root Cause Analysis (Theory)

I suspect the issue occurred when:

1. A workflow was actively running when its Horizon container was unexpectedly evicted by Kubernetes
2. The unclean shutdown prevented proper cleanup, leaving the workflow's database status as "running"
3. When the job was later picked up by another worker, it attempted to transition from "running" to "running"
4. This invalid state transition caused the job to be released back to the queue
5. The cycle repeated indefinitely since the status never changed

Questions for Discussion

1. Is the database status check redundant?

Since WithoutOverlappingMiddleware already prevents concurrent execution of the same workflow, is it necessary to also validate the database status field before allowing a workflow to run?

2. Should we rely on database state for runtime logic?

Given that:

• Kubernetes can occasionally evict containers, causing unclean shutdowns
• Database replicas may have stale data (eventual consistency)
• Unclean shutdowns can leave stale status values
• Redis is already used for atomic operations

Would it be better to treat the status field as informational only (for human readability) rather than using it for execution control?

3. Proposed solution

Remove the database status validation from the execution path and rely solely on WithoutOverlappingMiddleware for concurrency control. This would:

• Eliminate the deadlock scenario
• Work better with eventually consistent databases
• Simplify the state management logic

Request for Feedback

• Does my analysis of the root cause seem accurate?
• Are there important use cases I'm missing where the database status check is essential?
• Would the project consider removing this dependency on the status field for execution control?

Any insights or alternative approaches would be greatly appreciated!