DatabaseQueue production-ready #1364Description
This is my take on the issues with the DatabaseQueue adapter in Laravel.
Problems
Deadlocks
People are running the database queue adapter in production, even if Taylor has said it's not intended for production use.
The issues arise when running multiple queue workers. The DatabaseQueue adapter queries the jobs table to find an appropriate job to do, and if one is found, and update is performed to ensure no other workers try to take it. Because of this 2-step process, and to avoid race conditions, queries within the adapter use lockForUpdate(), which in some situations leads to deadlocks. I have witnessed that the more workers you have, the increasing likelihood of deadlocking.
Data loss potential
One other current issue, as I see it, is the possibility of losing a job when attempting to release back onto the queue because:
- The release is done in 2 steps – the original record is deleted, and a new record is inserted
- The above is not done in a single transaction – a lost DB connection after the first could lose data
Solution
Deadlocks
My solution would be to flip around the SELECT and UPDATE operations in a way that would avoid the need for transactions. Popping a job off the queue would look like this:
- Generate a random string (
str_random()) used as a temporary identifier for the current process of taking a job - Use an
UPDATEsimilar to theSELECTquery inDatabaseQueue::getNextAvailableJob, whereby the temporary identifier is stored against an available job, andLIMIT 1used to only affect 1 record - A
SELECTquery to find the job using the temporary identifier could only return 1 job, but there would be no need for a transaction and a lock.
Because of this process, there would be no need to use lockForUpdate() when deleting / releasing the job either.
A change would be required to the jobs / failed_job table schemas to store the temp identifier.
Data loss potential
This would be remedied by wrapping the 2 operations in a transaction.
Thoughts?