Description
Symfony version(s) affected:
From 4.4 to master
Description
When message cannot be processed and cannot be written to failed transport, it gets lost.
How to reproduce
- Have a handler which processes a message and writes something to database. Database is down for some reason. Retries are disabled (or DB is down all the time during retry duration).
- Handler receives a message and throws an exception (because DB is down).
- Messenger component tries to send message to failed, but fails as well (because DB is down).
- In this case message remains in RabbitMQ (as expected).
- After a while consumer starts processing the same message. It fails again (because DB is still down).
- Now message is rejected from RabbitMQ and lost forever, this happens because of
RejectRedeliveredMessageMiddlewarewhich prevents message being stuck in queue.
In my opinion this is dangerous behavior which makes implementation of proper resilient applications impossible with Messenger component. The component must guarantee that with proper configuration messages will never be lost implicitly.
So expected behavior from my side:
- Message should remain in queue until it is either processed or put into failed transport.
- It is totally acceptable to have messages stuck in queue, because it must be caught by application monitoring.
Possible Solution
Remove this class and it's definition from DI: https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Messenger/Middleware/RejectRedeliveredMessageMiddleware.php
Also remove this code: https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Messenger/Worker.php#L116-L120
To prevent messages being stuck in queue other solutions should be applied:
- Proper failed transport configuration.
- If failed transport is not healthy, application monitoring should be useful.
Additional context
I can implement a change myself, but I would like to raise it first and discuss. Someone implemented RejectRedeliveredMessageMiddleware, so there could be other use-case for it.