Revert "[Messenger] Fix exception message of failed message is dropped #34082
Conversation
…d on retry" This reverts commit 8f9f44e.
|
Sorry for being late on #33600 stuff! Summary: A) We should merge this and revert #33600 (into 4.3) |
…e is dropped (Tobion) This PR was merged into the 4.3 branch. Discussion ---------- Revert "[Messenger] Fix exception message of failed message is dropped | Q | A | ------------- | --- | Branch? | 4.3 | Bug fix? | yes | New feature? | no <!-- please update src/**/CHANGELOG.md files --> | Deprecations? | no <!-- please update UPGRADE-*.md and src/**/CHANGELOG.md files --> | Tickets | | License | MIT | Doc PR | This reverts #33600 because it makes the message grow for each retry until AMQP cannot handle it anymore. On each retry, the full exception trace is added to the message. So in our case on the 5th retry, the message is too big for the AMQP library to encode it. AMQP extension then throws the exception > Library error: table too large for buffer (ref. alanxz/rabbitmq-c#224 and php-amqp/php-amqp#131) when trying to publish the message. To solve this, I suggest to revert #33600 (this PR) and merge #32341 instead which does not re-add the exception on each failure. Btw, the above problem causes other problematic side-effects of Symfony messenger. As the new retry message fails to be published with an exception, the old (currently processed message) also does not get removed (acknowledged) from the delay queue. So rabbitmq redelivers the message and the same thing happens forever. This can block the consumers and have a huge toll on your service. That's just another case for #32055 (comment). I'll try to fix this in another PR. Commits ------- 3dbe924 Revert "[Messenger] Fix exception message of failed message is dropped on retry"
|
Still got same problem here. I could reproduce it by simply adding following in the handler Then I got |
|
@Tobion I can also confirm the bug still exists, can you re-check if it was really fixed please? Also is there a test which validates the bugfix? symfony/messenger v5.3.10 |
|
Okay, for those who still gets the above error and uses RabbitMQ broker. In short, the key is message's headers: a type header and stamps headers. If you don't provide or corrupt such headers, you'll get the growing message problem. This helped me, hopefully this will help you.
|
|
In my case I got this error after an exception with way too long message. It was ServiceNotFoundException which listed all other services in a ServiceLocator instance which was several hundreds class names. This got into ErrorDetailsStamp which ended up way too large. However I saw the same error once even after fixing that. I think it can grow too large simply from the exception traces in ErrorDetailsStamp. |
This reverts #33600 because it makes the message grow for each retry until AMQP cannot handle it anymore. On each retry, the full exception trace is added to the message. So in our case on the 5th retry, the message is too big for the AMQP library to encode it. AMQP extension then throws the exception
(ref. alanxz/rabbitmq-c#224 and php-amqp/php-amqp#131) when trying to publish the message.
To solve this, I suggest to revert #33600 (this PR) and merge #32341 instead which does not re-add the exception on each failure.
Btw, the above problem causes other problematic side-effects of Symfony messenger. As the new retry message fails to be published with an exception, the old (currently processed message) also does not get removed (acknowledged) from the delay queue. So rabbitmq redelivers the message and the same thing happens forever. This can block the consumers and have a huge toll on your service. That's just another case for #32055 (comment). I'll try to fix this in another PR. Edit: Here it is #34107