[Messenger] ErrorDetailsStamp +RedeliveryStamp can lead to an oversize message for SQS #44943
Comments
|
👍 on this one. I had my whole worker setup crash in a loop due to this. This is super critical, I had to:
The failure is completely out of my code, all inside the framework, so I had no way to understand easily which message it was, even with Sentry: Luckily I had verbose logs on, so I was able to read from the crashing worker which exception was tripping everything. But this has to be solved in some way, because the messenger is making itself trip due to too much stamp data. |
|
Thank you for the ping. I dont think this is a AsyncAWS issue. All transports have issues/limitations. What do you think about listening for a |
|
That's probably the main core of the issue: it's hard to pin down a possible approach or workaround, because the issue rises at the end of the process, after the serialization already happened, so it's hard to find a way to "slim down" the envelope. |
|
I have same problem while using SQS. I cheched stack trace and main problem for oversized messages is, that arguments of functions are serialized too.
I can think of one solution: It is only a theory, I must try it. With this solution you lose function arguments in stack trace, but better than oversized message. |
|
Our team just had a similar issue. I think that the right solution would be to remove method arguments from the stack trace from |
|
With rabbitmq and the amqp PHP library, the header limit is even just 128 KB (overall, not per header). I adjusted our application to remove the error details stamp alltogether (which would not be a good general solution of course). How about a configuration for the allowed header size (that maybe could be default to the correct value if known?) and in addition to reduce verbosity of the stack trace also shorten the trace to strictly prevent errors even on very deep traces with long namespaces? its tricky that the limit is overall. maybe serialize everything, then check the length and if it is too long shorten the error details by the difference + 100 bytes of safety margin? btw: i am surprised that the amqp library allows to send messages that break the system, they could check the header limit on sending as well and not only on receiving... |
That's not what's happening: envelope gets bigger before getting sent back again, and goes over the limit. |
are you saying the message we send goes very close to the limit and then rabbitmq adds something of its own to the header that pushes it over the limit? that would either mean rabbit adds quite some volume of data, or it must be extremly unlucky to exactly cross the limit if only some dozens of bytes get added. though its possible, we process several 100k messages a day in our system so even statistically unlikely things are bound to happen sometimes... how much below the absolute limit should the header then be, to avoid problems? |
|
What I'm saying is that I'm seeing the issue only when sending the message (and Rabbit/SQS rejects it), there's no overhead added. But even when rejecting/retrying a message, in reality the trasport is in fact sending the message again, adding a new stamp (normally the |
|
It looks like we're experiencing same issue in our project. More debugging needs to be done, but everything looks like it's the same. |
|
Just in case anyone needs a workaround: https://github.com/mtanasiewicz/symfony-messenger-sqs-workaround |
|
An option at the transport level to entirely disable the framework:
messenger:
transports:
async:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
attach_error_header_on_failure: true|false
error_header_size: 10 // attach the last 10 items in the stack trace array |
|
Another workaround to completely disable the default behavior without additional classes is to remove the <?php
namespace App;
use Symfony\Bundle\FrameworkBundle\Kernel\MicroKernelTrait;
use Symfony\Component\DependencyInjection\Compiler\CompilerPassInterface;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\HttpKernel\Kernel as BaseKernel;
class Kernel extends BaseKernel implements CompilerPassInterface
{
use MicroKernelTrait;
public function process(ContainerBuilder $container): void
{
$container->removeDefinition('messenger.failure.add_error_details_stamp_listener');
}
} |
|
As an alternative to completely disabling the stamp, perhaps some stamp configuration is a possibility? Either something like For now, the workaround of implementing a custom listener and stamp class seems to work. However, I'd say that this is less than ideal considering that this feels like it should be configurable. Alternatively, perhaps a step in the right direction would be a parameter for |
|
Hey, thanks for your report! |
|
Could I get a reply or should I close this? |
|
Keep this open please |
|
Hey, thanks for your report! |
|
it is stil an issue |
|
Hey, thanks for your report! |
|
this has not been adressed afaik |
|
The real issue is that the error isn't handled gracefully and prevents the message from hitting the |
Symfony version(s) affected
5.4
Description
SQS has a limit of 262,144 bytes (256 KB) message size.
When there is an error during message handling, the
ErrorDetailsStampadds a stacktrace that can lead to a message bigger than this limit.How to reproduce
Possible Solution
Not sure, maybe an option to remove stack trace from
ErrorDetailsStampAdditional Context
No response
The text was updated successfully, but these errors were encountered: