[Messenger] deserialize third-party messages into a generic class for consumption #31230

dvc · 2019-04-24T11:23:43Z

Hello everybody!

Description
I want to receive and handle messages from 3rd party publishers without required headers.
May be some "global" fallback, and/or default type for every transport.

Can i?

Example

Simple message with payload and without ['headers']['type'] throws error:
'Encoded envelope does not have a "type" header.'

The text was updated successfully, but these errors were encountered:

weaverryan · 2019-04-24T14:18:24Z

Hey @dvc!

What you need for this is a custom transport serializer. Without the type header, we just can't know what type of object you want to deserializer into.

But, two more things:

In Symfony 4.3, you will be able to specify the serializer on a transport-by-transport basis. In 4.2 and earlier, there is only one "global" serializer for all transports.
Indeed, we could add some sort of feature to make this easier, so that you don't need to create a custom serializer for every situation. I'm not sure exactly how that should look.

dvc · 2019-04-25T06:30:46Z

Hello @weaverryan !

Thanks for comment. I don't think about custom serializer. I can configure custom transport (for transport level customization ) to use custom Receiver logic (https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Messenger/Transport/AmqpExt/AmqpReceiver.php#L66).

But! It is interesting idea to support default deserialization type by Serializer configuration (concrete class, some generic object with array payload inside or just stdClass/array). Imho, it could be useful for some use cases.

And after that allow to set/override this default in some messenger config. For example:

# config/packages/messenger.yaml
framework:
    messenger:
        default_type: 'array'
        transports:
            amqp: 
                 dsn: "%env(MESSENGER_TRANSPORT_DSN)%"
                 default_type: 'App\Entity\Message'

weaverryan · 2019-04-26T15:08:05Z

We can see if other people are also interested in something like this. It can already be accomplished currently (as we discussed), so it's a question of: is this common enough that we should make it easier to accomplish :).

Cheers!

dvc · 2019-04-26T16:52:41Z

Agree. Request for comments...

sroze · 2019-04-28T14:20:39Z

Indeed, creating your own serializer (decorating the default one for simplicity) is the best thing to do IMHO. Did you manage to do this? If so, can you share your piece of code so that others can get inspired?

dvc · 2019-04-29T08:14:15Z

Hello @sroze!
May be serializer is best extension point. I have no work code yet.

In 4.2 default serializer has public static constructor: https://github.com/symfony/symfony/blob/4.2/src/Symfony/Component/Messenger/Transport/AmqpExt/AmqpReceiver.php#L35

So, i have to extend Symfony\Component\Messenger\Transport\Serialization\Serializer and override decode-method with some logic to set default type. Something like this:

<?php
declare(strict_types=1);

namespace App\Serializer;

use Symfony\Component\Messenger\Envelope;
use Symfony\Component\Messenger\Transport\Serialization\Serializer;

final class SerializerWithDefaultDecodeType extends Serializer
{
    /**
     * @var string
     */
    private $defaultDecodeType;

    public function setDefaultDecodeType(string $defaultDecodeType): void
    {
        $this->defaultDecodeType = $defaultDecodeType;
    }

    public function decode(array $encodedEnvelope): Envelope
    {
        if ($this->defaultDecodeType && empty($encodedEnvelope['headers']['type'])) {
           // !!! Editing input parameters looks dirty. More clean for me to customize Receiver
            $encodedEnvelope['headers']['type'] = $this->defaultDecodeType;
        }
        parent::decode($encodedEnvelope);
    }
}

framework:
    messenger:
        transports:
            amqp.in: '%env(MESSENGER_TRANSPORT_DSN)%/rsa-request'
            amqp.out: '%env(MESSENGER_TRANSPORT_DSN)%/rsa-response'

        routing:
            'App\Rsa\Dkbm\Message\SayHello': amqp.in
            'App\Rsa\Dkbm\Message\SayHelloResponse': amqp.out

services:
    messenger.transport.amqp.in.symfony_serializer:
        calls:
            - method: setDefaultDecodeType
                arguments:
                    - 'App\Rsa\Dkbm\Message\SayHello'

    messenger.transport.amqp.factory:
        arguments:
            $serializer: '@messenger.transport.amqp.in.symfony_serializer'

But this configuration affects also amqp.out transport and all amqp. Because of using Transport factory. I have no information in transport factory to customize concrete transport with defaults for receive type.
Looks like serializer is not the best point in souch case.

dvc · 2019-04-29T08:53:54Z

In my use case default type is about custom transport/exchange/queue. I believe in remote configuration for this message source. Only one type in this channel.

For most common defaults:
it looks like my code is proxy from one subsystem to another, without needs to know about message structure. Just receive and send to other (logger, UI,...). For this case simple type is good choice.
May be sometimes better to not (de)serialize payload and send string as is (for performance reason or if payload has crypto signature). And it could be noted by envelop stamps.

IMHO, both of this cases is not about serializers, but about transports.

Tobion · 2019-05-30T01:48:20Z

Configuring a default_type for a transport like in #31230 (comment) is not a good solution to me.

the data might be in a format that the symfony serializer does not understand, so a default type wouldn't help
the serializer could be php serializer that was introduced. in this case the default_type does not fit at all since the expected type is in the data and can't be changed with a default_type.

So configuring a custom serializer is the only option, esp. if you want to deserialize into a custom type. What could be useful in core is a serializer that more easily allows to consume third party messages without having to create a custom serializer. It could simply create a fixed GenericMessage object that gives access to the raw body and headers. This way, you can consume those messages and wire your handlers using the GenericMessage class.

dvc · 2019-05-30T06:44:54Z

@Tobion, lets separate use cases:

my symfony app just proxy. I don't plan to analyze messages in deep. GenericMessage is good for me for some/any transport and any message type from some/any transport.
my symfony app is consumer for existing third party message queue with well known exchange address and only one message type in it, but without required type header. I can't change message structure in queue. But i beleve in stability of queue configuration. Changes in exchange should break my app.
So, i don't need custom deserialize logic for concrete transport. I need only add type header (if not exists may be?) in message for default serializer. Why not? It just metadata. Just for concrete transport. One-to-one.

Lets say about ability to adding any custom headers by transport for any needs. RequestId, sourceId, message encoding, translate key,.. anything. Some of this requires functions support. Hooks/events or middlewares. This headers should be added to envelop, imho.

BTW, i'm not sure about php serializer. I think it should works well because of zero config on serializer level.

And. Invalid message content (invalid default or custom type header) - is real use case. It requires custom ExceptionEvent (with envelop).

Tobion · 2019-05-30T16:12:29Z

I agree with those use-cases. For 1) I proposed a solution. For 2) I think a custom serializer is the correct solution. The question is only if we need to make it even easier to create custom serializers with different classes to deserialize into, so you could configure transport x to deserialize all messages to class y.

dvc · 2019-05-31T07:10:15Z

Well, i am not insist on my variant. Convenient serializer configuration is good solution.

sroze · 2019-06-02T15:44:39Z

But this configuration affects also amqp.out transport and all amqp. Because of using Transport factory. I have no information in transport factory to customize concrete transport with defaults for receive type.

Now (in 4.3) you can configure the Serializer service per transport. So you can easily have your only for one transport. If think that this closes the issue, I agree with @Tobion on the other points :)

Nyholm · 2019-08-22T15:50:57Z

This issue is getting referenced time to time on Slack. I would like to share my custom built serializer just for this purpose: https://github.com/Happyr/message-serializer

It basically does a json_encode but also forces you do add a message type/id and a version.

xabbuh added Feature Messenger labels Apr 24, 2019

Tobion changed the title ~~[Messenger] add default ['headers']['type'] on recieve if empty~~ [Messenger] deserialize third-party messages into a generic class for consumption May 30, 2019

sroze closed this as completed Jun 2, 2019

Tobion mentioned this issue Jun 14, 2019

[Messenger] make it easier to consume third-party messages #32049

Closed

Tobion mentioned this issue Jul 17, 2019

[Messenger] pass the transport name to the factory #32526

Closed

Tobion mentioned this issue Aug 29, 2019

[Messenger] Very naive serialization #33368

Closed

pounard mentioned this issue Aug 29, 2019

[Messenger] Message cannot be bare strings or empty, message type cannot be an arbitrary string #33369

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Messenger] deserialize third-party messages into a generic class for consumption #31230

[Messenger] deserialize third-party messages into a generic class for consumption #31230

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[Messenger] deserialize third-party messages into a generic class for consumption #31230

[Messenger] deserialize third-party messages into a generic class for consumption #31230

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!