Description
Is this related to an existing feature request or issue?
Which Powertools for AWS Lambda (Python) utility does this relate to?
Event Source Data Classes
Summary
This RFC proposes the implementation of a generic way to decode/unwrap nested events in Event Source Data Classes. When handling events, it may happen that the event passed into the lambda_handler wraps another event that was originated on a different resource. For example, an SQSEvent that wraps an event originated on Amazon S3.
The goal is to simplify how developers access and handle nested events, even when they are multi-level.
Use case
At this point, when the developer decorates the lambda_handler with @event_source, they declare the data_class for the event that invoked the Lambda function. If this event wraps another event - or a chain of events originated somewhere else, the developer has to manually instantiate new objects using the appropriate Event Source Data Classes.
This was partially addressed by #2348, but it covers only SQSEvents wrapping S3Events and SNSEvents. In practice, there can be dozens of different combinations of events, with more to come as AWS adds new features and use cases to the services. Moreover, nested events as such may be multi-level: S3 -> SNS -> SQS.
Implementing on Powertools each possible combination is not feasible, therefore I'm proposing a generic implementation that would also facilitate how developers are using this utility. With this approach, the developer would be able to traverse the chain of events until the root event, and still have access to relevant attributes from the parent i.e., the event wrapper.
Proposal
The proposal is to have a class called EventWrapper extending DictWrapper. This new class has the generic implementation to decode/unwrap nested events. Each relevant Event Source Data Class, such as SNSEvent, extends EventWrapper instead of DictWrapper. With that, SNSEvent (and the other relevant data classes) inherits a method called decode_nested_events (or decode_nested_event in singular - I'll elaborate on the singular vs. plural further down on the Potential Challenges section).
The method decode_nested_events accepts nested_event_class (the wrapped data class), and optionally the nested_event_content_deserializer (the implementation of the nested content deserializer, that can be in the format of JSON, Base64, GZIP, etc.). If no nested_event_content_deserializer is passed, the default is self._json_deserializer. For the latter, I intend to add implementations for the most common deserializers in EventWrapper. An example of that is compressed JSON CloudWatch Logs data coming in KinesisStreamEvents.
Here is a mock implementation (with no types yet):
def decode_nested_events(
self,
nested_event_class,
nested_event_content_deserializer = None):
if nested_event_content_deserializer is None:
nested_event_content_deserializer = self._json_deserializer
for content in self.nested_event_contents:
yield nested_event_class(nested_event_content_deserializer(content))I'm also planning to add a reference to the parent event into the nested event, so that the chain of events can be traversed back and forth. Another shortcut that I'm planning to add is a direct access to the root event. For the chain S3 -> SNS -> SQS, there would be a method decode_root_event to return an S3Event directly from the SQSEvent.
Here is a mock implementation of nested_event_contents:
@property
def nested_event_contents(self):
for record in self["Records"]:
yield record["body"]The implementation of nested_event_contents assumes that self has an attribute called Records, and each Record has the attribute body with the actual content of the nested event. At this point, I assume this is the most common use case, but I know there are exceptions, hence this mock implementation may change. Nevertheless, the outliers have the possibility to override just this method to return the right content. An example of that is SNSEvent that would override that to return self.sns_message instead.
Note: Based on the analysis I've done so far, we can implement the proposal on this RFC without breaking changes.
Out of scope
Nothing so far.
Potential challenges
Plural vs. singular
One implementation challenge that affects the UX is the JSON format of these events. For instance, the JSON object representing an SQS event has the attribute Records. Within each Record, there is a nested event. As a consequence, using the same example, SQSEvent contains a collection of nested events instead of just one. Note that in the sample code above, I return an Iterator instead of just one event because of that.
In practice, there are multiple use cases that we empirically know that the event contains just one single Record. With that in mind, EventWrapper could provide a method called decode_nested_event (in singular) to allow decoding the nested evant in the Record[0]. This aims to allow an experience as such: event.decode_nested_event(SNSEvent).decode_nested_event(S3Event) to get direct access to the S3Event wrapped within an SNSEvent with is wrapped within an SQSEvent.
For the Iterable version (plural), the UX would be similar to (consider that event is an instance of SQSEvent):
sns_events = event.decode_nested_events(SNSEvent)
for sns_event in sns_events:
s3_events = sns_event.decode_nested_events(S3Event)
for s3_event in s3_events:
print(s3_event.bucket.name)Dependencies and Integrations
No response
Alternative solutions
No response
Acknowledgment
- This feature request meets Powertools for AWS Lambda (Python) Tenets
- Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET