Description
Key information
- RFC PR: (leave this empty)
- Related issue(s), if known: Feature request: Better failures handling while using both BatchProcessing and LargeMessageHandling #596
- Area: SQS
- Meet tenets: yes
Summary
We should handle large messages (with s3) for SQS and SNS and provide a better user experience (error handling, batch...) than today with the @SqsLargeMessage annotation.
Motivation
- Lambda Powertools Java does not handle large message for SNS (doc, code)
- Current annotation is global to the handler and handles all messages in a batch (code). Error handling is thus global. It would be better to have the annotation at the item level. See Feature request: Better failures handling while using both BatchProcessing and LargeMessageHandling #596.
- Large message handling is currently mixed up with batch processing. While they can be use jointly, large message handling is not needed in all batch cases. As it requires additional libraries (payloadoffloading-common, s3 sdk), it should be an independent module we add specifically for this purpose.
Proposal
SQS
This feature should be considered in conjunction with #797 but should be in an individual module (eg. powertools-large-messages ?). The aim of #797 is to provide a simple interface so that developers can focus on handling each message of the batch while the library takes care of failures. This module should permit to handle large messages individually.
if #797 is providing the following interface:
public void processMessage(SQSMessage message, Context context) {
}this new module should provide the annotation @LargeMessage to handle the message from S3 (implementation would be quite similar as today, replacing the body with the content of the s3 object):
@LargeMessage
public void processMessage(SQSMessage message, Context context) {
// message.getBody() will contain the content of the S3 object
}We can also keep the option to not delete the object in S3 (@LargeMessage(deletePayload = false)).
SNS
On SNS, despite SNSEvent has a list of records, there will always be only one message (see doc). The developer would have to create a method to handle a single SNSRecord (as first parameter):
@LargeMessage
public void processRecord(SNSRecord record, Context context) {
// record.getSNS().getMessage() will contain the content of the S3 object
}Drawbacks
For SNS, it forces the developer to have the following code in their handler:
public class App implements RequestHandler<SNSEvent, Object> {
@Override
public Void handleRequest(SNSEvent input, Context context) {
processRecord(input.getRecords().get(0), context);
return null;
}
@LargeMessage
public void processRecord(SNSRecord record, Context context) {
// record.getSNS().getMessage() will contain the content of the S3 object
}
}Rationale and alternatives
- should we handle the annotation at the handler level? Would be easier for SNS but degraded for SQS (handling failures in batch mode)
Unresolved questions
Metadata
Metadata
Assignees
Type
Projects
Status