migrate Kinesis to ASF #6166
Merged
migrate Kinesis to ASF #6166
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size. |
|
i rebased the branch and added a feature flag/deprecation path to the commit |
1 task
d7e7629 to
4148ab4
Compare
23fbfde to
c1e7779
Compare
e82f5e5 to
98b196e
Compare
98b196e to
5826dec
Compare
abff1b1 to
198b4d4
Compare
thrau
approved these changes
Aug 30, 2022
There was a problem hiding this comment.
As discussed with @alexrashed, we will merge this now to get it into v1.1, and at some later point, should the need arise, refactor the serializer API according to the feedback.
This was referenced Aug 30, 2022
Merged
macnev2013
pushed a commit
to macnev2013/localstack
that referenced
this pull request
Sep 4, 2022
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR migrates Kinesis to ASF. It uses the new streaming support in ASF introduced with #6086 for
SubscribeToShard.In addition, the ASF infrastructure needed to be enhanced with a limited support for HTTP Content-Negotiation in some serializers. For example, the JSON protocols can switch their content to CBOR if requested in the client's request.
Since we now implement more features than botocore's or moto's marshalling, this PR also needed to change the proxy handling to moto and other HTTP backends.
Now, all responses which have been forwarded to moto or any other backend using the fallback mechanism (i.e. the backend implements a specific ASF operation / API) are parsed (using botocore's parsers) and re-serialized (using our - enhanced - serializer).
An obvious downside of this approach is the performance: We have an additional marshalling roundtrip for the responses coming from a fallback backend.
However, in order to properly implement the Content-Negotiation, there isn't really an alternative because we need to support additional features compared to our backends (like moto) as well as boto (which is used to serialize the requests to the fallback backend).
This PR also removes three
AwsApiListenerextensions (addresses #6668):KinesisApiListener(primary goal of the migration / refactoring)StsAwsApiListener(was also implementing a Content-Negotiation, but for the query protocol)LogsAwsApiListener(useless, the serializer already added the correct content type header)StepFunctionsApiListener(only glue between provider and fallback dispatcher)TODO:
Flip feature flag before merging into master(the ASF provider is directly becoming the default with the next minor version)