Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
When I subscribe to a Kinesis stream, I stop seeing messages come through after 60 seconds, even though the request remains open.
Expected Behavior
On AWS, Kinesis streams messages for 5 minutes, then upstream closes the request.
How are you starting LocalStack?
With a docker-compose file
Steps To Reproduce
Docker Compose service:
kinesis-mock:
image: localstack/localstack
environment:
SERVICES: kinesis
LS_LOG: INFO
KINESIS_INITIALIZE_STREAMS: stream1:1:ap-southeast-2
healthcheck:
test: "curl --fail http://localhost:4568/healthcheck || exit 1"
interval: 5s
timeout: 5s
retries: 20
volumes:
- ../localstack/localstack:/opt/code/localstack/localstack
ports:
- "4566:4566"# Start LocalStack
docker compose up kinesis-mock
# Create a stream consumer
awslocal kinesis register-stream-consumer --region ap-southeast-2 --stream-arn arn:aws:kinesis:ap-southeast-2:000000000000:stream/stream1 --consumer-name my-consumer
# Subscribe to the stream (from the current time) - emulating an AWS SDK, and a simpler demonstration than using one
curl -sSN http://localhost:4566/ -H "X-Amz-Target: Kinesis_20131202.SubscribeToShard" -H "X-Amz-Date: 20220407T094445Z" -H "Authorization: AWS4-HMAC-SHA256 Credential=test/20220407/ap-southeast-2/kinesis/aws4_request, SignedHeaders=x, Signature=x" -d '{"ConsumerARN":"'"$(awslocal kinesis list-stream-consumers --region ap-southeast-2 --stream-arn arn:aws:kinesis:ap-southeast-2:000000000000:stream/stream1 | jq -r '.Consumers[0].ConsumerARN')"'","ShardId":"shardId-000000000000","StartingPosition":{"Type":"AT_TIMESTAMP","Timestamp":'"$(ruby -e 'puts Time.now.to_i')"'}}' --http2-prior-knowledge -v | strings
# In another terminal, start submitting records
while :; do; awslocal kinesis put-record --region ap-southeast-2 --stream-name stream1 --data "$(date +%Y+%m+%dT%H+%M+%S)=" --partition-key x; sleep 1; doneEnvironment
- OS: Arch Linux (MacOS for @oeoeaio)
- LocalStack:
- Docker image `sha256:1f3ed54d2e69478bad9efe8836577050b55ee0c8de4165a75e8dcc62eac1aaae`
- Mounting in the latest master (77a9b615e7347f4c3cee5fd84a31881ff530e9c1)Anything else?
@oeoeaio and I have spent a while debugging LocalStack, and determined that it's caused by the request timing out. We have a WIP fix here. Basically:
-
Disabling request timeout globally allows streaming for longer than 60 seconds:
app.config["RESPONSE_TIMEOUT"] = None
But it would probably be better to scope this to just the streaming request. We have very little Python/Quart experience, so are having trouble finding a way to do that.
-
The stream is then closed after ~2mins 30secs, which is half of the intended 5 minutes - due to the one second sleep only happening when there are no records returned from the poll. So I've adjusted this to end the loop based on time rather than iteration count.
With those changes applied, it then correctly matches the behaviour of AWS Kinesis.