8000 upgrade cryptography and amazon-kclpy by alexrashed · Pull Request #12005 · localstack/localstack · GitHub
[go: up one dir, main page]
Skip to content

upgrade cryptography and amazon-kclpy #12005

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Dec 11, 2024
Merged

upgrade cryptography and amazon-kclpy #12005

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b942019
remove limit on cryptography, update all lock files
alexrashed Dec 9, 2024
3ed707e
upgrade kclipy helper for v3 in compat mode
alexrashed Dec 10, 2024
e0aa05e
increase timeout on flaky test_firehose_kinesis_to_s3
alexrashed Dec 10, 2024
e775b46
upgrade kclpy min version, optimize config
alexrashed Dec 10, 2024
a3c8398
re-add v2 compatibility mode for KCL
alexrashed Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions localstack-core/localstack/utils/kinesis/kclipy_helper.py
View file
Open in desktop
< 8000 td id="diff-1955075f1c980583efb2e35a63e844a3e1d7c5aa5eec6b4adedf5ee5bb0eb8c4L3" data-line-number="3" class="blob-num blob-num-context js-linkable-line-number">
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env python

import os
import sys
from glob import glob

from amazon_kclpy import kcl
Expand Down Expand Up @@ -93,16 +94,18 @@ def create_config_file(
**kwargs,
):
if not credentialsProvider:
credentialsProvider = "DefaultAWSCredentialsProviderChain"
credentialsProvider = "DefaultCredentialsProvider"
# TODO properly migrate to v3 of KCL and remove the clientVersionConfig
content = f"""
executableName = {executableName}
streamName = {streamName}
applicationName = {applicationName}
AWSCredentialsProvider = {credentialsProvider}
clientVersionConfig = CLIENT_VERSION_CONFIG_COMPATIBLE_WITH_2x
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it necessary to set this?
The AWS help page says:

This KCL 2.x compatibility setting allows your KCL 3.x application to run in a mode compatible with KCL 2.x and use the load balancing algorithm for KCL 2.x until all workers in your consumer application have been upgraded to KCL 3.x. When the migration is complete, KCL will automatically switch to full KCL 3.x functionality mode and start using a new KCL 3.x load balancing algorithm for all running workers.

https://docs.aws.amazon.com/streams/latest/dev/kcl-migration-from-2-3.html

Since we are not migrating anything, using the new load distribution algorithm should have almost no impact on us (unless it uses unsupported features), right?

kinesisCredentialsProvider = {credentialsProvider}
dynamoDBCredentialsProvider = {credentialsProvider}
cloudWatchCredentialsProvider = {credentialsProvider}
processingLanguage = python/3.10
processingLanguage = python/{sys.version_info.major}.{sys.version_info.minor}
shardSyncIntervalMillis = 2000
parentShardPollIntervalMillis = 2000
idleTimeBetweenReadsInMillis = 1000
Expand Down
6 changes: 2 additions & 4 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,15 +80,13 @@ runtime = [
# pinned / updated by ASF update action
"awscli>=1.32.117",
"airspeed-ext>=0.6.3",
# TODO upgrade to kclpy 3+
"amazon_kclpy>=2.0.6,!=2.1.0,!=2.1.4,<3.0.0",
"amazon_kclpy>=3.0.0",
# antlr4-python3-runtime: exact pin because antlr4 runtime is tightly coupled to the generated parser code
"antlr4-python3-runtime==4.13.2",
"apispec>=5.1.1",
"aws-sam-translator>=1.15.1",
"crontab>=0.22.6",
# TODO remove upper limit once https://github.com/getmoto/moto/pull/7876 is in our moto-ext version
"cryptography>=41.0.5,<43.0.0",
"cryptography>=41.0.5",
# allow Python programs full access to Java class libraries. Used for opt-in event ruler.
"jpype1-ext>=0.0.1",
"json5>=0.9.11",
Expand Down
4 changes: 2 additions & 2 deletions requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
airspeed-ext==0.6.7
# via localstack-core
amazon-kclpy==2.1.5
amazon-kclpy==3.0.1
# via localstack-core
annotated-types==0.7.0
# via pydantic
Expand Down Expand Up @@ -107,7 +107,7 @@ coveralls==4.0.1
# via localstack-core (pyproject.toml)
crontab==1.0.1
# via localstack-core
cryptography==42.0.8
cryptography==44.0.0
# via
# joserfc
# localstack-core
Expand Down
4 changes: 2 additions & 2 deletions 8000 requirements-runtime.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
airspeed-ext==0.6.7
# via localstack-core (pyproject.toml)
amazon-kclpy==2.1.5
amazon-kclpy==3.0.1
# via localstack-core (pyproject.toml)
annotated-types==0.7.0
# via pydantic
Expand Down Expand Up @@ -78,7 +78,7 @@ constantly==23.10.4
# via localstack-twisted
crontab==1.0.1
# via localstack-core (pyproject.toml)
cryptography==42.0.8
cryptography==44.0.0
# via
# joserfc
# localstack-core
Expand Down
4 changes: 2 additions & 2 deletions requirements-test.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
airspeed-ext==0.6.7
# via localstack-core
amazon-kclpy==2.1.5
amazon-kclpy==3.0.1
# via localstack-core
annotated-types==0.7.0
# via pydantic
Expand Down Expand Up @@ -101,7 +101,7 @@ coverage==7.6.9
# via localstack-core (pyproject.toml)
crontab==1.0.1
# via localstack-core
cryptography==42.0.8
cryptography==44.0.0
# via
# joserfc
# localstack-core
Expand Down
4 changes: 2 additions & 2 deletions requirements-typehint.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
airspeed-ext==0.6.7
# via localstack-core
amazon-kclpy==2.1.5
amazon-kclpy==3.0.1
# via localstack-core
annotated-types==0.7.0
# via pydantic
Expand Down Expand Up @@ -111,7 +111,7 @@ coveralls==4.0.1
# via localstack-core
crontab==1.0.1
# via localstack-core
cryptography==42.0.8
cryptography==44.0.0
# via
# joserfc
# localstack-core
Expand Down
2 changes: 1 addition & 1 deletion tests/aws/test_integration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ def _assert_active():
)
assert stream_info["DeliveryStreamDescription"]["DeliveryStreamStatus"] == "ACTIVE"

retry(_assert_active, sleep=1, retries=30)
retry(_assert_active, sleep=1, retries=60)

# create target S3 bucket
s3_create_bucket(Bucket=TEST_BUCKET_NAME)
4603 Expand Down
Loading
0