New AWS client #7240

dfangl · 2022-11-24T15:27:47Z

Summary

The new AWS API client stack with support for sending additional metadata for internal calls.

We want to gradually move all functionality and use of aws_stack to this and remove it.

Supported Scenarios

Our new AWS client factory needs to take different scenarios of usage into account.

Scenarios

Internal Hidden

Request which either AWS does not do, or does not document in any way. Used for LS workarounds or completely hidden calls.

❌ CloudTrail
❌ IAM Enforcement
✔️ In Memory Possible

Internal as Service

Call of a service to another service as that service principal. Usually has a sourcearn of the resource on which behalf the request was made.

✔️ CloudTrail
✔️ IAM Enforcement
✔️ In memory possible

Internal as Role

Call of a service which assumed a provided role (assumption is enforced via “Internal as Service` itself) beforehand, as that role.

✔️ CloudTrail
✔️ IAM Enforcement
✔️ In memory possible

Internal as calling User

According to @dominikschubert this is the case for some CloudFormation functionality. Basically, CloudFormation does requests on behalf of the user which created the stack.

✔️ CloudTrail
✔️ IAM Enforcement
✔️ In memory possible

External for third-party providers

Used for communication to provider backends outside of our control. For example, dynamodblocal or kinesis-mock.

❌ CloudTrail
❌ IAM Enforcement
❌ In memory possible

External for testing

Used by testing clients to test our resources using boto3.

✔️ CloudTrail
✔️ IAM Enforcement
❌ In memory possible

Factories

This PR provides two subtypes of client factories.

Scenario 1-4 will be taken care of by the InternalClientFactory

Scenario 5-6 will be taken care of by the ExternalClientFactory

Sample usage

Calls made from outside of LocalStack (eg. in integration tests etc.)

from localstack.aws.connect import connect_to

client = connect_to().s3

client.list_objects(BucketName='foo')

Calls made internally across services

from localstack.aws.connect import connect_to

client = connect_to().s3

client.publish(
  TopicArn='arn:aws:sns:ap-south-1:000000000000:lorem', \
  Message='blah', \
  _ServicePrincipal='s3',
)

Performance

New client clocks faster for equivalent aws_stack functionality:

$ AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1 ipython
Python 3.10.8 (main, Nov 10 2022, 18:37:57) [GCC 12.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.11.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from localstack.aws.connect import connect_to, connect_externally_to
   ...:
   ...: def test():
   ...:     c = connect_to(region_name='eu-central-1').ec2
   ...:     d = connect_to(region_name='eu-central-1').ec2
   ...:     e = connect_to(region_name='ap-south-1').ec2
   ...:     f = connect_to(region_name='ap-south-1').ec2
   ...:     i = connect_externally_to(region_name=None, aws_access_key_id=None, aws_secret_access_key=No
8000
ne).dynamodb
   ...:     j = connect_externally_to(region_name=None, aws_access_key_id=None, aws_secret_access_key=None).dynamodb
   ...:     k = connect_externally_to(region_name='ap-south-1', aws_access_key_id='test', aws_secret_access_key='test').dynamodb
   ...: %timeit test()
   ...:
38.8 µs ± 225 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [2]: from localstack.utils.aws.aws_stack import connect_to_service, create_external_boto_client
   ...: def test_old():
   ...:     c = connect_to_service('ec2')
   ...:     d = connect_to_service('ec2')
   ...:     e = connect_to_service('ec2', region_name='ap-south-1')
   ...:     f = connect_to_service('ec2', region_name='ap-south-1')
   ...:     i = create_external_boto_client('dynamodb')
   ...:     j = create_external_boto_client('dynamodb')
   ...:     k = create_external_boto_client('dynamodb', region_name='ap-south-1', aws_access_key_id='test', aws_secret_access_key='test')
   ...: %timeit test_old()
   ...:
58.2 µs ± 251 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Other notes

Created by a mob programming session by @alexrashed @dominikschubert @simonrw @thrau @viren-nadkarni @dfangl

Sample use in DynamoDB: aws-client...aws-client-dynamodb

Tests

Unit tested, against a fake Gateway with the real handlers.
The unit tests aim to test the above mentioned scenarios, even though there is some duplication.
Some of the unit tests will in the next iteration be used to check some "convenience utilities", like creating a client directly for a to-be-assumed role, etc.

Potential future use for test clients:

# this is only a demonstration how a fixture could look like, and that the approach matches our requirements
class TestFactoryTestUsage:
    @pytest.fixture(scope="module")
    def test_client_factory(self):
        return ExternalClientFactory()

    @pytest.fixture(scope="class")
    def clients(self, test_client_factory):
        """Clients fixture which will allow all kind of clients to be initialized"""
        botocore_config = botocore.config.Config()

        # can't set the timeouts to 0 like in the AWS CLI because the underlying http client requires values > 0
        if os.environ.get("TEST_DISABLE_RETRIES_AND_TIMEOUTS"):
            botocore_config = botocore_config.merge(
                botocore.config.Config(
                    connect_timeout=1_000, read_timeout=1_000, retries={"total_max_attempts": 1}
                )
            )

        if os.environ.get("TEST_TARGET") == "AWS_CLOUD":
            return test_client_factory(config=botocore_config)

        return test_client_factory(
            region_name="us-east-1",
            aws_access_key_id="test",
            aws_secret_access_key="test",
            endpoint_url=config.get_edge_url(),
            config=botocore_config,
        )

    @pytest.mark.skip
    def test_something_with_boto_clients(self, clients):
        functions = clients.awslambda.list_functions()
        print(functions)

Future work

Add convenience method for assume a role and creating a client directly
Add convenience method proxy for type-safe adding of SourceArn and SourcePrincipal

github-actions · 2022-11-24T16:48:25Z

LocalStack integration with Pro

      2 files ±0       2 suites ±0 1h 35m 47s ⏱️ + 4m 32s
1 797 tests +5 1 412 ✔️ +5 385 💤 ±0 0 ❌ ±0
2 152 runs +5 1 587 ✔️ +5 565 💤 ±0 0 ❌ ±0

Results for commit db858a3. ± Comparison against base commit f4d39cf.

♻️ This comment has been updated with latest results.

coveralls · 2023-01-04T07:37:49Z

Coverage: 85.025% (-0.04%) from 85.066% when pulling dc7c36f on aws-client into f4d39cf on master.

tests/unit/aws/test_connect.py

localstack/aws/accounts.py

localstack/aws/handlers/internal_requests.py

Co-authored-by: Viren Nadkarni <viren.nadkarni@localstack.cloud>

thrau

looks great! let's get it over the line :-)

dominikschubert

Didn't look in detail over the tests, but rest LGTM 👍

dominikschubert · 2023-03-13T13:11:24Z

localstack/aws/connect.py

+    """
+    TODO
+    """


Suggested change

"""

TODO

"""

Ah yes, forgot about that thing. Will add a description!

alexrashed

This is great! The scenarios make it easier to think about the different ways of connecting two services, the typing will be great, and the tests are getting more and more extensive! 🚀
I only have a few smaller questions and comments (which in turn are mostly about comments in the code)...

localstack/aws/app.py

localstack/aws/handlers/internal_requests.py

localstack/aws/connect.py

alexrashed

Thanks for fixing all my pedantic and nitpicky comments! Great to see this new client being merged, this will change the way LocalStack services communicate with each other quite substantially! 🚀

add mob-programming based client prototype

add7928

dfangl temporarily deployed to localstack-ext-tests November 24, 2022 15:27 Inactive

< 8000 div class="TimelineItem-body"> viren-nadkarni self-assigned this Jan 2, 2023

WIP

f57429d

viren-nadkarni temporarily deployed to localstack-ext-tests January 2, 2023 14:29 — with GitHub Actions Inactive

viren-nadkarni mentioned this pull request Jan 4, 2023

bug: DynamoDB DescribeTable treats the queried table as a replica #7426

Closed

1 task

viren-nadkarni added 2 commits January 4, 2023 12:05

Merge branch 'master' into aws-client

ab0c79f

Fix imports

89e43f5

viren-nadkarni temporarily deployed to localstack-ext-tests January 4, 2023 06:38 — with GitHub Actions Inactive

Merge branch 'master' into aws-client

247340c

viren-nadkarni temporarily deployed to localstack-ext-tests January 6, 2023 06:56 — with GitHub Actions Inactive

Updates

0352fae

viren-nadkarni temporarily deployed to localstack-ext-tests January 6, 2023 12:32 — with GitHub Actions Inactive

Fallback to default internal credentials

548ced3

viren-nadkarni temporarily deployed to localstack-ext-tests January 12, 2023 09:39 — with GitHub Actions Inactive

Proper loading of default credentials

f6f37fa

viren-nadkarni temporarily deployed to localstack-ext-tests January 12, 2023 10:32 — with GitHub Actions Inactive

Move to its own module

f392242

viren-nadkarni temporarily deployed to localstack-ext-tests January 12, 2023 15:40 — with GitHub Actions Inactive

Fix datetime

f8edc9c

viren-nadkarni temporarily deployed to localstack-ext-tests January 13, 2023 12:22 — with GitHub Actions Inactive

viren-nadkarni mentioned this pull request Jan 13, 2023

Use new AWS client in SES #7484

Closed

viren-nadkarni changed the title ~~Add boto client creation via factory pattern~~ New internal AWS client Jan 16, 2023

Allow module to be used for external clients also

b53e068

viren-nadkarni temporarily deployed to localstack-ext-tests January 16, 2023 10:15 — with GitHub Actions Inactive

viren-nadkarni changed the title ~~New internal AWS client~~ New AWS client Jan 16, 2023

Use headers for internal call arg func

f8d8d8e

viren-nadkarni temporarily deployed to localstack-ext-tests January 16, 2023 11:50 — with GitHub Actions Inactive

dfangl self-assigned this Feb 24, 2023

simonrw reviewed Feb 24, 2023

View reviewed changes

tests/unit/aws/test_connect.py Outdated Show resolved Hide resolved

dfangl added 2 commits February 27, 2023 11:44

Merge branch 'master' into aws-client

f790be7

add some preliminary changes

48cdd07

dfangl temporarily deployed to localstack-ext-tests February 27, 2023 16:03 — with GitHub Actions Inactive

viren-nadkarni reviewed Mar 6, 2023

View reviewed changes

localstack/aws/accounts.py Outdated Show resolved Hide resolved

add typed interface, add some tests + some test scaffolds

5d49524

dfangl temporarily deployed to localstack-ext-tests March 9, 2023 18:08 — with GitHub Actions Inactive

Merge branch 'master' into aws-client

4b6ff91

viren-nadkarni reviewed Mar 13, 2023

View reviewed changes

localstack/aws/accounts.py Outdated Show resolved Hide resolved

viren-nadkarni reviewed Mar 13, 2023

View reviewed changes

localstack/aws/handlers/internal_requests.py Outdated Show resolved Hide resolved

add more tests, fix internal call detection

0641ae3

dfangl temporarily deployed to localstack-ext-tests March 13, 2023 11:08 — with GitHub Actions Inactive

Apply suggestions from code review

f10f983

Co-authored-by: Viren Nadkarni <viren.nadkarni@localstack.cloud>

dfangl temporarily deployed to localstack-ext-tests March 13, 2023 11:49 — with GitHub Actions Inactive

fix imports, remove unnecessary tests

db858a3

dfangl temporarily deployed to localstack-ext-tests March 13, 2023 12:16 — with GitHub Actions Inactive

dfangl requested review from viren-nadkarni and thrau March 13, 2023 12:19

thrau approved these changes Mar 13, 2023

View reviewed changes

viren-nadkarni approved these changes Mar 13, 2023

View reviewed changes

dominikschubert approved these changes Mar 13, 2023

View reviewed changes

alexrashed reviewed Mar 13, 2023

View reviewed changes

fix nits

dc7c36f

dfangl temporarily deployed to localstack-ext-tests March 13, 2023 16:54 — with GitHub Actions Inactive

alexrashed approved these changes Mar 13, 2023

View reviewed changes

dfangl merged commit 1195593 into master Mar 13, 2023

dfangl deleted the aws-client branch March 13, 2023 18:24

joe4dev mentioned this pull request Apr 4, 2023

Disable HTTPS(S) proxies for internal calls #8066

Merged

alexrashed mentioned this pull request Jan 17, 2025

[ESM] Handle polling of batches exceeding SQS message limits #12118

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

New AWS client #7240

New AWS client #7240

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

New AWS client #7240

New AWS client #7240

Uh oh!

Conversation

Uh oh!

Summary

Supported Scenarios

Scenarios

Internal Hidden

Internal as Service

Internal as Role

Internal as calling User

External for third-party providers

External for testing

Factories

Sample usage

Performance

Other notes

Tests

Future work

Uh oh!

Uh oh!

LocalStack integration with Pro

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!