[Lambda DevX] Introducing Lambda Debug Mode Config #11363

MEPalma · 2024-08-15T18:56:17Z

Motivation

Currently, Lambda Debug Mode supports debugging of only one Lambda function at a time, using a remote debugger connected via a port specified through Lambda Docker Flags during startup.

The proposed changes introduce a new configuration feature for Lambda Debug Mode that enables users to assign specific debug ports to each Lambda function, as well as other parameters. This is achieved though a yaml file, such as:

functions:
  arn:aws:lambda:eu-central-1:000000000000:function:foo:
    debug-port: 19891
  arn:aws:lambda:eu-central-1:000000000000:function:bar:
    debug-port: 19892
    enforce-timeouts: true

In this initial release, users can also disable the automatic timeout extensions that are normally applied by starting LocalStack with Lambda Debug Mode on.

The configuration can be passed to LocalStack through the new LAMBDA_DEBUG_MODE_CONFIG_PATH environment variable, which once set is only sourced and reflected in the running logic of the app is LAMBDA_DEBUG_MODE is set.

Changes

Added a new environment variable LAMBDA_DEBUG_MODE_CONFIG_PATH which sources the yaml Lambda Debug Mode Config file
Added courtesy query function for various debug parameters of lambda functions
Added a new pydantic based specification of Lambda Debug Mode Config files, including mechanisms to parse and validate the file provided through LAMBDA_DEBUG_MODE_CONFIG_PATH, as well as useful error messaging logs for each of these steps
Added a singleton lambda debug session object which allows the rest of the system to interact with details of the lambda debug configuration
Added various non state affecting changes to configure timeouts and debug ports of lambda functions and containers
Unit testing for base lambda debug mode config file parsing

Testing

We do not yet have the necessary stack to test for lambda debugging integration tests. What follows is a guide to manually test a base scenario of this feature.

Deploy Lambda Functions

Deploy two Python Lambda functions, each with an evaluation timeout set to 1 second.
These functions are integrated with API Gateway, which has a 50ms timeout.

Example Lambda Function 1:

DEBUGGER_ENDPOINT = "0.0.0.0"
DEBUGGER_PORT = 19891

def handler_1(event, context):
    """Lambda handler that will get invoked by the LocalStack runtime"""

    # wait for the debugger to get attached
    import debugpy
    debugpy.listen((DEBUGGER_ENDPOINT, DEBUGGER_PORT))
    debugpy.wait_for_client()  # blocks execution until client is attached

    # print the incoming invocation event
    print(event)
    return {"body": "Hello from handler_1"}

Example Lambda Function 2:

DEBUGGER_ENDPOINT = "0.0.0.0"
DEBUGGER_PORT = 19892

def handler_2(event, context):
    """Lambda handler that will get invoked by the LocalStack runtime"""

    # wait for the debugger to get attached
    import debugpy
    debugpy.listen((DEBUGGER_ENDPOINT, DEBUGGER_PORT))
    debugpy.wait_for_client()  # blocks execution until client is attached

    # print the incoming invocation event
    print(event)
    return {"body": "Hello from handler_2"}

Configure Debugging

Use the Lambda Debug Mode Config File to set up the debugging environment:
- Lambda Function 1 debugged on port 19891.
- Lambda Function 2 debugged on port 19892.
Lambda Debug Mode Config file

functions:
  arn:aws:lambda:eu-central-1:000000000000:function:handler_1-handler_1:
    debug-port: 19891
  arn:aws:lambda:eu-central-1:000000000000:function:handler_2-handler_2:
    debug-port: 19892

Enable Lambda Debug Mode
- Activate Lambda Debug Mode to adjust runtime evaluation parameters.
- How:
```
LAMBDA_DEBUG_MODE=1
LAMBDA_DEBUG_MODE_PATH=/path/to/file/debug_config.yaml
```
Debug
- Invoke Lambda Function 1 through ApiGateway
- Verify the system hangs and the specified timeouts are not respected for ApiGateway nor Lambda
- Connect remotely to the specified debug port
- Verify that both Lambda functions can be debugged and upon termination the function returns the expected result
- Repeat for Lambda Function 2

github-actions · 2024-08-15T19:17:16Z

S3 Image Test Results (AMD64 / ARM64)

2 files 2 suites 3m 28s ⏱️
420 tests 368 ✅ 52 💤 0 ❌
840 runs 736 ✅ 104 💤 0 ❌

Results for commit ae01009.

♻️ This comment has been updated with latest results.

github-actions · 2024-08-15T20:04:25Z

LocalStack Community integration with Pro

2 files ±0 2 suites ±0 1h 36m 7s ⏱️ + 1m 2s
3 412 tests ±0 3 014 ✅ ±0 398 💤 ±0 0 ❌ ±0
3 414 runs ±0 3 014 ✅ ±0 400 💤 ±0 0 ❌ ±0

Results for commit ae01009. ± Comparison against base commit 627c27c.

♻️ This comment has been updated with latest results.

joe4dev

Excellent work enabling multi-function debugging for Lambda 👏👏
Lovely validations and config test coverage 💯

We should enable telemetry for the debug flag and add the new configs to the config list for CLI compatibility. Otherwise, nothing blocking a merge (just nits, discussions).

Given the lack of automated testing (for now), I highly recommend adding a ## Testing section with manual instructions including suggested testing steps for reviewers, potentially linking to an example (the example you demonstrated in our Tue session would be very suitable).

I would appreciate an extra pair of 👀 from @dfangl on this one ✨

joe4dev · 2024-08-22T08:51:17Z

localstack-core/localstack/services/lambda_/invocation/executor_endpoint.py

        if is_lambda_debug_mode():
            timeout_seconds = DEFAULT_LAMBDA_DEBUG_MODE_TIMEOUT_SECONDS
        else:
-            # TODO: integration timeouts should be enforced instead.
-            # Do not wait longer for an invoke than the maximum lambda timeout plus a buffer


nit: should we keep the buffer explanation here (it still applies)?

joe4dev · 2024-08-22T08:53:46Z

localstack-core/localstack/services/lambda_/invocation/executor_endpoint.py

+        # The following logic selects which maximum waiting time to consider depending
+        # on whether the application is being debugged or not. Note however, that if
+        # debugging timeouts are disabled for the lambda function invoked at this endpoint,
+        # the lambda function will already enforce the expected timeouts.
        if is_lambda_debug_mode():
            timeout_seconds = DEFAULT_LAMBDA_DEBUG_MODE_TIMEOUT_SECONDS


nit: could we add a clarifying comment as to why we still set a timeout here in debug mode? You motivated a good reason such that we ensure the hanging container eventually gets cleaned up.

It's mentioned in the first sentence above, but might be worth explicitly pointing out why this also applies to debug mode (feel free to ignore if it's too nit-picky; just wanted to perverse our motivation here)

The explanation for DEFAULT_LAMBDA_DEBUG_MODE_TIMEOUT_SECONDS clarifies 👍

joe4dev · 2024-08-22T08:56:24Z

localstack-core/localstack/services/lambda_/invocation/executor_endpoint.py

+        # The following logic selects which maximum waiting time to consider depending
+        # on whether the application is being debugged or not. Note however, that if
+        # debugging timeouts are disabled for the lambda function invoked at this endpoint,
+        # the lambda function will already enforce the expected timeouts.


I don't fully understand the last part of the sentence, this refers to the timeouts enforced by the Lambda RIE, right?

correct -- I have adjusted this comment for clarity

localstack-core/localstack/utils/lambda_debug_mode/lambda_debug_mode_config.py

joe4dev · 2024-08-22T11:39:30Z

localstack-core/localstack/utils/lambda_debug_mode/lambda_debug_mode_config.py

+
+class LambdaDebugConfig(BaseModel):
+    debug_port: Optional[int] = Field(None, alias="debug-port")
+    timeout_disable: bool = Field(False, alias="timeout-disable")


Glancing over our existing LocalStack configurations, we typically use DISABLE as a prefix. Would it be more consistent to use disable-timeout instead?

joe4dev · 2024-08-22T11:57:04Z

localstack-core/localstack/utils/lambda_debug_mode/lambda_debug_mode_config.py

+    functions: dict[Arn, LambdaDebugConfig]
+
+
+class LambdaDebugModeConfigException(Exception): ...


Awesome validations and clear error messages 💯 👏👏

joe4dev · 2024-08-22T12:04:13Z

localstack-core/localstack/utils/lambda_debug_mode/lambda_debug_mode_session.py

+
+
+class _LambdaDebugModeSession:
+    _instance = None


@dfangl Does this match our common way of creating singletons?

No, we usually create singletons with a .get() function:

@staticmethod @singleton_factory def get() -> "_LambdaDebugModeSession": """Returns a singleton instance of the debug mode session.""" return _LambdaDebugModeSession()

I would prefer to avoid overriding __new__ here. Instead ofusing the module global, we can always call _LambdaDebugModeSession.get(). Might be nice to rename it to avoid the leading _ then, though.

joe4dev · 2024-08-22T12:06:24Z

tests/unit/lambda_debug_mode/test_config_parsing.py

+    debug-port: null
+"""
+
+DEBUG_CONFIG_NULL_TIMEOUT_DISABLE = """


Should this contain a timeout disable flag?

joe4dev · 2024-08-22T12:21:50Z

localstack-core/localstack/config.py

@@ -433,6 +433,9 @@ def in_docker():
 # When enabled it triggers specialised workflows for the debugging.
 LAMBDA_DEBUG_MODE = is_env_true("LAMBDA_DEBUG_MODE")


Can we add LAMBDA_DEBUG_MODE to the TRACKED_ENV_VAR in localstack-core/localstack/runtime/analytics.py for telemetry insights?

joe4dev · 2024-08-22T12:23:15Z

localstack-core/localstack/config.py

@@ -433,6 +433,9 @@ def in_docker():
 # When enabled it triggers specialised workflows for the debugging.
 LAMBDA_DEBUG_MODE = is_env_true("LAMBDA_DEBUG_MODE")

+# path to the lambda debug mode configuration file.
+LAMBDA_DEBUG_MODE_CONFIG_PATH = os.environ.get("LAMBDA_DEBUG_MODE_CONFIG_PATH")


Can we add all new environment variables to CONFIG_ENV_VARS in config.py at the bottom such that they work using the CLI?

joe4dev · 2024-08-22T15:42:58Z

localstack-core/localstack/utils/lambda_debug_mode/lambda_debug_mode_config.py

+
+class LambdaDebugConfig(BaseModel):
+    debug_port: Optional[int] = Field(None, alias="debug-port")
+    timeout_disable: bool = Field(False, alias="timeout-disable")


I'm wondering about the naming: Would enforce_timeouts: True|False be clearer? What's your naming suggestion @dfangl @dominikschubert ?

Current behavior:

LAMBDA_DEBUG_MODE=0 -> never disable any timeout

LAMBDA_DEBUG_MODE=1

timeout-disable not set or no debug config for Lambda function ARN available -> extend timeouts

timeout-disable=False -> extend timeouts

timeout-disable=True -> enforce timeouts as expected

Am I getting confused here or is this a logic error (the default value is False, which gets negated in is_lambda_debug_timeout_for) 🤔

sidenote: Technically, we're not fully disabling the timeouts, but extend them.

We need to document the behavior clearly after merging ...

PS: migrated from wrong follow-up PR

I agree with you here, it is confusing. For a variable called timeout-disable I would expect the flag being reversed. I personally prefer enforce_timeout in this case, but even if we reverse it, I am not a bit fan of timeout-disable, I think even disable-timouts would be clearer here.

dfangl

All in all, this is a great step towards improving DevX! I just had some minor nits, but nothing major!

dfangl · 2024-08-26T10:28:54Z

localstack-core/localstack/utils/lambda_debug_mode/lambda_debug_mode.py

+    return debug_configuration.debug_port
+
+
+def is_lambda_debug_timeout_for(lambda_arn: Arn) -> bool:


Would be nice to rename this function to be clearer, and maybe add docstrings to this entire module.

is_lambda_debug_timeout_for does not really make sense for me. Maybe is_lambda_debug_timeout_enabled_for?

dfangl · 2024-08-26T10:33:32Z

localstack-core/localstack/utils/lambda_debug_mode/lambda_debug_mode_config.py

+    if is_qualified and lambda_arn_parts[-1]:
+        return lambda_arn
+
+    # The arn is not unqualified, but probably erroneous, pass the value upstream.
+    is_unqualified = lambda_arn_parts_len == 7
+    if not is_unqualified:
+        return lambda_arn


As far as I can see, we just ignore invalid arns, for example those ending with : (so they have 8 parts, but are missing the qualifier, or totally invalid arns) and still set them in the internal config? Personally, I would prefer raising an error to silently accepting them, because they will never match anything.

dfangl · 2024-08-26T10:36:10Z

localstack-core/localstack/utils/lambda_debug_mode/lambda_debug_mode_session.py

+
+
+class _LambdaDebugModeSession:
+    _instance = None


No, we usually create singletons with a .get() function:

@staticmethod @singleton_factory def get() -> "_LambdaDebugModeSession": """Returns a singleton instance of the debug mode session.""" return _LambdaDebugModeSession()

I would prefer to avoid overriding __new__ here. Instead ofusing the module global, we can always call _LambdaDebugModeSession.get(). Might be nice to rename it to avoid the leading _ then, though.

dfangl · 2024-08-26T10:40:47Z

localstack-core/localstack/utils/lambda_debug_mode/lambda_debug_mode_config.py

+def _validate_lambda_debug_mode_config(
+    validation_state: _LambdaDebugModeConfigValidationState, config: LambdaDebugModeConfig
+):


This function is called validate_ but does actively change the values of the config (by replacing unqualified ARNs with qualified ARNs). It should be clear from the method that we are not only validating here, but also "canonicalizing". So maybe renaming it, or at least adding it in a docstring or something.

dfangl · 2024-08-26T10:43:03Z

tests/unit/lambda_debug_mode/test_config_parsing.py

+functions:
+  arn:aws:lambda:eu-central-1:000000000000:function:functionname:
+    debug-port: null
+    enforce-timeout: null


The flag is now called enforce-timeouts, not enforce-timeout. So we are not testing the null value here, but actually the default, as pydantic per default ignores extra fields.

MEPalma added 4 commits August 2, 2024 14:28

base support for debug mode

32478c3

refactor feature name

8d752ba

Merge branch 'MEP-debug_mode_base' into MEP-LambdaDevX-debug_config

d2aac31

support for lambda debug mode confi path

d34f369

MEPalma added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Aug 15, 2024

MEPalma added this to the 3.7 milestone Aug 15, 2024

MEPalma self-assigned this Aug 15, 2024

conflicts

d683f53

rename to lambda config field to functions, minors

7ab646a

MEPalma marked this pull request as ready for review August 21, 2024 08:23

MEPalma requested review from joe4dev, dominikschubert, dfangl and thrau as code owners August 21, 2024 08:23

unit tests

8cefb59

MEPalma mentioned this pull request Aug 21, 2024

[Lambda DevX] Control Lambda Container Startup Timeout when In Lambda Debug Mode #11398

Merged

joe4dev approved these changes Aug 22, 2024

View reviewed changes

joe4dev reviewed Aug 22, 2024

View reviewed changes

rename timeout flag, other review items

77b2dad

dfangl approved these changes Aug 26, 2024

View reviewed changes

MEPalma added 3 commits August 26, 2024 14:04

review items

878b776

Merge branch 'master' into MEP-LambdaDevX-debug_config

2c06d40

fix timeouts, docstr

c280c6e

MEPalma mentioned this pull request Aug 27, 2024

[Lambda DevX] Bypass Concurrency Settings #11418

Merged

MEPalma added 2 commits August 28, 2024 09:29

minor

a99f80b

Merge branch 'master' into MEP-LambdaDevX-debug_config

ae01009

MEPalma merged commit 8068e9d into master Aug 28, 2024
41 checks passed

MEPalma deleted the MEP-LambdaDevX-debug_config branch August 28, 2024 08:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Lambda DevX] Introducing Lambda Debug Mode Config #11363

[Lambda DevX] Introducing Lambda Debug Mode Config #11363

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

		functions: dict[Arn, LambdaDebugConfig]


		class LambdaDebugModeConfigException(Exception): ...

		@@ -433,6 +433,9 @@ def in_docker():
		# When enabled it triggers specialised workflows for the debugging.
		LAMBDA_DEBUG_MODE = is_env_true("LAMBDA_DEBUG_MODE")

		return debug_configuration.debug_port


		def is_lambda_debug_timeout_for(lambda_arn: Arn) -> bool:

Uh oh!

[Lambda DevX] Introducing Lambda Debug Mode Config #11363

[Lambda DevX] Introducing Lambda Debug Mode Config #11363

Uh oh!

Conversation

Uh oh!

Motivation

Changes

Testing

Uh oh!

Uh oh!

S3 Image Test Results (AMD64 / ARM64)

Uh oh!

Uh oh!

LocalStack Community integration with Pro

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!