Open
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
I get the following docker log on all my lambda functions in Github actions:
localstack/prebuild-lambda-arn_<function-arn> (<function-name>)
**********************************************************************
* Name : <function-name>
* Image : <function-image>
* Status: Exited (0) Less than a second ago
**********************************************************************
time="2025-02-17T11:36:43Z" level=warning msg="First fatal error stored in appctx: Runtime.InvalidEntrypoint" func=go.amzn.com/lambda/appctx.StoreFirstFatalError file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/appctx/appctxutil.go:171"
time="2025-02-17T11:36:43Z" level=error msg="Init failed" func=go.amzn.com/lambda/rapid.handleInitError file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/exit.go:103" InvokeID= error="fork/exec /var/runtime/bootstrap: exec format error"
time="2025-02-17T11:36:43Z" level=error msg="Runtime init failed to initialize: InitDoneFailed. Exiting." func=main.main file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/main.go:236"
Now the error in itself is clear enough, but after a lot of debugging, I do not see why. Running the same locally works fine. I will list the hunches I have had, and looked into so far, that could be a reason as to why the entrypoint is not found:
- Problems with the gradle build: I have verified that the build output is created, and looks similar as it does locally
- Problems with Lambda optimization layer: I have verified that the optimization zip is build correctly
- CDK failing on deploy: It does not. All successes all the way
- Config errors: There could still be something here, but since it works correctly when running locally, I doubt it being the culprit
- AWS config (awslocal, cdklocal, AWS_ENDPONT_URL etc etc.): I have double and triple checked all these, and validating the Locastack logs, it seems like everything i communicating well.
Expected Behavior
I would expect lambdas to work in Localstack in Github actions the same way as it does locally.
How are you starting LocalStack?
With a docker-compose file
Steps To Reproduce
How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)
docker compose up -d
With the following docker compose:
localstack:
image: localstack/localstack:latest
ports:
- "127.0.0.1:4566:4566"
environment:
LOCALSTACK_DOCKER_HOST: unix:///var/run/docker.sock
LOCALSTACK_GATEWAY_LISTEN: 0.0.0.0:4566
LOCALSTACK_SERVICES: sqs, lambda, ssm, cloudformation, ec2, route53, iam, sns, events, logs, secretsmanager, apigateway
LOCALSTACK_AWS_DEFAULT_REGION: eu-north-1
LOCALSTACK_EAGER_SERVICE_LOADING: 1 # Load localstack on startup rather than on demand
LOCALSTACK_LAMBDA_LIMITS_CODE_SIZE_ZIPPED: 500000000
LOCALSTACK_LAMBDA_LIMITS_CREATE_FUNCTION_REQUEST_SIZE: 500000000
LOCALSTACK_LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT: 30
LOCALSTACK_LAMBDA_PREBUILD_IMAGES: 1
volumes:
# Startup script for localstack
- "./localstack-setup.sh:/etc/localstack/init/ready.d/script.sh"
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock" # Mount docker socket to allow localstack to enable Lambda support
healthcheck:
test: "curl --silent --fail localstack:4566/_localstack/health | grep '\"lambda\": \"available\"'"
interval: 30s
retries: 3
start_period: 10s
timeout: 2sAnd the following startup-script
#!/bin/sh
echo "Initializing localstack"
echo "Setting up secrets"
awslocal secretsmanager create-secret --name <secret-1> --secret-string '<redacted>'
awslocal secretsmanager create-secret --name <secret-2> --secret-string '<redacted>'
awslocal secretsmanager create-secret --name <secret-3> --secret-string '<redacted>'
awslocal secretsmanager create-secret --name <secret-4> --secret-string '<redacted>'
echo "Setup complete"Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)
Since this is in Github actions, I have a on-pull-request file that again runs a script:
integration-test:
name: Integration Tests
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup JDK
uses: actions/setup-java@v4
with:
java-version: '21'
distribution: 'corretto'
cache: 'gradle'
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: 18
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Install CDK
run: |
npm install -g aws-cdk-local aws-cdk
cdklocal --version
- name: Install dependencies
run: |
pip3 install awscli awscli-local
- name: Build and CDK setup
run: |
mkdir -p ~/.aws
echo "[integration-test]" >> ~/.aws/credentials
echo "aws_access_key_id=test" >> ~/.aws/credentials
echo "aws_secret_access_key=test" >> ~/.aws/credentials
echo "aws_session_token=test" >> ~/.aws/credentials
echo "[default]" >> ~/.aws/config
echo "region=eu-north-1" >> ~/.aws/config
./gradlew build --no-daemon
cd local-development
./start-local-environment.sh
cd ..
./gradlew integrationTest
- name: Dump docker logs on failure
if: failure()
uses: jwalton/gh-docker-logs@v2
start-local-environment.sh
docker compose up -d
echo -e "Bootstrapping local CDK"
cdklocal bootstrap --profile integration-test
echo -e "Synthesize stack"
cdklocal synth --profile integration-test
echo -e "Deploying stack"
cdklocal deploy "<stack-prefix>/*" --require-approval never --profile integration-test
echo -e "Migrating Database"
awslocal lambda invoke --region eu-north-1 --endpoint-url http://localhost:4566 output.txt --function-name DatabaseMigrationEnvironment
- OS: Github Actions ubuntu-latest
- LocalStack version: 4.1.2.dev22
LocalStack build date: 2025-02-17
LocalStack build git hash: 2d7ba1b23Anything else?
Now, for some context, here is a little info about the stack:
- Its a Kotlin monorepo
- Built with gradle
- Deployed with CDK
- Localstack is started with Docker compose
- Lambdas are deployed as zip with an OptimizationLayer
- The problem only occurs on Github actions. It works locally