ECS credentials provider should not expect JSON response #2316
Description
Describe the bug
I have a very simple go service which initializes the V2 SDK and looks up a secret value from the secret manager. On my local workstation this works fine, but on an ECS fargate task this fails with:
panic: operation error Secrets Manager: GetSecretValue, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, failed to load credentials, deserialization failed, failed to decode error message, invalid character 'F' looking for beginning of value
Expected Behavior
To load a secret successfully
Current Behavior
The code panics with:
panic: operation error Secrets Manager: GetSecretValue, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, failed to load credentials, deserialization failed, failed to decode error message, invalid character 'F' looking for beginning of value
Reproduction Steps
Here is my function which runs successfully on my local workstation but fails on ECS:
func GetDbPassword() (string, error) {
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion(util.GetEnv("AWS_REGION", "us-east-2")))
if err != nil {
return "", err
}
manager := secretsmanager.NewFromConfig(cfg)
input := &secretsmanager.GetSecretValueInput{
SecretId: aws.String("db-password"),
VersionStage: aws.String("AWSCURRENT"),
}
result, err := manager.GetSecretValue(context.TODO(), input)
if err != nil {
return "", err
}
return *result.SecretString, nil
}
The ECS task is configured with the following permissions in terraform:
resource "aws_iam_role" "backend-task-role" {
name = "backend-task-role"
assume_role_policy = jsonencode(
{
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = [
"ec2.amazonaws.com",
"ecs-tasks.amazonaws.com"
]
}
},
]
Version = "2012-10-17"
}
)
}
resource "aws_iam_role_policy_attachment" "backend-task-role-secrets" {
role = aws_iam_role.backend-task-role.name
policy_arn = "arn:aws:iam::aws:policy/SecretsManagerReadWrite"
}
resource "aws_ecs_task_definition" "api" {
family = "api"
task_role_arn = aws_iam_role.backend-task-role.arn
...
}
Possible Solution
No response
Additional Information/Context
No response
AWS Go SDK V2 Module Versions Used
github.com/aws/aws-sdk-go-v2 v1.21.2
github.com/aws/aws-sdk-go-v2/config v1.19.0
github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.21.5
github.com/aws/aws-sdk-go-v2/credentials v1.13.43 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.13 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.43 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.37 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.45 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.37 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.15.2 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.17.3 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.23.2 // indirect
github.com/aws/smithy-go v1.15.0 // indirect
Compiler and Version used
go version go1.21.1 linux/amd64
Operating System and version
public.ecr.aws/docker/library/alpine:latest