8000 Standard retryer incorrectly counts non-retryable errors against retries · Issue #3186 · aws/aws-sdk-go-v2 · GitHub
[go: up one dir, main page]
Skip to content

Standard retryer incorrectly counts non-retryable errors against retries #3186

New issue
New issue
Closed
Closed
Standard retryer incorrectly counts non-retryable errors against retries#3186
Labels
response-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 7 days.
@PleasingFungus

Description

@PleasingFungus
PleasingFungus
opened on Sep 9, 2025

Acknowledgements

Describe the bug

The standard retryer (retry/standard.go) claims retry token(s) every time a retry attempt is made. These token(s) are released whenever a request completes successfully.

The intent of this system is to limit load on AWS systems. When an AWS system is shedding load - as demonstrated by throttling exceptions, internal server errors, etc - we don't want clients to add more load via retries.

However, the retryer's behavior is incorrect for non-retryable errors, such as "Not Found". It does not release tokens for attempts which end in such an error. Since such errors do not indicate the need to reduce retries, this is incorrect.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

When an AWS request is retried and then gets a non-retryable error, it should release retry token(s).

Current Behavior

When an AWS request is retried and then gets a non-retryable error, retry token(s) are not released.

Reproduction Steps

	cfg, err := config.LoadDefaultConfig(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	s3c := s3.NewFromConfig(cfg)
	for range runtime.GOMAXPROCS(0) {
		go func() {
			_, err := s3c.HeadObject(context.Background(), &s3.HeadObjectInput{
				Bucket: aws.String("some-bucket"),
				Key:    aws.String("does-not-exist"),
			})
			var e *awshttp.ResponseError
			if !errors.As(err, &e) || e.HTTPStatusCode() != 404 {
				// This will eventually hit a "failed to get rate limit token:
				// retry quota exceeded..." error, even at a low retry rate.
				log.Fatal(err)
			}
		}()
	}

Possible Solution

At the bottom of retry/standard.go, perhaps something like this (untested):

...
	fn, err := s.options.RateLimiter.GetToken(ctx, cost)
	if err != nil {
		return nil, fmt.Errorf("failed to get rate limit token, %w", err)
	}

	rt := &releaseToken{
		isErrorRetryable: s.IsErrorRetryable,
		f:                fn,
	}
	return rt.release, nil
}

func nopRelease(error) error { return nil }

type releaseToken struct {
	isErrorRetryable func(error) bool
	f                func() error
}

func (rt *releaseToken) release(err error) error {
	if err != nil && rt.isErrorRetryable(err) {
		return nil
	}
	return rt.f()
}

Additional Information/Context

We ran into this in production while running a system which had an extremely high rate of 404s while retrieving objects from S3.

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2@v1.39.0 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2@v1.39.0 go@1.22
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.7.1 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.7.1 go@1.22
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2/credentials@v1.18.11
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.18.7
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.3
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2/service/sso@v1.29.2
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.34.3
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2/service/sts@v1.38.3
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1
github.com/aws/aws-sdk-go-v2/config@v1.31.7 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.13.7
github.com/aws/aws-sdk-go-v2/config@v1.31.7 go@1.22
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.18.7
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 github.com/aws/aws-sdk-go-v2/service/sso@v1.29.2
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.34.3
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 github.com/aws/aws-sdk-go-v2/service/sts@v1.38.3
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.13.7
github.com/aws/aws-sdk-go-v2/credentials@v1.18.11 go@1.22
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.18.7 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.18.7 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.18.7 go@1.22
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/config@v1.31.7
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/credentials@v1.18.11
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.7.1
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.18.7
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.3
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/internal/v4a@v1.4.7
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.8.7
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.13.7
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.19.7
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/service/sso@v1.29.2
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.34.3
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 github.com/aws/aws-sdk-go-v2/service/sts@v1.38.3
github.com/aws/aws-sdk-go-v2/feature/s3/manager@v1.19.5 go@1.22
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7 go@1.22
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/internal
7B63
/endpoints/v2@v2.7.7 go@1.22
github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.3 go@1.22
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.4.7 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.4.7 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.4.7 go@1.22
github.com/aws/aws-sdk-go-v2/service/cloudfront@v1.54.1 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/cloudfront@v1.54.1 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/cloudfront@v1.54.1 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/cloudfront@v1.54.1 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/cloudfront@v1.54.1 go@1.22
github.com/aws/aws-sdk-go-v2/service/ec2@v1.251.1 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/ec2@v1.251.1 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/ec2@v1.251.1 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/ec2@v1.251.1 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1
github.com/aws/aws-sdk-go-v2/service/ec2@v1.251.1 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.13.7
github.com/aws/aws-sdk-go-v2/service/ec2@v1.251.1 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/ec2@v1.251.1 go@1.22
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2@v1.50.3 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2@v1.50.3 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2@v1.50.3 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2@v1.50.3 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2@v1.50.3 go@1.22
github.com/aws/aws-sdk-go-v2/service/iam@v1.47.4 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/iam@v1.47.4 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/iam@v1.47.4 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/iam@v1.47.4 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/iam@v1.47.4 go@1.22
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1 go@1.22
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.8.7 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.8.7 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.13.7
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.8.7 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.8.7 go@1.22
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.13.7 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.13.7 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.13.7 go@1.22
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.19.7 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.19.7 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.19.7 go@1.22
github.com/aws/aws-sdk-go-v2/service/route53@v1.58.1 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/route53@v1.58.1 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/route53@v1.58.1 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/route53@v1.58.1 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/route53@v1.58.1 go@1.22
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.7.1
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 github.com/aws/aws-sdk-go-v2/internal/v4a@v1.4.7
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.8.7
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.13.7
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.19.7
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/s3@v1.88.0 go@1.22
github.com/aws/aws-sdk-go-v2/service/secretsmanager@v1.39.3 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/secretsmanager@v1.39.3 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/secretsmanager@v1.39.3 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/secretsmanager@v1.39.3 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/secretsmanager@v1.39.3 go@1.22
github.com/aws/aws-sdk-go-v2/service/ses@v1.34.2 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/ses@v1.34.2 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/ses@v1.34.2 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/ses@v1.34.2 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/ses@v1.34.2 go@1.22
github.com/aws/aws-sdk-go-v2/service/sso@v1.29.2 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/sso@v1.29.2 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/sso@v1.29.2 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/sso@v1.29.2 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/sso@v1.29.2 go@1.22
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.34.3 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.34.3 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.34.3 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.34.3 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.34.3 go@1.22
github.com/aws/aws-sdk-go-v2/service/sts@v1.38.3 github.com/aws/aws-sdk-go-v2@v1.39.0
github.com/aws/aws-sdk-go-v2/service/sts@v1.38.3 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.4.7
github.com/aws/aws-sdk-go-v2/service/sts@v1.38.3 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.7.7
github.com/aws/aws-sdk-go-v2/service/sts@v1.38.3 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1
github.com/aws/aws-sdk-go-v2/service/sts@v1.38.3 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.13.7
github.com/aws/aws-sdk-go-v2/service/sts@v1.38.3 github.com/aws/smithy-go@v1.23.0
github.com/aws/aws-sdk-go-v2/service/sts@v1.38.3 go@1.22
github.com/aws/smithy-go@v1.23.0 go@1.22

Compiler and Version used

go version go1.25.0 darwin/arm64

Operating System and version

Ubuntu 24.04

Metadata

Metadata

Assignees

No one assigned

    Labels

    response-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 7 days.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
      0