8000 implement S3 MD5 checksum check for UploadPart and improve logic by bentsku · Pull Request #11795 · localstack/localstack · GitHub
[go: up one dir, main page]
Skip to content

implement S3 MD5 checksum check for UploadPart and improve logic #11795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Nov 7, 2024
Merged

implement S3 MD5 checksum check for UploadPart and improve logic #11795

merged 4 commits into from
Nov 7, 2024

Conversation

bentsku
Copy link
Contributor
@bentsku bentsku commented Nov 6, 2024 •
edited
Loading

Motivation

As reported and re-opened in #2562, we've had a regression compared to 0.11 (yes!), as some behavior was untested and not caught during early migrations to ASF (S3 v2) and now v3.

We did not make a difference between InvalidDigest and BadDigest (Invalid is raised when the given b64 encoded checksum is not a proper b64 encoded checksum, and Bad when the checksum does not match the item).

Part of the MD5 behavior was re-added with #9064.

Changes

  • update the logic when comparing the ContentMD5
  • check that the ContentMD5 is a proper b64-encoded MD5 hash
  • add test for UploadPart MD5 behavior: funny enough, error messages are not the same between PutObject and UploadPart (not using the same values)

fixes #2562

@bentsku bentsku added aws:s3 Amazon Simple Storage Service semver: patch Non-breaking changes which can be included in patch releases labels Nov 6, 2024
@bentsku bentsku added this to the 4.0 milestone Nov 6, 2024
@bentsku bentsku requested a review from cloutierMat November 6, 2024 18:48
@bentsku bentsku self-assigned this Nov 6, 2024
@bentsku bentsku requested a review from thrau as a code owner November 6, 2024 18:48
@github-actions GitHub Actions
Copy link
github-actions bot commented Nov 6, 2024

S3 Image Test Results (AMD64 / ARM64)

  2 files    2 suites   4m 13s ⏱️
423 tests 369 ✅  54 💤 0 ❌
846 runs  738 ✅ 108 💤 0 ❌

Results for commit 37153fc.

@github-actions GitHub Actions
Copy link
github-actions bot commented Nov 6, 2024

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 42m 23s ⏱️ +18s
3 534 tests ±0  3 119 ✅ ±0  415 💤 ±0  0 ❌ ±0 
3 536 runs  ±0  3 119 ✅ ±0  417 💤 ±0  0 ❌ ±0 

Results for commit 37153fc. ± Comparison against base commit 359f45f.

cloutierMat
cloutierMat approved these changes Nov 7, 2024
View reviewed changes
Copy link
Contributor
@cloutierMat cloutierMat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! That is an impressive level of parity with aws now! 🚀

Nice digging to get through all of those testing use case

@bentsku bentsku merged commit d2d179a into master Nov 7, 2024
48 checks passed
@bentsku bentsku deleted the s3-checksum-upload-part branch November 7, 2024 10:19
zaingz pushed a commit that referenced this pull request Nov 11, 2024
@bentsku @zaingz
implement S3 MD5 checksum check for UploadPart and improve logic (#11795
720f702 
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cloutierMat cloutierMat cloutierMat approved these changes

@thrau thrau Awaiting requested review from thrau thrau is a code owner

Assignees

@bentsku bentsku

Labels
aws:s3 Amazon Simple Storage Service semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Milestone
4.0
Development

Successfully merging this pull request may close these issues.

Simulate S3 BadDigest and InvalidDigest errors
2 participants
@bentsku @cloutierMat
0