8000 feat(helm): Add S3 storage support for archives, streams, and logs input. by junhaoliao · Pull Request #1825 · y-scope/clp · GitHub
[go: up one dir, main page]
Skip to content

feat(helm): Add S3 storage support for archives, streams, and logs input. #1825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
junhaoliao wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

feat(helm): Add S3 storage support for archives, streams, and logs input. #1825

junhaoliao wants to merge 2 commits into from

Conversation

@junhaoliao
Copy link
Member
@junhaoliao junhaoliao commented Dec 19, 2025 •
edited
Loading

Warning

This PR has a strong dependency on both #1817 and #1818, and therefore it should be merged AFTER those.

Description

Note

This PR is part of the ongoing work for #1309. More PRs will be submitted until the Helm chart is complete and fully functional.

This PR adds S3/AWS storage support to the CLP Helm chart, enabling archives and stream outputs to be stored in S3 instead of local filesystem storage. The changes include:

New Configuration Options:

  • Add aws_config_directory to values.yaml for specifying the path to AWS config files (e.g., ~/.aws)
  • Support for type: "s3" in archive_output.storage and stream_output.storage
  • Support for type: "s3" in logs_input configuration

New Resources:

  • compression-worker-staged-archives-pv.yaml and compression-worker-staged-archives-pvc.yaml for staging archives before S3 upload
  • query-worker-staged-streams-pv.yaml and query-worker-staged-streams-pvc.yaml for staging streams before S3 upload

Helper Templates:

  • Add clp.awsConfigVolumeMount and clp.awsConfigVolume helpers for mounting AWS configuration

Deployment Changes:

  • Make logsInputVolume mounts conditional on logs_input.type == "fs"
  • Make shared-data-archives mounts conditional on archive_output.storage.type == "fs"
  • Make shared-data-streams mounts conditional on stream_output.storage.type == "fs"
  • Add aws_config_directory volume mounts (conditional) to compression-scheduler, compression-worker, query-worker, garbage-collector, and webui deployments
  • Add AWS credentials environment variables to webui deployment (when using credentials auth type)
  • Add staged-archives mount to compression-worker (when archive_output.storage.type == "s3")
  • Add staged-streams mount to query-worker (when stream_output.storage.type == "s3")

ConfigMap Updates:

  • Add S3 configuration support for archive_output.storage, stream_output.storage, and logs_input
  • Add aws_config_directory field to clp-config
  • Update webui-client-settings.json and webui-server-settings.json to support S3 stream files

Checklist

  • The PR satisfies the contribution guidelines.
  • This is a breaking change and that has been indicated in the PR title, OR this isn't a
    breaking change.
  • Necessary docs have been updated, OR no docs need to be updated.

Validation performed

@coderabbitai
Copy link
Contributor
coderabbitai bot commented Dec 19, 2025

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@junhaoliao
0