- Russia has illegally annexed Crimea in 2014 and brought the war in Donbas followed by full-scale invasion of Ukraine in 2022.
- Russia has brought sorrow and devastations to millions of Ukrainians, killed hundreds of innocent people, damaged thousands of buildings, and forced several million people to flee.
- Putin khuylo!
Glory to Ukraine! 🇺🇦
S3BC stands for S3 Bulk Convert. This is a simple CLI tool, which can update the storage class of files in an AWS S3 bucket or in a compatible storage solution such as Minio
The inspiration for this tool came from a very old project, rather a task, to convert a bunch of buckets to the REDUCED_REDUNDANCY storage type for the cost saving purpose.
Today you can use AWS S3 lifecycle rules to achieve the same goal. Nevertheless, this tool is completely functional and you can use it to change the storage class for the whole bucket or certain files in it.
Please, keep in mind that this is just a fun side-project. Some functionality is still WIP and might not be thoroughly tested. If you have found a bug or just want to suggest a general improvement, feel free to create a new issue or open a pull request.
You can compile an S3BC binary yourself or use it in a Docker container. Releases are still work in progress.
To build the binary locally:
- Ensure that you have Go installed. Since this project uses vendored Go modules, there's no need to download them separately.
- Clone this repository:
git clone https://github.com/grem11n/s3bc.git - Run:
make build
This will compile the binary and put it in the bin/ directory.
Alternatively using Docker:
- Make sure that
dockercommand is available in your system. - Clone this repository:
git clone https://github.com/grem11n/s3bc.git - Run:
make docker-build-dev
This will build an s3bc:dev image.
You can use help to get hints on how to use S3BC:
❯ bin/s3bc help
S3BC or S3 Bulk Convert is a CLI tool to update the storage class of the files in an AWS S3 bucket.
Usage:
s3bc [command]
Available Commands:
completion Generate the autocompletion script for the specified shell
convert Bulk convert objects in an S3 bucket to the given storage class
help Help about any command
validate Check if files in a bucket have desired storage class.
version Print s3bc version.
Flags:
-b, --bucket string Target S3 bucket
-h, --help help for s3bc
-s, --storage-class string Storage class to set (default "STANDARD")
-v, --verbose Verbose output
Use "s3bc [command] --help" for more information about a command.This is the default action. You can use s3bc convert [arguments] or just s3bc [arguments] for this.
To exclude certain file patterns from the conversion, you can use -e or --exclude flag with a regexp pattern for exclusion. Warning! This functionality is not yet tested!
s3bc convert -b <bucket> -s standard -e '.*\.css$'You can use S3BC to check if all the objects in a bucket have a desired storage class. There are two modes of this check:
Normal mode: simply tells you whether objects in a bucket have a desired storage class. If not, exits with 1 exit code.
❯ s3bc validate -b test -s STANDARD
Retrieving bucket objects...
2001 objects found in test bucket
Not all the objects in the "test" bucket have desired storage class
Desired storage class: STANDARD
2001 files in "test" bucket have different storage class.
To get the list of the files, use "--verbose" of "-v" flag.
exit status 1Verbose mode: Outputs the list of files that do not comply in stdout. Be careful, this list may be huge depending on how many files you have in a bucket. To use S3BC in the verbose mode, add --verbose or -v flag.
S3BC is a fun project, in which I tried to get myself familiar with writting CLI applications in Go. I'm using Cobra library (without Viper) as well as a simple plain directory structure.
.
├── ...
├── action # Contains logic for each subcommand
├── build # Contains Dockerfile as well as Docker Compose file for Minio
├── client # AWS-related client code
├── cmd # Cobra commands definitions
├── config # Common config for the app
├── testdata # Dummy data for Minio. Used for local E2E testing
└── version # Code for the `version` subcommand
If you spot a bug or want to contribute to this project, feel free to open a pull request!
This project has CI based on the GitHub Actions. You can find the CI configuration in the .github/workflows directory.
All the tasks are automated with the Makefile. You can use make lint and make test to run linters and unit tests for this app.
You can also run linter and tests in a Docker container, but you need to build a test image first. You can do that with make docker-build-dev and then: make docker-lint && make docker-test.
S3BC is tested using Minio to mimic AWS S3 API locally. Potentially you can also use Localstack for E2E tests. However, Minio keeps the data on the filesystem by default, so you don't have to re-generate test fixtures each time you run E2E tests. It saves some time. The downside is that you need to keep those fixtures in the repository. Minio's data can be found in the testdata/ directory.
You can use Docker Compose to start a Minio server locally:
# Starts Minio server in the background
docker-compose -f build/docker-compose-minio.yaml up -dAfter that Minio API should be available on http://127.0.0.1:9000.
You can provide a custom AWS Endpoint to S3BC using AWS_URL environment variable. For example:
AWS_URL="http://localhost:9000" ./bin/s3bc validate -b test -s STANDARDor even:
AWS_URL="http://localhost:9000" go run . validate -b test -s STANDARDIf you'd like Localstack better or just want to use a different set of the test data, you can easily generate some objects with a simple script like below (mind the --endpoint-url):
# Create a new test bucket
aws --endpoint-url=http://127.0.0.1:9000 s3api create-bucket --bucket=test --region=us-east-1
# Populate this bucket with files
for i in {0..2000}; do uuidgen > ./test-file-${i}.txt && aws --endpoint-url=http://localhost:9000 s3 cp ./test-file-${i}.txt s3://test/ && rm -f ./test-file-${i}.txt ; doneAutomation for E2E tests is currently work in progress.
You can find more information about me and my work at https://grem1.in
Apache 2 Licensed. See LICENSE for details.