Deploy py-compiled build #3701
Deploy py-compiled build #3701
Conversation
for more information, see https://pre-commit.ci
Maybe this should be a separate pr? |
|
Well most of this PR is re-writing that deployment script. Once we have it, doing the pyc deployment is just a couple of lines in the CI setup since most of the work for it is done in #3700 |
|
OK the diff should be much smaller now that the other PR is merged. Though we would need to fix |
| find dist/ -type f -print0 | xargs -0 -n1 -I@ bash -c "echo \"Compressing @\"; gzip @; mv @.gz @;" | ||
| aws s3 sync dist/ "s3://pyodide-cdn2.iodide.io/v${CIRCLE_TAG}/full/" --exclude '*.zip' --exclude '*.whl' --exclude "*.tar" --cache-control 'max-age=30758400, immutable, public' --content-encoding 'gzip' # 1 year cache | ||
| aws s3 sync dist/ "s3://pyodide-cdn2.iodide.io/v${CIRCLE_TAG}/full/" --exclude '*' --include '*.zip' --include "*.whl" --include "*.tar" --cache-control 'max-age=30758400, immutable, public' --content-type 'application/wasm' --content-encoding 'gzip' # 1 year | ||
| python3 tools/deploy_s3.py dist/ "v${CIRCLE_TAG}/full/" --bucket "pyodide-cdn2.iodide.io" --cache-control 'max-age=30758400, immutable, public' |
There was a problem hiding this comment.
Good to see this deployment logic factored out and tested. In the future I think it would be good to move this whole recipe into a Python script that we can test (other than install requirements of course).
tools/deploy_s3.py
Outdated
| typer.echo(" - content-encoding: gzip") | ||
|
|
||
| if rm_remote_prefix: | ||
| _rm_s3_prefix(bucket, str(remote_prefix).lstrip("/")) |
There was a problem hiding this comment.
Maybe we want to validate the prefix in some way to reduce chance of accidentally deleting everything?
There was a problem hiding this comment.
Shouldn't pretend also apply to deleting stuff?
There was a problem hiding this comment.
Shouldn't pretend also apply to deleting stuff?
Yes, good that someone is reviewing this code ^^ Fixed.
Maybe we want to validate the prefix in some way to reduce chance of accidentally deleting everything?
Indeed. Let's do more validation of it and only allow deletion of prefixes starting with "/dev/".
| elif file_path.suffix == ".ts": | ||
| # This will not be correctly detected by mimetypes. | ||
| # However, JsDelivr will currently not serve .ts file in the | ||
| # custom CDN configuration, so it does not really matter. | ||
| content_type = "text/x.typescript" |
There was a problem hiding this comment.
Well the MIME type for .ts and .d.ts files is a bit unclear but it doesn't really matter, since both are currently not allowed to be served via JsDelivr
https://cdn.jsdelivr.net/pyodide/v0.22.1/full/pyodide.ts (returns 403 forbidden)
So people get it from npm anyway.
[skip ci] Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
|
The |
This enables deployment of a py-compiled build under,
Here both the packages and the stdlib would be py-compiled.
With the stdlib being unvendored (thanks @ryanking13 ) it's actually relatively straightforward to do this as a post-processing step after the build.
This is built on top of #3700 so that PR should be merged first.
Closes #3269 Closes #3269
This also replaces the previous
aws clicommands with a Python script, so that CI setup is less repetitive, and we can validate things better.TODO:
cibuilds/githubimage we use is no longer maintainer.