Description
It is common in enterprise environments that simple open-source tools and workflows fall a little short on meeting a few common requirements.
Typically, this includes (relevant to this tool):
- Never pulling anything directly from internet during CI workflows
- Any open-source GIT repos get forked to github enterprise or gitlab etc
- All package manager access get proxied and cached through JFrog Artifactory, Sonatype Nexus, Gitlab, etc.
- Never trust any public source download without explicit/manual checksum declaration and verification
This can cause pain for build engineers who need to use a wide range of tools (like this one). Many old tools are simply deemed unusable, or sometimes get forked and sources modified to be used in these environments. Either case is a bummer.
Many modern tools are in a better position to support these things first class, and many do. In this case, there are a few trivial things which could be done if you are interested in supporting such cases:
- Support overriding the URL's for all downloads
- Support providing a SHA which gets validated right after download
- In both cases supporting these values in all of Config, CLI Argument, and EnvVar would be ideal
Here is a pretty robust and mature example of how you could do it, taken from the Conan package manager:
https://github.com/conan-io/conan-center-index/blob/master/recipes/cpython/all/conandata.yml
Of note, conandata.yml only has deal with URL's for one "package" at a time because each package gets it's own conandata.yml. For this tool, you download (and thus need to describe) URL's and shas for multiple "packages", so perhaps it makes sense for your data structure to be different.
You could still choose to support the monolithic single-file-yaml data structure, keyed entirely based on the python version (like in the Conan example). Or, you could support a simpler structure like this, and force the user to use different configs per-version like this:
configs/portable_python_3_9_12.yml
source_urls:
cpython:
url: https://my_local_server/Python3_9_12.tar.gz
sha: "...."
zlib:
url: https://my_local_server/ZLib1_2_12.tar.gz
sha: "...."
configs/portable_python_3_10_4.yml
source_urls:
cpython:
url: https://my_local_server/cpython-3_10_4.tar.gz
sha: "...."
zlib:
url: https://my_local_server/zlib-1_2_12.tar.gz
sha: "...."
I'm not sure which is better but I'd be extremely happy with either.
The good news is that either case seems fairly trivial to support, because you already have robust handling in place for the config file and it looks super easy to add overriding of the URL's thanks to your clean code.
Please let me know what you think about the prospect of this?