Speed up Travis #440
Speed up Travis #440
Conversation
There was a problem hiding this comment.
The parallel stuff is great!
But I would really rather not add a 62MB file to the repo, and have each and every clone of the repo now be forced to download this. Right now the whole download is only about 40MB. Every time we update the toolchain, it will be another 60MB unless you wipe out the history.
If the ppa fetching is slow (maybe it's throttled), could you keep the deb on AWS or the Adafruit CDN or somewhere like that?
14af315 to
71451bd
Compare
|
The qemu and unix sub-jobs failed. Does the elapsed time per sub-job look better or worse than with the .deb in the repo? |
|
restarting .10 and .11. The AWS fetches failed (?). |
|
Well, the fetch is plenty fast: 2.7s ! |
This does two major things: 1. It stores the arm gcc in the repo. The download from the PPA was taking between 2 and 7 minutes. 2. It splits each board into a separate job so they are built in parallel.
71451bd to
b1998cb
Compare
|
What about building a docker image that's already setup with the PPAs and ARM GCC, etc. image for travis to use? More details here: https://docs.travis-ci.com/user/docker/ Build the docker image ahead of time and it can be populated and published to docker's universe with all the necessary build tools installed and no time waiting for Travis' servers to set it up. It could be faster and also has no cost from AWS (it's not huge but quick back of envelope is 60mb * 10 pulls a day * 20 days a month = 12GB of transfer, about $1 in bandwidth and storage costs). Also I'd double check travis CI config won't start jobs unless something very important happens like a pull is accepted from a core contributor--if it's triggered by opening issues or something anyone could do it could be an 'attack' vector that raises costs. |
|
@tdicola I glanced at that Docker doc but don't have a lot of experience with Docker. I can't imagine it starts up faster than the Ubuntu VM. I think the few dollar bandwidth cost per month is worth paying to have continuous testing for everyone. Having it run early on pull requests reduces our load of checking that it works later on. Also, if it became exorbitant we could use Travis' persist cache to persist it locally. The ARM GCC PPA is also being requested as a whitelisted source for Travis here which would make it usable in container based builds. For now, I think this is a good first step and we can evolve it later as needed. |
|
Yeah totally agree it's worth having fast builds for good continuous integration tests. A docker or other pre-built image is handy because it can be used by anyone, i.e. both the CI system and users who want to build circuitpython. It becomes the canonical way to install and build circuitpython regardless of platform. In my testing docker is super fast to startup and definitely worth benchmarking vs. an ubuntu VM that's installing everything from scratch. Putting apt packages on AWS is a smart idea and easy step for now but maybe worth keeping this issue open to explore more options for speeding up the build and making the build 'portable'? |
|
This is a pull request, not an issue.
How about making another issue for adding ESP8266 build support to the CI
and mentioning that Docker may be a good way to do it?
…On Thu, Nov 16, 2017 at 10:00 AM Tony DiCola ***@***.***> wrote:
Yeah totally agree it's worth having fast builds for good continuous
integration tests. A docker or other pre-built image is handy because it
can be used by anyone, i.e. both the CI system and users who want to build
circuitpython. It becomes the canonical way to install and build
circuitpython regardless of platform. In my testing docker is super fast to
startup and definitely worth benchmarking vs. an ubuntu VM that's
installing everything from scratch. Putting apt packages on AWS is a smart
idea easy step for now but maybe worth keeping this issue open to explore
more options for speeding up the build and making the build 'portable'?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#440 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AADNqdI4wvHAcGG9iqtanaKAylTV2zgeks5s3HhFgaJpZM4Qf6f5>
.
|
This does two major things: