After playing around with asv a bit more, I'd like to get feedback on an implementation plan. Some quick background for people who haven't used asv before: it's sort of like pytest, you create a folder of code snippets you want to benchmark and invoke asv on them to generate a set of timings. Those timings are associated with the machine and the pvlib git commit used to run the benchmarks. When new code is committed, the benchmarks are rerun, creating a history of timings. This history is then used to generate an HTML report like this example from astropy that shows the timing history for each benchmark. It also has some other nice features like regression detection logic and a binary search to determine which commit introduced the regression.

A goal that seems useful and achievable to me is to have a machine somewhere that does a nightly benchmark test on whatever commits were added to master that day. Running the benchmarks manually and inspecting the results is a bit of a hassle, so I want it to be automated and generating results consistently over time so that they are available whenever they're needed.

Choice of machine: In terms of timing stability, a dedicated machine is ideal. However, a virtual cloud server is more convenient. I've been running a basic pvlib benchmark suite on a cheapo cloud VPS and getting surprisingly stable results -- not as stable as the astropy timings, but good enough to detect significant shifts. Could be that I'm just getting lucky and running during stable conditions.

Tracking timing history: the timing history is stored in json files, so it is reasonable to use git to keep track of them over time. asv also has a convenient command to push the HTML report to the gh-pages branch for hosting on github pages. However, keeping track of the results in the main pvlib repo would add a lot of clutter to master's commit history. NumPy and Astropy, and probably others, keep the results in a separate respository, which seems like a good idea to me. That way a cron job could push to it as much as it wants without making a mess of the main repo's commit history.

Long term it might be cool to somehow integrate benchmarks with a CI PR check, but I think it makes sense to take smaller steps and just benchmark master for now to get more familiar with asv and figure out how to get the most value from it. So to summarize, here's what I'm proposing:

Create a new benchmarks folder (at the top level?) in this repo for the benchmark suite, it would look something like this: https://github.com/kanderso-nrel/pvlib-python/tree/asv_setup/asv_bench/benchmarks
Create a new pvlib-benchmarks repository in the pvlib organization to store benchmark results and the HTML plotting report (like the astropy example). The HTML would be on the gh-pages branch so that it would be available at http://pvlib.github.io/pvlib-benchmarks or something like that.
Set up one or more benchmark machines with a nightly cron job that pulls master, runs the suite on any new commits, and pushes results and the HTML report to pvlib-benchmarks so that the Github Pages website is kept up to date and shows the entire history. @wholmgren mentioned a dedicated machine may be available, but either way I volunteer to maintain a cheapo VPS as a point for comparison.

Does this sound like a good plan?

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions