ENH: Pass optimizations arguments to asv build #17284

seiko2plus · 2020-09-10T13:40:09Z

ENH: Pass optimizations arguments to asv build #17284

This patch allows passing -j, --cpu-baseline, --cpu-dispatch and --disable-optimization
to ASV build when argument --bench-compare is used.

TODO:

local test

seiko2plus · 2020-09-10T16:04:29Z

@seberg, I just don't understand why ASV doesn't provide a direct way to specify custom build options? Am I missing something?

seberg · 2020-09-10T17:01:43Z

I looked around a bit, and don't see a way other than as part of the config. However, we could use --config to use a custom config file rather than deleting the current config? May want to ping Pauli, though, at least if that doesn't see right.

Qiyu8 · 2020-09-11T02:50:21Z

I think this is a good solution to benchmark flexible universal intrinsic usage. but asv community still needs to considering the necessity of adding {build_args} to asv.config.json. @pv what do you think?

seiko2plus · 2020-09-11T13:42:26Z

@seberg,

I looked around a bit, and don't see a way other than as part of the config.

I confirm it, thank you!

we could use --config to use a custom config file rather than deleting the current config?

I followed your suggestion, I kept the main config file as-is so it can be used with ASV command.

@Qiyu8,

I think this is a good solution to benchmark flexible universal intrinsic usage.

An effective workaround but we still have other obstacles such as providing auto-generated test data and
provides the ability to check the sanity of it, similar to what OpenCV do, another nightmare I guess.
It would be easier for me to drop ASV and create our own benchmark tool, but I guess the community wouldn't easily
accept such a move like that.

runtests.py

seberg · 2020-09-11T14:02:06Z

Maybe its right and we should consider upstreaming things to ASV. I am a bit worried that running plain ASV won't have access to this, but there may not be a good way to do that.
Does this give the baseline/optimized/disabled versions different environment names in the ASV run?

seiko2plus · 2020-09-11T14:32:54Z

@seberg,

Does this give the baseline/optimized/disabled versions different environment names in the ASV run?

No, it generates a config file with the required build options then pass it through
--config, it also clears ~~whole cache~~ the involved commits cache if the user changed the build options.

runtests.py

seiko2plus · 2020-09-11T15:15:40Z

@seberg,

I am a bit worried that running plain ASV won't have access to this, but there may not be a good way to do that.

I didn't touch the original config file, I created a new asv config instead asv_compare.conf.json and it only works through --bench-compare.

benchmarks/asv_compare.conf.json

runtests.py

This patch allows passing `-j`, `--cpu-baseline`, `--cpu-dispatch` and `--disable-optimization` to ASV build when argument `--bench-compare` is used.

seiko2plus

Looks good to me, what do you think @pv?

charris · 2020-09-27T23:37:23Z

This needs documentation so people will know how to use it. Maybe in the README.rst or runtests.py files?

charris · 2020-10-10T20:27:57Z

Thanks Sayed. I put this in to aid the ongoing SIMD work, but more usage documentation would be helpful.

seiko2plus mentioned this pull request Sep 10, 2020

ENH: add neon intrinsics to fft radix2&4 #17231

Closed

charris added 01 - Enhancement component: benchmarks labels Sep 10, 2020

seiko2plus force-pushed the pass_opt2asv_run branch 4 times, most recently from 4fcf2bb to 6edbba2 Compare September 11, 2020 13:31