Description
Hi!
I continue my quest to enable BenchmarkDotNet to be used as an toolkit for performance unit tests.
For this time, the biggest deal: performance.
As it is for now, it takes 30-80 seconds to run relatively simple benchmark. Also, the runner produces 50-100 mb of temp files. This is definitely a gamestopper.
A typical perftests project can contain hundreds of perftests and waiting for a hour or two just to run them... well, this is bad:)
Thing are getting much worse if the project is built using third-party CI service (such as appveyor.com).
Typical time limit on these rarely exceeds 30-40 minutes.
Now, the proposal: add an ability to run benchmarks in-process.
There are no time spent to build the code, no temporary binaries. The speedup should be significant.
In fact, it is. I've wrote fast-and-dirty proofcode... and it works:)
DISCLAIMER: the code is just a prototype, not a final solution. It definitely not complete, but it's good enough to proof that the idea works at all.
So, the results. Code from the develop branch, release build, an benchmark class with 20 benchmark methods.
- Classic toolchain: 35 seconds.
- Inline run toolchain: 1.2 seconds.
In both cases measurements are pretty close, difference is not bigger than between two consecutive runs under classic toolchain.
Now, the question: are you interested in it at all and does this proposal fit into the roadmap of the BenchmarkDotNet?
If yes, I'll need your guide to rewrite the current throw-it-out prototype into something more usable:)