TSTL Benchmarks

These benchmarks are written in typescript and transpiled to lua by using tstl.

Memory benchmarks

To add a new benchmark add a new file to memory_benchmarks and default export a function with the following type: () => void. To prevent the benchmark from reporting "useful" results of your benchmark function as garbage, simply return the result. The memory used by the returned result won't count towards the total garbage amount.

For example (memory_benchmarks/my_benchmark.ts):

export default function myBenchmark() {
  const n = 123;
  const result = [];
  for (let i = 0; i < n; i++) {
    // Do something memory instensive
  }
  return result; // Return results so they wont be counted as garbage
}

Goal

The goal of memory benchmarks is to track how much (memory) "garbage" is created by tstl. For that reason garbage collection is disabled in the benchmarks.

You can force the creation of "garbage" by creating a lot of anonymous functions or temporary tables (see lua-users.org for more information).

To avoid crashes in the CI your benchmark should not use more than 500MB of memory.

Runtime benchmarks

To add a new runtime benchmark (execution time), add a new file to runtime_benchmarks and default export a function with the following type: () => void.

Running locally

Create a benchmark baseline called "benchmark_baseline.json": tstl -p tsconfig.53.json && cd dist && lua -- run.lua benchmark_baseline.json
Make some changes to tstl.
Create an updated benchmark and compare with the baseline: tstl -p tsconfig.53.json && cd dist && lua -- run.lua benchmark_updated.json benchmark_baseline.json
The above command will output comparison data as json to stdout. If you provide a path as third argument the comparison data will be written to that path instead. tstl -p tsconfig.53.json && cd dist && lua -- run.lua benchmark_updated.json benchmark_baseline.json result.md

Uh oh!