ESBench: A modern benchmarking tool

Kaciras — Tue, 24 Sep 2024 14:46:02 +0000

benchmark.js ends in April 2024, it's time for a new generation of tools to emerge!

ESBench is a new JavaScript benchmarking tool released in 2024. It is designed to provide simple, scalable benchmarking support for modern JS projects.

GitHub repository

The Problem

In the beginning JavaScript was run directly from source, but as applications have gotten more complex this has become less and less the case. Modern applications often need to be built, which requires benchmarking tools to be able to integrate the build process, and the impact of the build on performance needs to be taken into account.
JavaScript doesn't have an “official runtime”, browsers, Node, Deno, and more recently, Bun, all claim high performance. But what about in your own code? It would be cool if there was a tool that could run benchmarks in multiple runtimes and export the results to one report —— that was my primary motivation to create ESBench.
There are some useful features that I don't see JS benchmark tools being able to do, such as calculating asymptotic complexity, validate the return value, and plot the results into an interactive chart.

Benchmark with ESBench

To solve these problems, I decided to create a new tool that contained all the features I needed and had a simple API.

After about a year of development, ESBench was born.

You can try ESBench online

Write Suite

ESBench opts for declarative APIs and closures, which is the most popular way JS is written.

Compare Set.has and Array.includes:

// benchmark/array-vs-set.js
export default scene => {
    const length = 1000;
    const array = Array.from({ length }, (_, i) => i);

    const set = new Set(array);
    const value = array[Math.floor(array.length / 2)];

    scene.bench("Set", () => set.has(value));
    scene.bench("Array", () => array.includes(value));
};

Run pnpm exec esbench to execute the suite, the result:

Suite: benchmark/array-vs-set.js
| No. |  Name |      time | time.SD |
| --: | ----: | --------: | ------: |
|   0 |   Set |   3.64 ns | 0.00 ns |
|   1 | Array | 326.36 ns | 0.17 ns |

A Little More Features

Parameterization and baselines are frequent requirements, ESBench supports them with simple options.

export default {
    baseline: { type: "type", value: Set },
    params: {
        length: [10, 10_000],
        type: [Set, Array],
    },
    setup(scene) {
        const { length, type } = scene.params;

        // Prepare
        const array = Array.from({ length }, (_, i) => i);
        const set = new Set(array);
        const value = array[Math.floor(array.length / 2)];

        // Support conditions
        if (type === Set) {
            // Define benchmark cases
            scene.bench("create", () => new Set(array));
            scene.bench("has", () => set.has(value));
        } else {
            scene.bench("create", () => [...array]);
            scene.bench("has", () => array.includes(value));
        }
    },
};

The text report:

Cross Runtime

Back to the problem above, running across runtimes:

// esbench.config.js
import { defineConfig, ProcessExecutor, ViteBuilder, WebRemoteExecutor } from "esbench/host";

export default defineConfig({
    toolchains: [{
        // Build your benchmark code with Vite, require vite installed.
        builders: [new ViteBuilder()],
        executors: [
            // Run suite on Node.
            new ProcessExecutor("node"),

            // Run suite on Bun.
            new ProcessExecutor("bun"),

            // Open the default browser to run benchmark,
            // in my computer it's Firefox.
            {
                name: "Firefox",
                use: new WebRemoteExecutor({ open: {} }),
            },
        ],
    }],
});

You can also set a runtime as baseline:

import { defineSuite } from "esbench";

export default defineSuite({
    baseline: { type: "Executor", value: "node" },
    setup(scene) {
        const length = 1000;
        const array = Array.from({ length }, (_, i) => i);

        const set = new Set(array);
        const value = array[Math.floor(array.length / 2)];

        scene.bench("Set", () => set.has(value));
        scene.bench("Array", () => array.includes(value));
    },
});

The result:

| No. |  Name | Executor |      time | time.SD | time.ratio |
| --: | ----: | -------: | --------: | ------: | ---------: |
|   0 |   Set |     node |   3.69 ns | 0.03 ns |   baseline |
|   1 |   Set |      bun |   0.00 ns | 0.00 ns |   -100.00% |
|   2 |   Set |  Firefox |   0.00 ns | 0.00 ns |   -100.00% |
|     |       |          |           |         |            |
|   3 | Array |     node | 325.02 ns | 1.00 ns |   baseline |
|   4 | Array |      bun | 324.87 ns | 0.08 ns |     -0.04% |
|   5 | Array |  Firefox | 516.70 ns | 0.75 ns |    +58.98% |
Warnings:
[No.1] Set: The function duration is indistinguishable from the empty function duration.
[No.2] Set: The function duration is indistinguishable from the empty function duration.

More Usage Cases

ESBench can do much more than basic usage:

Conclusion

If you're tired of writing benchmarks in JavaScript, ESBench is the library you’ve been waiting for.

DEV Community: Kaciras