DEV Community

Cover image for PHPBench: Microbenchmarking PHP Without Fooling Yourself
Gabriel Anhaia
Gabriel Anhaia

Posted on

PHPBench: Microbenchmarking PHP Without Fooling Yourself

#php #phpbench #performance #benchmarking

You rewrite a hot function. You wrap the old version and the new version in microtime(true) calls, loop each one a thousand times, print the difference. The new one is 3x faster. You ship it. Request latency in production doesn't move a millisecond.

The benchmark wasn't wrong about the number it printed. It was wrong about what that number meant. A single microtime loop measures your function plus timer overhead plus whatever the CPU was doing that second plus the state of the opcache on the first call. You read one of those signals and attributed it to your code.

PHPBench exists to stop that. Its knobs (revs, iterations, warmup, retry threshold) are not ceremony. Each one removes a specific way a microbenchmark lies.

Install and the smallest real benchmark 

composer require --dev phpbench/phpbench
Enter fullscreen mode Exit fullscreen mode

A benchmark is a class in benchmarks/, and subjects are methods prefixed with bench:

<?php

use PhpBench\Attributes as Bench;

final class SerializerBench
{
    private array $payload;

    #[Bench\BeforeMethods('setUp')]
    public function setUp(): void
    {
        $this->payload = range(1, 500);
    }

    #[Bench\Revs(1000)]
    #[Bench\Iterations(5)]
    #[Bench\Warmup(2)]
    public function benchJsonEncode(): void
    {
        json_encode($this->payload);
    }
}
Enter fullscreen mode Exit fullscreen mode

Point PHPBench at the folder with a phpbench.json:

{
    "runner.bootstrap": "vendor/autoload.php",
    "runner.path": "benchmarks"
}
Enter fullscreen mode Exit fullscreen mode

Run it:

vendor/bin/phpbench run --report=default
Enter fullscreen mode Exit fullscreen mode

Three attributes carry the whole method: Revs, Iterations, Warmup. Get those wrong and every number below them is fiction. So start there.

Revs and iterations are different knobs

People conflate these because both are "how many times you run the thing." They defend against different problems.

A rev (revolution) is one call to your subject inside a single timed measurement. PHPBench starts the timer once, calls the subject revs times, stops the timer, then divides by revs. That division is the point. A single json_encode on a small array can run faster than your system timer's resolution. Time one call and you measure the clock, not the code. Time a thousand calls and divide, and you get per-call resolution well below what the timer alone can see.

An iteration is one full measurement, repeated. Five iterations give you five independent samples of that per-call time. You need the sample because one measurement can't tell you whether the number is stable. Five can. PHPBench reports the spread as a deviation, and the deviation is the honesty check.

The --report=default table has a column per measurement, but the two fields that matter reduce to a shorthand like this (numbers are illustrative):

benchJsonEncode  mean 12.480μs  rstdev ±1.90%
Enter fullscreen mode Exit fullscreen mode

The mean is the time per call. The ±1.90% is the deviation across iterations. That second number is the one that tells you whether to trust the first. If deviation is 40%, your mean is noise wearing a lab coat. More revs shrink timer error. More iterations expose run-to-run instability. You want both.

Warmup: the first call is a liar

The first time PHP executes a code path, work happens that never happens again in the same process. The opcache compiles the file. With the JIT enabled on PHP 8.4, hot paths get traced and compiled to machine code after they cross a threshold. CPU instruction and data caches fill. None of that is your function's steady-state cost, but all of it lands on the first few calls.

Warmup(2) runs two revs before the timer starts, so measurement begins after the cold-start tax is already paid:

#[Bench\Revs(1000)]
#[Bench\Iterations(5)]
#[Bench\Warmup(3)]
public function benchRouteMatch(): void
{
    $this->router->match('/orders/42/items');
}
Enter fullscreen mode Exit fullscreen mode

If your subject touches the JIT, warmup is the difference between measuring compiled code and measuring the compiler. Skip it and your fast function looks slow, because iteration one paid for compilation and dragged the mean up. Turn warmup on and watch the deviation drop. That drop is the cold-start noise leaving the sample.

Warmup does not help when the "warm" and "cold" states are genuinely different work, which brings up the trap below.

The cache trap: measuring the wrong second run

Warmup and revs both assume every call does the same work. When your subject memoizes, that assumption breaks and the benchmark quietly measures nothing.

final class ConfigBench
{
    private ConfigLoader $loader;

    #[Bench\BeforeMethods('setUp')]
    public function setUp(): void
    {
        $this->loader = new ConfigLoader('config/app.php');
    }

    #[Bench\Revs(1000)]
    #[Bench\Iterations(5)]
    public function benchLoad(): void
    {
        // load() caches after the first call.
        $this->loader->load();
    }
}
Enter fullscreen mode Exit fullscreen mode

Rev one parses the file. Revs two through a thousand return a cached array. PHPBench divides total time by 1000, so 999 near-free calls bury the one that did work. The report says the loader is blazing. It says nothing about loading.

If you mean to measure the cold path, build fresh state per rev in a Before method and use Revs(1) with more iterations. If you mean to measure the warm path, say so out loud in the method name. What you cannot do is leave it ambiguous and read the number as "load performance."

Retry threshold: fighting the noisy machine

Your laptop is a hostile measurement environment. Turbo boost changes clock speed mid-run. Thermal throttling kicks in after a warm minute. A background Spotlight index steals a core. Any of these inflates one iteration and skews the mean.

The retry threshold tells PHPBench to reject unstable samples. If an iteration deviates from the fastest recorded iteration by more than the threshold percent, PHPBench reruns it until the kept iterations agree:

#[Bench\Revs(1000)]
#[Bench\Iterations(5)]
#[Bench\RetryThreshold(2.0)]
public function benchHydrate(): void
{
    $this->hydrator->hydrate(Order::class, $this->row);
}
Enter fullscreen mode Exit fullscreen mode

Or set it globally so every subject inherits it:

{
    "runner.bootstrap": "vendor/autoload.php",
    "runner.path": "benchmarks",
    "runner.retry_threshold": 2.0
}
Enter fullscreen mode Exit fullscreen mode

A 2% threshold means PHPBench keeps retrying until your samples cluster within 2% of the best one. It uses the fastest iteration as the reference because the fastest run is the one with the least interference: no throttle, no context switch, no stolen core. Retry threshold does not make a noisy machine quiet. It makes PHPBench refuse to report until the noise stops. On CI runners, pin the threshold and expect longer runs. That trade is correct.

Regression assertions that fail the build

A number you read once and forget protects nothing. The reason to keep benchmarks in the repo is to catch the commit that makes a hot path slower. PHPBench does this two ways.

An absolute budget lives in the attribute:

#[Bench\Revs(2000)]
#[Bench\Iterations(5)]
#[Bench\Assert('mode(variant.time.avg) < 50 microseconds')]
public function benchEncode(): void
{
    $this->encoder->encode($this->message);
}
Enter fullscreen mode Exit fullscreen mode

Run it in CI and the process exits non-zero when the mean crosses 50μs. The build goes red on the commit that caused it, not three sprints later when someone notices p99 crept up.

Absolute budgets are brittle across machines, though. A CI runner is slower than your laptop, so the number that passes locally fails in the pipeline. The fix is to compare against a stored baseline instead of a hardcoded ceiling:

# store the current numbers under a tag
vendor/bin/phpbench run --tag=main

# on a PR, compare against that reference
vendor/bin/phpbench run \
    --ref=main \
    --report=default \
    --assert='mode(variant.time.avg) <= mode(baseline.time.avg) +/- 10%'
Enter fullscreen mode Exit fullscreen mode

Now the gate is relative. "This PR is allowed to be within 10% of main." Hardware drops out of the equation because both numbers came from the same runner in the same job. That is the assertion worth putting in CI. Cross a real 10% regression and it fails. Absorb normal jitter and it passes.

The mistakes that produce fiction

Every honest PHPBench setup is defending against the same short list. Recognize them and you stop shipping benchmarks that lie:

  • Too few revs. Timer resolution dominates and your mean is clock jitter. Push revs until deviation stabilizes.
  • No warmup on JIT paths. The first iteration pays for compilation and drags the mean up. Warm up, then measure.
  • Reading the mean, ignoring the deviation. A ±30% number is not a result. It's a demand for more iterations or a quieter machine.
  • Benchmarking a cache. If the subject memoizes, you measure the second call, which is free. Build fresh state per rev or name the method honestly.
  • Absolute assertions across machines. A microsecond budget that passes on your laptop fails on CI. Compare against a stored baseline instead.
  • Optimizing the wrong 0.1%. A function that's 3x faster and 0.1% of request time is a rounding error. Profile first, benchmark the part that matters.

The tool cannot save you from the last one. It can only tell you the truth about the thing you chose to measure. Choosing the right thing is still on you.

Where a benchmark like this belongs says something about your architecture. When the work you want to time is pure (a serializer, an encoder, a domain calculation that takes values in and gives values out) you can point PHPBench straight at it with no database, no HTTP, no framework boot in the way. That is exactly what a decoupled core buys you: the parts worth measuring sit at the center, free of the I/O that makes honest measurement impossible. Pushing those concerns to the edge is what Decoupled PHP is about.

Decoupled PHP — Clean and Hexagonal Architecture for Applications That Outlive the Framework

Available on Kindle, Paperback, and Hardcover. English, German, and Japanese editions out now — Portuguese and Spanish coming soon.

Top comments (0)