Your benchmark's single score is a weighting nobody agreed to

Ibrahim Hajjaj — Mon, 07 Sep 2026 13:00:00 +0000

Disclosure before anything else, because it is the thing that should make you read the rest suspiciously: I sell one of the nine plugins in the benchmark this post is about. It is currently second on the board.

I built the benchmark to answer a simple question. Which WordPress backup plugin is fastest? Two plugins get byte-identical fixed snapshots on identical sites, backup and restore run in containers with no outbound network, each pairing does a warm-up and three measured runs, and the results feed an Elo rating.

The measuring was the easy part. The hard part was working out whether the number at the top of the page was allowed to exist at all.

The two axes disagree, and not by accident

Every run produces two things worth comparing: how long the backup took, and how big the archive came out. Rank the field by each and you get two orders.

Kendall's tau between them is -0.50.

That is not "unrelated". That is actively opposed. Across the whole field, the plugins that finish fastest write the biggest archives. Which, stated that way, is obvious: compression costs CPU time, and skipping it buys you seconds and costs you bytes. What is less obvious is what it does to a leaderboard.

If I rank by speed, I am partly ranking by who compresses least. Not as a quirk of one plugin. As a rule, across nine of them.

Six of the nine are Pareto-optimal: no other plugin beats them on both speed and size at once. Only three are beaten on both axes and can be honestly called worse. For the other six, "better" is a question about your constraint, not about the software.

So any single ranking number I publish is a weighting between two axes that disagree. I would be picking that weighting on your behalf, applying it to your decision, and not telling you I had done it.

This is settled ground, and someone already wrote the rules

I went looking for whether serious benchmark bodies had a position on composite scores. They do, and it is stricter than I expected.

SPEC's Fair Use rules have a whole section on what they call a derived value: any number that is a function of a benchmark metric plus something else. Composites are explicitly allowed. What is not allowed is presenting one as the benchmark's own metric.

Their worked example is close enough to my problem to sting. A gaming society builds a composite from a weighted subset of one SPEC benchmark plus a weighted subset of another, and calls it "GamePerfMark". SPEC's ruling: the composite may be useful and interesting, the weighting and the subsetting were done by the society rather than by SPEC, and calling it "SPECgame" would be a violation.

Two other rules in the same document do real work:

The basis for comparison must be stated. Not "the fastest". The fastest among what, measured how, retrieved when.
Several benchmarks define required metrics that must be quoted alongside any headline figure, so the flattering number cannot travel on its own.

None of that is about honesty in the abstract. It is a description of how a number gets misused once it leaves the page it was measured on, written by people who watched it happen for thirty years.

The second place a number gets overconfident

Ranking is one problem. The error bar next to the ranking is another, and it took a real bug to show me.

The confidence interval started as the textbook approximation:

CI = 1.96 * (400 / sqrt(n))      n = battles played

    5 battles  ->  +/- 351
   50 battles  ->  +/- 111
  200 battles  ->  +/-  55

More battles, tighter interval. That formula assumes each battle is an independent observation.

Then I looked at a record where 24 of 40 battles were failures, and every one of the 24 was the same fault reproduced from one bug.

That is one fact about the plugin, observed 24 times. It is not 24 facts. Feed it in as 24 and sqrt(n) hands you an interval that claims a precision the evidence does not have.

The fix is the cluster-sampling design effect, taken at its most conservative: outcomes sharing a root cause are treated as perfectly correlated, so a cluster of any size counts once.

export function effectiveSampleSize(
  battleCount: number,
  clusterSizes: number[],
): number {
  if (battleCount <= 0) return 0;
  let effective = battleCount;
  for (const size of clusterSizes) {
    if (size > 1) effective -= size - 1;
  }
  // Never claim fewer observations than the number of distinct things seen.
  return Math.max(effective, Math.min(battleCount, clusterSizes.length || 1));
}

Failures are grouped by harness version, hosting tier and phase. Forty battles with one cluster of 24 become 17 effective observations, and the interval goes from ±124 to ±190. Wider, which is the entire point, and wider by an amount you can check: 1.96 * (400 / sqrt(17)).

The part I want to be clear about, because it is the part that makes this honest rather than convenient: the rating itself does not move. The losses are real, the plugin earned them, and nothing about the cluster makes them count for less. Only the stated uncertainty widens, and it widens against the plugin's own record looking decisive.

The test that guards it is a one-liner:

it("widens the interval rather than narrowing it", () => {
  const honest = calculateConfidence(effectiveSampleSize(40, [24]));
  const naive = calculateConfidence(40);
  expect(honest).toBeGreaterThan(naive);
});

What that costs the person who built it

The board carries two Elo columns, because of everything above. One for speed, one overall. My plugin, SafeGuard, is second on the speed board at 1268. BackWPup is third at 1263. Both carry ±113.

Five points inside a ±113 interval is a tie. Not a narrow win, not "edges out". A tie, and the page says so in those words rather than letting me quote the higher number and walk away.

Now the part I could have quietly left out. On the overall column SafeGuard is first, at 1442 against All-in-One WP Migration's 1238. That is a 204-point gap, and it is the number I would use if I were selling. Their intervals still overlap by 22 points, so I am not going to call that settled either.

Both figures are true. They are on the same page, in the same row, and which one you meet first is a design decision I made. That is the entire argument of this post, aimed at myself: a leaderboard is an editorial act, and the only defence is showing the reader the axis you sorted on and the one you did not.

I also had to check whether a single rating was even valid here, because Elo assumes transitivity: if A beats B and B beats C, A should beat C. When that fails, ratings start reflecting who happened to be scheduled against whom. Measured across the pairwise tier medians, the cyclic component sits at 0.48 of the transitive one for speed and 0.52 for size. Both comfortably below 1, so ranking each axis on its own holds up. The rating system was not the problem. The single number on top of it was.

Three things to check in any benchmark you read

Is the headline one number, and does the page say what it weighted? If it is a composite and the weighting is not stated, you are reading somebody's opinion with a decimal point on it.
Do the axes agree? If the benchmark reports more than one thing, ask whether they move together. If they move against each other, ranking by one is silently ranking by the inverse of the other.
Does the error bar assume independence it does not have? A record where one bug decided half the results is one observation wearing a large number. And check the arithmetic while you are there: I found a stale figure in my own methodology page writing this post, an interval quoted as ±453 where the formula gives ±190, because somebody had passed the number of clusters into sqrt(n) instead of the effective sample size. The correction was right and the number printed beside it was not.

The benchmark is at backuparena.com. 581 completed runs, 180 scored, 401 excluded, and the methodology page carries all of the above in more detail than this post. I should be precise about that exclusion number rather than let it sound braver than it is: 360 of the 401 are simply superseded by a later run of the same pairing, which is bookkeeping. Forty-one were excluded for cause. Every one of the 401 is browsable at /battles?status=excluded, and opening any row shows the reason it was thrown out.

The largest of those is worth stating because it is the kind of fault that flatters nobody. On the media-heavy profile our archive lister read one part of a multi-part archive at a time, so two plugins recorded 28,801 upload files against 30,582 for every other plugin. A silent undercount, penalising exactly the two plugins that happened to split their archives, and nothing about the numbers looked wrong.