DEV Community: Ibrahim Hajjaj

Your Search Console script is undercounting and it will never tell you

Ibrahim Hajjaj — Thu, 24 Sep 2026 13:00:00 +0000

I have shipped both of these bugs. So, I would guess, has anyone who has written more than about fifty lines against the Google Search Console API. Neither throws, neither logs, and both produce numbers confident enough to put in a report.

They are the same mistake wearing two costumes: treating a row that is not there as a row whose value is zero.

Bug one: summing a dimension Google censors

You want clicks for last month. The obvious call gives you a breakdown by query, so you add up the clicks column.

const { data } = await searchconsole.searchanalytics.query({
  siteUrl,
  requestBody: { startDate, endDate, dimensions: ["query"] },
});
const clicks = data.rows.reduce((sum, r) => sum + r.clicks, 0);  // wrong

Search Console does not return every query. It withholds queries issued by very few users, as a privacy measure, and it does not tell you how many it held back or how much traffic they carried. The rows you get are real. The set is incomplete, and incomplete by a different amount every window.

So that sum is not "clicks last month". It is "clicks from the queries Google felt comfortable showing me last month", which is a smaller number that moves for reasons having nothing to do with your site.

Now do the thing everyone does next and compare two months. You have manufactured a trend out of Google's disclosure policy. A month where more of your traffic came from long-tail queries looks like a month where you lost traffic.

The fix is one line and completely unobvious:

// Clicks and impressions come from the date dimension. Summing the query
// dimension undercounts, because Search Console withholds low-volume queries,
// and that gap reads later as a real decline rather than as withheld rows.
const byDate = await searchAnalytics({ ...base, dimensions: ["date"], rowLimit: 500 });

Totals come from the date dimension. Always. The query and page dimensions are for "what are my top queries", never for "how much did I get". If you want both, that is two calls, and they will not reconcile, and the difference between them is the censored tail rather than a bug in your code.

Bug two: a cut-off list read as a complete one

This one is worse, because it does not undercount. It fabricates an event.

Search Console caps rows per query and returns them ordered by clicks, descending. My insight tools asked for 5,000 rows, got 5,000 rows, and said nothing about it:

function insightRequest(window, dimensions) {
  return { ...window, dimensions, rowLimit: 5000 };  // wrong
}

Sitting on a site with more than 5,000 rows, that cut lands exactly on the low-click rows. Which are exactly the rows a striking-distance report, a CTR-gap report and a cannibalisation report exist to find. The tools were reliably blind in precisely their own subject area, and nothing said so.

Then the period comparison joined two of those lists:

const joined = new Set([...currentByKey.keys(), ...previousByKey.keys()]);
// for each key: (current?.clicks ?? 0) - (previous?.clicks ?? 0)

?? 0. There it is again.

A query that ranked 4,900th last month and 5,100th this month is absent from the current list. It is scored as zero clicks. The tool reports it as having lost every click it had, ranks it near the top of the losers table, and hands you a collapse that never happened. From the tool whose only job is telling you what changed.

The fix is a probe row and a third state

Ask for one row more than you intend to analyse. That single extra row is the only thing that separates "there were exactly this many" from "the list was cut":

const INSIGHT_ROW_LIMIT = 5000;

function insightRequest(window, dimensions) {
  // The one extra row is the probe.
  return { ...window, dimensions, rowLimit: INSIGHT_ROW_LIMIT + 1 };
}

async function fetchInsightRows(clients, siteUrl, requestBody) {
  const response = await clients.searchConsole.searchanalytics.query({ siteUrl, requestBody });
  const fetched = response.data.rows ?? [];
  const rows = fetched.slice(0, INSIGHT_ROW_LIMIT).map(/* ... */);
  return { rows, truncated: fetched.length > INSIGHT_ROW_LIMIT };
}

The probe row is never analysed. It exists to answer one boolean.

Then that boolean has to change the arithmetic, not just add a footnote:

for (const key of joinedKeys) {
  const currentRow = currentByKey.get(key);
  const previousRow = previousByKey.get(key);
  // A cut-off list says nothing about the rows below the cut, so a key absent
  // from a truncated side is unknown there, not zero. Scoring it as zero turns
  // a row that merely slipped past the cut into a total gain or total loss.
  if ((currentRow === undefined && currentTruncated)
   || (previousRow === undefined && previousTruncated)) {
    droppedAsUnknown += 1;
    continue;
  }
  // ...
}

Note that it is conditional on truncation. If the list came back complete, an absent key genuinely is zero and should be reported as a real loss. The correction only applies where the data actually stopped.

And the count of what was withheld goes in the output, because a tool that silently drops rows to avoid lying has just found a quieter way to lie:

Note: Search Console returned more than 5000 rows for the current window,
so a key missing from a cut-off window is unknown there rather than lost;
keys left out of this comparison for that reason: 37.

The rule underneath both

Two states are not enough. You need three.

present with a value
absent, and you know the list was complete, so absent means zero
absent, and the list was cut off or censored, so absent means unknown

?? 0 collapses the second and third into each other. It is a default value standing in for a fact you do not have, and it is invisible because the type system is perfectly happy: number | undefined becomes number and everything downstream typechecks and runs and produces a chart.

Every API that paginates, ranks, samples, or withholds for privacy has this shape. Search Console does all four.

The other thing Search Console will not do for you

Its window is 16 months, rolling. Whatever last spring looked like is gone, and no API call brings it back. You cannot answer "did the change I made six weeks ago work" unless six weeks ago you thought to write the numbers down.

That is most of why I built the tool this code lives in. It snapshots Search Console, App Store listings, Play installs and wordpress.org active installs to files you keep, so a future you has a before to compare against.

It is seo-console-mcp, MIT, an MCP server and a CLI over the same tool registry. Four of the tools run with no credentials at all if you want to try it without the service-account dance.

Yes, a good chunk of it is a wrapper around an API. The part that is not a wrapper is everything above: a missing row is unknown rather than zero, a truncated list says it was truncated, a failed surface is recorded in place rather than dropped from the output, and totals never come from summing a dimension Google censors.

Build a support agent on Cloudflare Workers that cites its sources

Ibrahim Hajjaj — Wed, 23 Sep 2026 13:00:00 +0000

Most "AI support agent" tutorials end with a chat box that confidently makes something up. The fix is not a better prompt. It is making the agent show which page it got the answer from, so the person reading can check it.

This walks through putting one on Cloudflare Workers: what the code is, and the four things that broke when I did it. The gotchas are the useful half.

Everything here is from recourse, which is MIT and self-hosted, but the Workers-specific parts apply to anything you build on this runtime.

The handler is already a Worker

A Worker is a function from a Request to a Response. If your chat handler is written against Request, Response, fetch and Web Crypto, there is nothing to adapt:

import { createChatHandler } from '@recourse-ai/core/server'
import { models } from '@recourse-ai/core/models'
import knowledge from './knowledge.json'

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const handler = createChatHandler({
      index: knowledge,
      model: models.fromEnvironment(env),
      embedder: false,
      persona: {
        name: 'Ada',
        business: 'Lumen Coffee Roasters',
        fallback: "I can't find that in our help pages.",
      },
    })
    return handler(request)
  },
}

The index is imported rather than fetched, so it is bundled with the Worker and a cold start is a JSON parse. No warm-up, no vector database to reach across the network before you can answer the first question.

The wrangler.jsonc is short, and what is missing from it is the point:

{
  "name": "recourse-worker",
  "main": "src/index.ts",
  "compatibility_date": "2026-08-01",
  "observability": { "enabled": true }
}

No nodejs_compat. No compatibility flags at all.

Gotcha 1: import subpaths, not the root

This is the one that gets everybody, because the failure is a bundle size and a polyfill rather than an error you can read.

The package root re-exports ingest, which reads documents off disk, which imports node:fs. Import the root on a Worker and you drag the filesystem into a runtime that has no filesystem.

// wrong: pulls node:fs through the ingest re-export
import { createChatHandler } from '@recourse-ai/core'

// right: none of these touch the filesystem
import { createChatHandler } from '@recourse-ai/core/server'
import { models } from '@recourse-ai/core/models'
import { createAgent } from '@recourse-ai/core/agent'

Worth automating rather than remembering. A build step that greps the bundle for Node built-ins and fails on a hit costs twenty lines and catches this the first time somebody adds an import:

// check-bundle.mjs, run in CI
const NODE_BUILTINS = /\bnode:(fs|path|crypto|stream|buffer|os)\b/
if (NODE_BUILTINS.test(bundle)) {
  throw new Error('a Node built-in reached the Worker bundle')
}
if (bundle.length > 200_000) {
  throw new Error(`bundle is ${bundle.length} bytes, budget is 200KB`)
}

For reference the working bundle is 129.4 KB with no Node built-ins. When that guard fires it is telling you something Node-only leaked onto the serving path, which is a design signal and not just a size complaint.

Gotcha 2: `process` throws, it does not return undefined

On a Worker there is no process global. Reading it is a ReferenceError, not undefined, so the usual defensive pattern does not save you:

// throws on a Worker. It does not evaluate to the fallback.
const key = process.env.API_KEY ?? 'fallback'

The environment arrives as the second argument to fetch, per request, and anything that reads configuration has to be handed it:

export default {
  async fetch(request: Request, env: Env) {
    return createChatHandler({
      index,
      model: models.fromEnvironment(env),
    })(request)
  },
}

Note that the handler is built inside fetch rather than at module scope. At module scope there is no env yet.

This one has a nasty second-order version. If you leave the model unset it can fall back to a hosted gateway, and that provider reads process.env internally. So you get process is not defined at request time, which reads like a bundling problem, when the actual cause is a missing configuration value. Pass an explicit model and the error goes away.

Gotcha 3: `wrangler dev` does not forward your shell

Correct behaviour, and surprising exactly once. Your exported variables are not visible to the Worker. They go in .dev.vars, which you git-ignore:

OPENAI_COMPATIBLE_BASE_URL = "http://localhost:11434/v1"
OPENAI_COMPATIBLE_MODEL = "qwen3:4b"
OPENAI_COMPATIBLE_API_KEY = "ollama"

That points at a local Ollama, so you can develop the whole thing without an API key or a bill. In production the same three variables point at whatever OpenAI-compatible endpoint you like, including Workers AI.

Citations, which is the actual point

Retrieval returns passages, and each passage remembers which document it came from. Keep that association through to the answer and render it, rather than flattening the passages into one blob of context:

To hand a conversation to a person, you can use the escalate function,
which marks the conversation as belonging to a person [1].

[1] Handing a conversation to the desk you already run
    · Getting out of the way once a person arrives

The reason to care is not neatness. An answer with a source is falsifiable: the reader can open the page and see you are wrong. An answer without one has to be taken on trust, and support answers taken on trust are how people end up following instructions for the previous version of your product.

This also runs with no API key at all. With no embedder configured, retrieval is BM25 keyword matching, which needs no credential and no vector store. You get worse recall than hybrid search, and you get it before signing up for anything, which is the right trade for a first run.

Adding D1, and the limit that will catch you

Conversations go in D1 through a binding, so there is no connection pool, no credential and nothing to exhaust:

{
  "d1_databases": [
    { "binding": "DB", "database_name": "recourse", "database_id": "..." }
  ]
}

import { d1Store } from '@recourse-ai/store-d1'

const store = d1Store({ db: env.DB })

The free tier's limit is 50 queries per invocation, not per day. Per invocation. A handler that runs a query per retrieved chunk can reach it inside one request while your dashboard shows almost no usage for the day, so the failure reads as an intermittent bug rather than a quota. Batch reads, or keep the index bundled and use D1 only for conversation state.

One check worth running before a customer does

Every credential here is passed as an option rather than read from a global, which is the right shape and has one cost: nothing validates it until a webhook arrives and fails. A wrong signing secret looks exactly like silence.

npx @recourse-ai/core doctor

  FAIL  embedding model  the index was built with "nomic-embed-text" but the
                         environment says "mxbai-embed-large"
  ok    index            28 chunks from 6 documents, hybrid
  ok    model            "qwen3:4b" is available

That first line is the check worth having on its own. A query vector from one embedding model compared against stored vectors from another is not a comparable number. Nothing errors. You just get quietly bad answers, and you go looking at your prompt.

What this does not solve

Being honest about the shape of it, because "self-hosted" is not free:

You are on call. A hosted tool has someone else on call.
There is no dashboard to hand a support lead. Configuration is code in your repository, which is the point if your refund flow does not fit a settings page, and a real cost if it does.
Keyword-only retrieval has worse recall than hybrid. Configure an embedder when you have one; just know it works before you do.

If none of that is a problem, the whole thing is a Request to a Response and a JSON file.

The repo is github.com/ibrahimhajjaj/recourse, MIT. There is a live demo answering from its own documentation at recourse-demo.ibrhajjaj.workers.dev if you want to try breaking it before you read any code.

Why your app still says connected after the network is gone

Ibrahim Hajjaj — Wed, 16 Sep 2026 13:00:00 +0000

Two phones, one shared list. I add an item on the first phone. The second phone does not get it. Both phones show a green pill that says synced.

That is worse than an error. An error tells you to try something. A confident green pill tells you the other person is ignoring you.

Why the socket does not know

The app is offline-first on PowerSync, and the pill was reading PowerSync's own connection state, which comes from the socket.

A socket learns it is dead in one of two ways: the other end closes it cleanly, or a keepalive eventually fails to come back.

The clean close is the case everyone tests. Kill the server, watch the client flip to offline, done.

The other case is the one phones live in:

mobile data toggled off while the radio stays registered
a handoff to a cell that never completes
a socket that comes back from a long background period holding a file descriptor pointing at nothing

Nothing closes. No FIN arrives. From the client's side the connection is still open, and it stays open until the library's own heartbeat times out, which is measured in tens of seconds because a shorter one would burn battery reconnecting every time somebody walks into a lift.

For that whole window the indicator says synced and nothing syncs. And because the reconnect logic only fired once the engine admitted it was disconnected, no reconnect was kicked either. The app sat there, sure of itself, doing nothing.

The fix I had already shipped, which made this worse

Two weeks earlier I had fixed a different, real complaint. Bringing the app back to the foreground briefly drops and re-establishes the socket, so the pill flashed offline for a moment on every app switch. That flicker is exactly the jank you never see in apps people like, so I debounced it: hold the current state and only admit offline if the socket is still down after a grace window, with a reconnect inside the window cancelling the pending flip. Connections still reflect immediately, only disconnections wait.

Good fix. It also means the pill is now structurally slow to deliver bad news, sitting on top of a signal that was already slow to deliver bad news.

The two requirements pull opposite ways:

do not say offline for a blip that is about to resolve
do say offline the instant the connection is actually gone

You cannot satisfy both from one signal. You need a second one that is faster and independent.

The second signal is a boolean that is allowed to be null

The OS knows about the network before the socket does. expo-network exposes isConnected and isInternetReachable, and on the transports that populate them, they flip well before a heartbeat gives up.

The trap is that both are boolean | null, and null does not mean false. Some transports never populate them at all. Treat unknown as offline and you have built a new bug: an app that shows offline forever on hardware you do not own.

So the whole decision comes down to one function that refuses to guess:

// Only an EXPLICIT false means the OS is sure there is no internet.
// null/undefined stay "unknown", so an unknown reading never triggers
// a false offline.
export function osOffline(i: Pick<ConnInputs, 'osReachable' | 'osConnected'>): boolean {
  return i.osReachable === false || i.osConnected === false;
}

Everything else falls out of that:

export interface ConnInputs {
  psConnected: boolean;         // PowerSync believes its socket is up
  psFlowing: boolean;           // an upload or download is in flight right now
  osReachable: boolean | null;  // null = unknown on this transport
  osConnected: boolean | null;
}

export function resolveSyncPill(i: ConnInputs): SyncPill {
  if (osOffline(i)) return 'offline';   // the OS wins; don't trust a lagging socket
  if (!i.psConnected) return 'offline';
  return i.psFlowing ? 'syncing' : 'synced';
}

// A socket PowerSync still calls "connected" while the OS says there is no
// internet is wedged. It needs a forced bounce, not a wait on the heartbeat.
export function needsForcedReconnect(i: ConnInputs): boolean {
  return i.psConnected && osOffline(i);
}

Note what needsForcedReconnect does not fire on. If PowerSync already admits it is disconnected, ordinary retry handles it and forcing a bounce would just fight the backoff. The forced reconnect exists for one situation only: the two signals disagreeing in the specific direction where the socket is the one that is wrong.

Two things about the bounce itself, both learned the hard way:

Rate-limit it. A flapping link will otherwise storm reconnects. Ten seconds between forced bounces was enough.

Never clear the local database to fix a connection. It is a tempting big hammer and it is the wrong shape entirely: a wedged socket is recovered by re-establishing the socket. Wiping the replica costs the user a full resync to fix a problem that was never in the data.

Pulling this out into a module with no imports is what made it testable at all. The provider it came from cannot be instantiated outside a device. The decision can:

it('socket "connected" but OS says unreachable -> offline (was wrongly "synced")', () => {
  expect(resolveSyncPill({ ...base, osReachable: false })).toBe('offline');
});

it('unknown OS reachability (null) follows PowerSync (no false offline)', () => {
  expect(resolveSyncPill({ ...base, osReachable: null, osConnected: null })).toBe('synced');
});

The regression that was already there

Forty-four minutes after the fix landed, the pill went back to lying, because of a line nobody thought was about connections at all. It was the oldest line in the repository.

The data layer refreshes a snapshot of everything the UI needs. That refresh had also been stamping the connection state in passing:

- sync: db.connected ? 'synced' : 'offline',
+ // The pill is owned solely by applyStatus/recompute (PowerSync status + OS
+ // reachability). A data refresh must NOT restamp it from db.connected alone,
+ // that would clobber an OS-driven "offline" back to "synced" while the socket
+ // is still wedged.

That line was present in the repository's first commit, and it had been correct in every commit since. It is not stale code and whoever wrote it was not careless. When the socket was the only input, stamping the pill from the socket was simply the truth written down twice.

It became wrong the moment the same derived value acquired a second input. That happened in a different file, on the same afternoon, and every test stayed green.

Derived state needs exactly one owner. Not "one place that mostly sets it".

The same commit closed a second hole. The forced reconnect is await disconnectSync() then await connectSync(), and a sign-out or a component teardown can land in the gap between them, so the reconnect resurrects a connection something else just tore down:

await disconnectSync();
// Cleanup or a sign-out can land during the await; don't resurrect a
// connection the teardown/auth-change just tore down.
if (disposed.current || !currentUserId()) return;
await connectSync();

Every await in a lifecycle-bound async function is a place the world is allowed to change underneath you. Re-check your preconditions after it, not just before.

The three parts worth stealing

When a signal is structurally slow, add a faster independent one rather than tuning the slow one. No timeout value would have fixed the heartbeat, because the heartbeat is long for a good reason.
null is not false. A tri-state input needs a tri-state decision. Only explicit evidence of absence counts as evidence; unknown stays unknown and defers to whatever else you have.
One owner per piece of derived state. Adding an input to a derived value silently promotes every existing writer of that value into a potential bug. The writer did not change and was not wrong when it was written; the meaning of what it was writing changed underneath it. Nothing in the type system or the tests notices, because a two-state read of a now-tri-state world still typechecks and still passes.

The app is Psst, shared lists on iOS and Android. The indicator is honest now, which mostly means it tells you nothing is happening at the moment nothing is happening, instead of a minute later.

Why changing your WordPress domain corrupts the database

Ibrahim Hajjaj — Thu, 10 Sep 2026 13:00:00 +0000

You are moving a site from example.com to example.org. You run a search and replace over the database. Most of the site comes back fine, and then the homepage is blank, or a page builder layout is empty, or a widget area has quietly vanished.

This happens because a WordPress database is not a pile of strings. Three separate encodings are stacked inside single column values, and a plain substitution corrupts all three in different ways.

The byte counts

PHP serialization stores the length of every string alongside it:

a:1:{s:3:"url";s:23:"https://example.com/img";}

That s:23 is a byte count. Replace example.com with a-much-longer-domain.com as plain text and you get:

a:1:{s:3:"url";s:23:"https://a-much-longer-domain.com/img";}

The string is now 36 bytes and still claims to be 23. unserialize() returns false, WordPress gets nothing back where it expected an array, and whatever read that option silently renders empty.

The fix is not to recalculate the numbers with a regex. It is to never work on the serialized text at all: unserialize, walk the structure, replace inside the leaf strings, then re-serialize and let PHP write the lengths.

One detail matters when you do this. Unserializing attacker-controlled or just old data can instantiate objects:

$decoded = @unserialize( $value, array( 'allowed_classes' => false ) );

allowed_classes => false turns every serialized object into a __PHP_Incomplete_Class instead of constructing it. You are reading arbitrary rows out of a database you may not have written. Do not construct their objects.

The JSON inside the serialization

Page builders make this worse. Elementor stores an entire layout as a JSON document, and then stores that JSON document as a string inside a serialized value. So the structure is: serialized PHP, containing a string, containing JSON, containing your URLs.

Unserializing gets you the JSON as one opaque string. Your replacement runs against it as text, which brings back the byte-count problem one level down, and it also misses matches entirely, for a reason that catches almost everybody.

The escaped slashes

json_encode() escapes forward slashes by default. So a URL that reads https://example.com in the browser is stored as:

https:\/\/example.com

Search for https://example.com and you match nothing. The URL is right there, visible in the row, and your query returns zero results.

You cannot fix this by searching for the escaped form instead, because now you are back to substituting text inside a JSON string inside a serialized value, breaking the outer byte count and possibly the inner escaping too.

You have to decode the JSON, walk it, replace in the leaves, and re-encode. And when you re-encode, the escaping has to come back the way you found it, because rewriting a row's encoding style is a change you did not ask for and cannot see in a diff:

$flags = JSON_PRESERVE_ZERO_FRACTION;
if ( false === strpos( $value, '\\/' ) ) {
    $flags |= JSON_UNESCAPED_SLASHES;
}
if ( false === strpos( $value, '\\u' ) ) {
    $flags |= JSON_UNESCAPED_UNICODE;
}

Read that the right way round. It does not decide how slashes should be escaped. It looks at what the original value did and matches it. If the stored JSON had \/ in it, the output keeps \/. If it had \uXXXX unicode escapes, the output keeps those too. Whatever wrote that row gets its own convention back.

Knowing when to refuse

The hard part is not the replacing. It is noticing when a round-trip through json_decode() and json_encode() would not be lossless, and stopping.

Two cases:

Large integers. JSON has no integer type distinct from float, and PHP's decoder will happily turn a 17-digit order ID into a float and hand it back to you rounded. That is silent data loss in a WooCommerce table.

Duplicate keys. A JSON object with the same key twice is legal to parse and impossible to represent in a PHP array. Decoding keeps the last one. Re-encoding writes a document that is missing data the original had.

Both are detectable before you touch anything:

if ( 1 === preg_match( '/[0-9]{16}/', $value ) || $this->json_has_duplicate_keys( $value ) ) {
    // do not round-trip this value
}

When either is true the right behaviour is to skip the row and say so, rather than write a subtly different document and call it success. Every skip carries a reason:

const SKIP_MALFORMED_SERIALIZED = 'malformed_serialized';
const SKIP_DEPTH_EXCEEDED       = 'depth_exceeded';
const SKIP_LOSSY_JSON           = 'lossy_json';

A tool that reports "42 rows changed, 3 skipped because a lossless round-trip was not possible" is more useful than one that reports 45 changed, because the second one is lying in a way you will discover months later.

Base64, briefly

Some plugins base64 the values they store. Same shape of problem: decode, replace, re-encode, and only when the decode is unambiguous. Strict mode alone is not enough, because plenty of ordinary short strings decode without complaint. The test that actually works is a round-trip:

$decoded = base64_decode( $value, true );

return false !== $decoded && base64_encode( $decoded ) === $value;

If re-encoding the decoded bytes does not reproduce the original character for character, it was not canonical base64 and you should leave it alone.

What this means for doing it at all

Three practical conclusions.

Preview before writing. Every one of these failures is invisible at the moment it happens and expensive later. If a tool cannot show you the matches grouped by table and column, with the changed characters marked, before anything is written, you are finding out what it did by browsing your own site afterwards.

Snapshot the affected rows. Not a full database dump, which nobody takes for a search and replace. Just the rows about to change, so undo is one operation.

Never write to the user tables. Not "excluded by default", which is a checkbox somebody will untick at 2am. wp_users and wp_usermeta should be structurally unreachable, because a replacement that touches them can lock you out of the site you are in the middle of fixing.

The implementation of all of this is GPLv2 at github.com/ibrahimhajjaj/lucid-search-replace, and it is free on wordpress.org. I found the escaped-slash case the way everyone finds it: staring at a URL that was plainly visible in the row while the search returned zero results, convinced for most of an evening that I had lost my mind rather than that json_encode had quietly put a backslash in front of every forward slash.

That is the one to check first in whatever tool you already use. Put a page builder URL in the search box. If it finds nothing while you can see the URL sitting in the row, that tool is not reading the JSON, and the next thing it does to your database will be worse than finding nothing.

Your benchmark's single score is a weighting nobody agreed to

Ibrahim Hajjaj — Mon, 07 Sep 2026 13:00:00 +0000

Disclosure before anything else, because it is the thing that should make you read the rest suspiciously: I sell one of the nine plugins in the benchmark this post is about. It is currently first on the overall board and second on the speed board, which turns out to be the point.

I built the benchmark to answer a simple question. Which WordPress backup plugin is fastest? Two plugins get byte-identical fixed snapshots on identical sites, backup and restore run in containers with no outbound network, each pairing does a warm-up and three measured runs, and the results feed an Elo rating.

The measuring was the easy part. The hard part was working out whether the number at the top of the page was allowed to exist at all.

The two axes disagree, and not by accident

Every run produces two things worth comparing: how long the backup took, and how big the archive came out. Rank the field by each and you get two orders.

Kendall's tau between them is -0.50.

That is not "unrelated". That is actively opposed. Across the whole field, the plugins that finish fastest write the biggest archives. Which, stated that way, is obvious: compression costs CPU time, and skipping it buys you seconds and costs you bytes. What is less obvious is what it does to a leaderboard.

If I rank by speed, I am partly ranking by who compresses least. Not as a quirk of one plugin. As a rule, across nine of them.

Six of the nine are Pareto-optimal: no other plugin beats them on both speed and size at once. Only three are beaten on both axes and can be honestly called worse. For the other six, "better" is a question about your constraint, not about the software.

So any single ranking number I publish is a weighting between two axes that disagree. I would be picking that weighting on your behalf, applying it to your decision, and not telling you I had done it.

This is settled ground, and someone already wrote the rules

I went looking for whether serious benchmark bodies had a position on composite scores. They do, and it is stricter than I expected.

SPEC's Fair Use rules have a whole section on what they call a derived value: any number that is a function of a benchmark metric plus something else. Composites are explicitly allowed. What is not allowed is presenting one as the benchmark's own metric.

Their worked example is close enough to my problem to sting. A gaming society builds a composite from a weighted subset of one SPEC benchmark plus a weighted subset of another, and calls it "GamePerfMark". SPEC's ruling: the composite may be useful and interesting, the weighting and the subsetting were done by the society rather than by SPEC, and calling it "SPECgame" would be a violation.

Two other rules in the same document do real work:

The basis for comparison must be stated. Not "the fastest". The fastest among what, measured how, retrieved when.
Several benchmarks define required metrics that must be quoted alongside any headline figure, so the flattering number cannot travel on its own.

None of that is about honesty in the abstract. It is a description of how a number gets misused once it leaves the page it was measured on, written by people who watched it happen for thirty years.

The second place a number gets overconfident

Ranking is one problem. The error bar next to the ranking is another, and it took a real bug to show me.

The confidence interval started as the textbook approximation:

CI = 1.96 * (400 / sqrt(n))      n = battles played

    5 battles  ->  +/- 351
   50 battles  ->  +/- 111
  200 battles  ->  +/-  55

More battles, tighter interval. That formula assumes each battle is an independent observation.

Then I looked at a record where 24 of 40 battles were failures, and every one of the 24 was the same fault reproduced from one bug.

That is one fact about the plugin, observed 24 times. It is not 24 facts. Feed it in as 24 and sqrt(n) hands you an interval that claims a precision the evidence does not have.

The fix is the cluster-sampling design effect, taken at its most conservative: outcomes sharing a root cause are treated as perfectly correlated, so a cluster of any size counts once.

export function effectiveSampleSize(
  battleCount: number,
  clusterSizes: number[],
): number {
  if (battleCount <= 0) return 0;
  let effective = battleCount;
  for (const size of clusterSizes) {
    if (size > 1) effective -= size - 1;
  }
  // Never claim fewer observations than the number of distinct things seen.
  return Math.max(effective, Math.min(battleCount, clusterSizes.length || 1));
}

Failures are grouped by harness version, hosting tier and phase. Forty battles that collapse to three effective observations take the interval from ±124 to ±453. Wider, which is the entire point, and wider by an amount you can check: 1.96 * (400 / sqrt(3)).

The part I want to be clear about, because it is the part that makes this honest rather than convenient: the rating itself does not move. The losses are real, the plugin earned them, and nothing about the cluster makes them count for less. Only the stated uncertainty widens, and it widens against the plugin's own record looking decisive.

The test that guards it is a one-liner:

it("widens the interval rather than narrowing it", () => {
  const honest = calculateConfidence(effectiveSampleSize(40, [24]));
  const naive = calculateConfidence(40);
  expect(honest).toBeGreaterThan(naive);
});

What that costs the person who built it

The board carries two Elo columns, because of everything above. One for speed, one overall. On speed my plugin, SafeGuard, is second at 1320. All-in-One WP Migration is first at 1465, and it is first by a distance I am not going to argue with.

Now the part I could have quietly left out. On the overall column SafeGuard is first, at 1427 against WPvivid's 1256, and All-in-One, the plugin that just beat me on speed, is fourth at 1148. That 171-point overall gap is the number I would use if I were selling. My interval is ±124 and it still overlaps second, so I am not going to call that settled either.

That inversion is this entire post in one row. The fastest plugin in the arena is fourth overall, and the rank correlation between the two axes is exactly -0.50. Whichever column you sort on, you are quietly sorting against the other one.

Both figures are true. They are on the same page, in the same row, and which one you meet first is a design decision I made. That is the entire argument of this post, aimed at myself: a leaderboard is an editorial act, and the only defence is showing the reader the axis you sorted on and the one you did not.

I also had to check whether a single rating was even valid here, because Elo assumes transitivity: if A beats B and B beats C, A should beat C. When that fails, ratings start reflecting who happened to be scheduled against whom. Measured across the pairwise tier medians, the cyclic component sits at 0.48 of the transitive one for speed and 0.52 for size. Both comfortably below 1, so ranking each axis on its own holds up. The rating system was not the problem. The single number on top of it was.

Three things to check in any benchmark you read

Is the headline one number, and does the page say what it weighted? If it is a composite and the weighting is not stated, you are reading somebody's opinion with a decimal point on it.
Do the axes agree? If the benchmark reports more than one thing, ask whether they move together. If they move against each other, ranking by one is silently ranking by the inverse of the other.
Does the error bar assume independence it does not have? A record where one bug decided half the results is one observation wearing a large number. And check the arithmetic while you are there, then check your check. Writing this post I decided a figure on my own methodology page was stale, an interval of ±453 where I had worked out the formula gives ±190, and I said so here. I was wrong. The page was right. The effective sample size is three, not seventeen, and ±453 is what the code produces; ±190 came from a unit-test fixture I had read instead of a function I could have run. Reading the formula told me one thing and running it told me another, and only one of those is evidence.

The benchmark is at backuparena.com. 581 completed runs, 180 scored, 401 excluded, and the methodology page carries all of the above in more detail than this post. I should be precise about that exclusion number rather than let it sound braver than it is: 360 of the 401 are simply superseded by a later run of the same pairing, which is bookkeeping. Forty-one were excluded for cause. Every one of the 401 is browsable at /battles?status=excluded, and opening any row shows the reason it was thrown out.

The largest of those is worth stating because it is the kind of fault that flatters nobody. On the media-heavy profile our archive lister read one part of a multi-part archive at a time, so two plugins recorded 28,801 upload files against 30,582 for every other plugin. A silent undercount, penalising exactly the two plugins that happened to split their archives, and nothing about the numbers looked wrong.

The code is not public, and that is a real limitation rather than an oversight. Every claim above is a measurement I can show you the log for. It is not one you can rerun yourself, and I would rather write that sentence than have somebody else write it for me.