My A/B test wasn't broken. My counter was.

I ran a 50/50 homepage split for a couple of weeks. The arm counts came back 178 to 57.

That is not 50/50. So I did what you'd do: I went and stared at the randomizer. Read the middleware. Read the cookie logic. Wrote a script to call the assignment function ten thousand times and count the buckets. It came back 4,981 / 5,019, which is exactly what Math.random() < 0.5 is supposed to do, and left me with a working randomizer and a number that said otherwise.

The randomizer was fine. The counter was lying.

Here's what happened. I had one function doing two jobs:

function setVariant(arm) {
  gtag("set", { ab_variant: arm });          // must run on every page load
  gtag("event", "ab_variant_assigned", {...}); // should run ONCE per visitor
}

Those two calls look like they belong together. They do not. The set call attaches the arm as a sticky dimension to every event that follows it, so it has to repeat on every single page load or your later events lose the label. The event call records an exposure: one human entering the experiment. It should fire once.

I called that function from the root layout. Every page load, sitewide, fired both. So my "exposure" count was really a pageview count.

And then the part that actually fooled me: the two arms rendered through different layout trees, so they accumulated pageviews at different rates. If both arms had over-counted evenly I'd have seen 178 vs 174, shrugged, and moved on with a wrong-but-harmless denominator. Instead the uneven inflation produced a lopsided ratio, which is exactly the shape a broken randomizer makes. The bug disguised itself as a different bug.

The fix was to split the two operations and guard only the exposure one, keyed by arm value so a re-bucketed visitor counts once per arm instead of vanishing. If the guard's storage throws (Safari private mode does this), it returns "yes, count it". Over-counting one arm is recoverable, silently dropping it is not.

Three things I took from it:

set and event have opposite idempotency requirements. One must repeat, one must not. If they share a function, one of them is wrong. Mine had been wrong for weeks.

Mounting anything in a root layout makes it a per-pageview thing. That's the whole job of a root layout. Any one-shot event you put there (experiment assignment, activation, funnel entry) quietly becomes a tally of page loads instead.

A ratio that contradicts your randomizer is not evidence about your randomizer. It's evidence that one of the two is wrong, and the counter is the one nobody tests. I spent a day proving the coin was fair. Nobody had checked whether I was writing the results down correctly.

The conversion counts were fine, incidentally. Only the denominators were inflated, which means every conversion rate I'd computed was garbage while every number feeding it was correct. That is a fun way to lose two weeks of a test.