"A Fair Coin Isn't Enough: When a Perfectly Randomized Experiment Is Impossible to Analyze"

Originally published on hexisteme notes.

We built a randomized experiment to earn the right to say one word — because. The random assignment was cryptographically fair; we verified it. Weeks in, an audit found that the experiment could not be analyzed at all, and never could have been. The coin was flawless. The join from the coin to the thing it decided did not exist in the data.

To make a randomized experiment analyzable you need three things that have nothing to do with the randomizer: a join that links each draw to the unit it was applied to and that unit's outcome, a guard that makes selective re-drawing impossible or at least detectable, and a power calculation done before you start so you know the sample can answer the question. We had a provably fair coin and none of the three. This is what each one is, why its absence is fatal, and what you actually have to record at the instant of assignment.

Why randomize at all: to earn one causal word

Most of what you can measure about a running system is observational — patterns under whatever policy happens to be in force. Observational data can generate hypotheses, but it cannot, by construction, tell you that a change caused an outcome, because the thing you changed is tangled up with everything else that moved at the same time. There is exactly one clean escape: assign the treatment at random. Randomization is what severs the treatment from all the confounders and lets you say the forbidden word — because — about the difference between arms.

So for one specific class of decisions we did the disciplined thing. Instead of always routing a task to the default handler, a dispatcher flipped a fair coin and sent it to arm A or arm B. That single randomized track was meant to be the one place in the whole system where a causal claim would be defensible. Everything hinged on it being done right.

The coin was provably fair — and that was the easy 10%

The randomizer used a cryptographically seeded source (SystemRandom().choice, backed by the OS entropy pool), so the draw was an honest, independent p=0.5 flip. When the observed split came in lopsided — roughly four to one — the instinct was to suspect a broken coin. That instinct was wrong: with that few draws, a two-sided binomial test puts a 4:1 split at P ≈ 0.375, nowhere near anything you would call biased. The coin was fine.

Here is the trap. Verifying the coin feels like verifying the experiment. It is vivid, it is mathematical, it produces a satisfying green check. And it is maybe ten percent of what makes a randomized experiment work. The other ninety percent is bookkeeping so boring nobody writes a test for it — and that is exactly where this one had already failed.

The fatal bug: nothing joined the draw to the task it dispatched

To analyze a randomized experiment you compute an intention-to-treat comparison: for every unit, what arm was it assigned, and what outcome did that unit get. That requires a join key — a stable identifier stamped on the unit at the moment of assignment and carried through to wherever the outcome is recorded. It is the spine of the entire analysis.

Our dispatcher was a function that took no arguments, returned an arm, and appended the draw to a log. The log recorded that a draw happened and which arm won. It did not record which task the draw was for. Nothing downstream ever read the dispatch log to tag the resulting work. In other words, the assignment and the outcome lived in two separate piles with no key joining them.

You cannot backfill this. The pairing between a draw and the task it dispatched is only knowable at the instant the dispatcher hands the task off. Once that moment passes without a shared id being written down, the link is gone — not hard to reconstruct, gone. Afterward you have N draws and M outcomes and no way to say which draw produced which outcome. You can count each pile. You cannot pair them. The intention-to-treat comparison — the only reason the experiment existed — is uncomputable by any code you could ever write against that data.

Three ways the ledger couldn't defend itself

The missing join was the headline, but the same audit found three smaller failures, each of which independently prevents an experiment from being trustworthy even by an honest operator.

• No guard against selective re-randomization. Nothing in the pipeline made "one unit, one assignment, recorded once" an enforced invariant, and nothing would have detected a second draw for the same unit. The danger here is not that anyone cheated — it is that the experiment has no mechanism to prove it couldn't have. An experiment that cannot rule out selective re-draws cannot defend its own result against that exact objection.
• Test draws share a pile with real draws. Draws made while building and debugging the harness landed in the same log as production draws, separated only by a note in a design doc — not a flag in the data. A cluster of near-simultaneous draws (42 milliseconds apart, machine-fast, obviously not real dispatches) sat unclassified in the log, occupying exactly the position where genuine data would be counted. The rule "developer draws don't count" is worthless if it lives in prose instead of an is_test column.
• The falsifier that can't fire. The experiment was pre-registered with a kill switch: "if assignment/outcome mismatch exceeds 20%, declare the design a failure and stop." But mismatch is a quantity you compute from the join — and there is no join. So the falsifier references a number that cannot be measured. It can never fire. A stopping rule you cannot compute is not a stopping rule; it is a comforting sentence in a spec.

## Even fixed, it was underpowered — and nobody had checked

Suppose we fix all of the above tomorrow. There is a separate, earlier failure hiding underneath: the experiment was never powered. No minimum detectable effect was computed before it started. When we projected the effective sample size forward to the pre-registered deadline, it came to roughly 14–19 usable units against a target of 40 — under half. The experiment was on track to end inconclusive no matter how clean the bookkeeping was.

This is the quietest and most common way experiments waste themselves. Without a pre-planned MDE, an inconclusive result is ambiguous: you cannot distinguish "the two arms are genuinely equivalent" from "we never had enough units to see a difference." Those are opposite conclusions that produce identical-looking null results. The power calculation is what forces you to find out, before you spend the sample, whether the question is even answerable at the scale you can afford.

What instrumenting a randomized experiment actually requires

The transferable lesson is that randomization is the part that comes free from a library, and everything that makes the randomization usable is manual work you have to design in from the first line. Concretely, before the first real draw:

• Stamp a joined record at assignment time. One row per draw carrying a stable unit_id, the assigned arm, an is_test flag, and a timestamp. The unit_id is non-negotiable — it is the key the entire analysis hangs from.
• Carry the unit id to the outcome. The place where you measure the result must write down the same unit_id. If the outcome side doesn't know the key, the assignment side stamping it was pointless.
• Make re-draws impossible or loud. Enforce one-assignment-per-unit, or at minimum emit a warning when the same unit is drawn twice within a short window, so selective re-randomization can't happen silently.
• Separate test from production in the data, not in prose. The is_test flag, set at draw time, is the only thing that reliably keeps development noise out of your effect estimate.
• Compute the minimum detectable effect first. Decide, before you begin, the smallest effect worth detecting and whether your reachable sample can detect it. If it can't, you don't have an experiment yet — you have a plan to collect an ambiguous null.

Randomizing an assignment is a one-line call to a good RNG. Turning that into an experiment you can actually analyze is a join, a guard, a flag, and an arithmetic check you do up front. We had the one-liner and skipped the four boring things, and the result was a beautifully fair coin deciding an experiment that, on the day we looked, could not answer its own question. The coin was never the hard part.

---

More notes at hexisteme.github.io/notes.