Most A/B testing vendors quote one performance number, and it is the wrong one.
The number is snippet size. Ours is about 31 KB gzipped. That figure tells a visitor nothing at all about what they saw, or when.
So here is the measurement we actually take, the method behind it, and the reasons it is a floor rather than a result.
The numbers
p75 time from navigation start to a variation landing in the DOM, over 200 page loads in each condition:
| Condition | p75 |
|---|---|
| Single page app route change | 2.2 ms |
| Repeat view, warm cache | 19.9 ms |
| First visit, empty cache, throttled | 529 ms |
The throttle is 1.6 Mbps down with a 150 ms round trip. The test runs in headless Chromium against the real built runtime, with one variation that rewrites a heading above the fold.
Separately, because it is the thing a visitor actually experiences: on that throttled first visit the original heading was on screen first on all 200 loads, for a median of 326 ms.
On a cached repeat view, no painted frame showed the original on 199 of the 200 loads. On the route change, none did at all.
Why 529 ms is a floor and not a result
The measurement runs against a local origin.
That means it carries the throttle and the transfer, but none of the DNS, TLS or edge latency a real first visit also pays.
It also throttles the network without throttling the processor. A first visit on a mid range phone is slower than 529 ms, not faster.
We would rather say that than let someone quote 529 ms at a prospect as a field figure.
Why the page flickers at all
The runtime is injected as a dynamic <script>. It does not block the parser.
That is a deliberate choice, and the cost of it is the 326 ms on that first visit. The original renders, then the variation replaces it.
The usual fix is an anti-flicker overlay that hides the page while the runtime decides. Ours exists, and it is unchecked by default.
Checked, the snippet adds a rule and puts the class on <html> at the top of the page:
.abtestly-async-hide { opacity: 0 !important }
The class comes off once variants apply, or after two seconds, whichever is first. The page never stays hidden.
The reason it is off by default is the arithmetic of who pays for it.
A page-wide hide costs every visitor on every page, including everyone who was never bucketed into any experiment. If the config fetch is ever slow, that is a visibly empty site.
Flicker only shows on the pages a variant actually changes. So the box is there for the test that needs it, and not before.
There is a second reason, and it is the one that makes this more than a preference. Our own speed guardrail flags a variation as slower when its p75 largest contentful paint sits 400 ms or more above control. An always-on overlay would ship exactly the regression that guardrail exists to catch.
Where the variant decision happens
In the browser, after the runtime arrives. There is no server hop in the path the benchmark measures.
Assignment is a deterministic hash with no shared state, so every visitor's bucket is computed from their own id alone:
hash1 = murmur3(salt + "/" + userId + "/Exposure") mod 10000
enter = hash1 < trafficAllocationBp
salt is the experiment's immutable UUID, minted at creation and never changed. userId is a UUIDv7 held in localStorage and a cookie. trafficAllocationBp is allocation in basis points, so 10000 is 100%.
Entry is decided first, variant second. The same two-step model Amplitude uses.
Because it is a pure function of those inputs, the same hash on the same inputs reproduces the same bucket in a Node script. You can check our assignment against your own implementation without asking us anything.
The guardrail admits what it is not
This is the part I want to hold up, because it is written in our own docs about our own feature rather than left for a prospect to discover:
There is no confidence interval, no bootstrap, and no significance test. The panel is a descriptive guardrail, not an inferential one.
So it reports a measurement. It does not prove a variation is safe.
The docs volunteer the consequence too. A difference just over 400 ms on 100 page loads per arm is not the same evidence as the same difference on 100,000. Same flag, very different weight.
The panel does not weight them for you. The page load count sits next to the row so that you can.
Four checks run before a row is judged at all, and they stop at the first failure:
- capture rate not above 100%
- at least 100 page loads on both arms
- a capture rate gap of no more than 20 percentage points
- at least 50% capture on each arm
Fail one and the row reads "collecting speed data" or "comparison unavailable", depending on which check it was, rather than guessing.
There is also no correction across devices or variations. A four variant test on two devices produces six comparisons, each judged on its own threshold. Six independent calls, not one corrected verdict.
That is the discipline a descriptive guardrail asks of you. Read the page load count behind a flag before acting on it, and read "not flagged" as nothing visible at this volume, rather than as clearance.
What to ask your own vendor
Not snippet size.
Ask for p75 time from navigation start to the variation landing in the DOM, separated by cached and uncached. Ask how long the original was actually painted. Ask for the method, and for the caveats.
If a vendor cannot produce that, the honest reading is that nobody there has measured it.
ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. The bucketing algorithm is documented at docs.abtestly.com and the prices are published at abtestly.com/pricing.
Top comments (0)