Every team running experiments has a rollback plan. Pause the test, ship the fix, move on.
Far fewer have a detection plan, which is the part that decides how long the broken thing was live before anyone hit the button.
A rollback you trigger on day nine is not a rollback. It is a post-mortem.
The release cadence changed under everyone
Chrome now ships a new Stable version every two weeks.
That is a two-week window between "this variation works" and "this variation runs on a browser that did not exist when it was written". Multiply by the other engines and their own cadences.
Most A/B test variations are DOM manipulation against selectors that were correct on the day they were authored. Nothing in that description survives contact with a fast release train automatically.
So the QA question stopped being "did it work when we built it" and became "how would we know if it stopped working".
What actually breaks, and when
The first version of a variation can look completely fine and still fail before the statistics ever run.
The failure modes are boringly consistent. A selector matches on first load and not after a client-side route change. A variation applies on desktop and silently no-ops on the mobile template. A goal fires on an element the variation replaced, so the exposure is recorded and the conversion never is.
None of those throw an error. None of them show up as a red thing on a dashboard. The test just quietly collects data that means nothing.
That is the detection gap. The variation is not down, it is wrong, and wrong looks identical to working until you check.
Editing a running test is the sharp edge
Here is the part most tools stay quiet about: a running test can be edited.
You found a bug in your variant code, you fix it, and now half your sample saw one thing and half saw another, reported as one number.
There are only two honest ways to handle that, and both of ours are confirmation-gated:
Pause, edit, resume. The clock resets on the results page and you get a new epoch. The old data is still there, it is just separated from what comes after.
Reset data. Discards the run so far and starts fresh with the updated code.
Reset is the heavier one, and it does more than clear counts. It rotates the experiment's bucketing salt, so visitors are assigned from scratch and someone who saw one variation may now see another.
That is the honest cost of a reset, and it is why it is not a shrug.
Two design decisions that follow from this
Material settings lock at 1,000 visitors.
Once a running experiment has 1,000 visitors since its last reset, six material settings become read-only in the editor:
Locked after 1,000 visitors. Pause the experiment to edit.
Pausing opens them again. The lock is not there to stop you, it is there to make the change deliberate, because changing traffic allocation mid-flight on a live test is how you manufacture a sample ratio mismatch.
The change notice does not go away.
If material settings changed during a run, the results page says so, and it keeps saying so until you reset the data.
That is deliberate. A notice that disappears once you have seen it is a notice that stops doing its job at the exact moment it matters, which is when a colleague reads the result three weeks later with no memory of what happened on day two.
Clearing it requires a reset, which advances the baseline so earlier changes fall outside the window. Nothing is deleted. The change events stay in the table, they just stop being in scope.
You cannot keep the numbers and lose the notice about them. That is the whole point.
Building an actual detection plan
Preview every variant on the live site before launching. Not in a staging clone, on the real page, because the thing that breaks is usually the interaction with production markup you did not write.
Watch the exposure count per arm in the first hour, not the conversion rate. Conversions are too sparse to tell you anything early. Exposures tell you immediately whether the variation is reaching people at the rate you expected.
Run the sample ratio check continuously, not at the end. An SRM that appears on day one is a bug you can still fix cheaply.
Decide in advance what number triggers a rollback, and write it down next to the hypothesis. "We will pause if exposures in variant B fall below 40% of control" is a detection plan. "We will keep an eye on it" is not.
And assume something will change underneath you in the next fortnight, because on the current release cadence, something will.
ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. The reset and locking behaviour is documented at docs.abtestly.com, and the prices are published at abtestly.com/pricing.
Top comments (0)