DEV Community: David Sert

Building a Test-and-Learn Framework for Experimentation Teams

David Sert — Fri, 21 Aug 2026 05:06:02 +0000

A "test and learn" culture is easy to declare in a strategy deck and hard to build in practice. Most teams that say it out loud already run some experiments — the gap is not activity, it's a framework that turns individual test results into a compounding body of knowledge instead of a pile of one-off wins and losses nobody revisits. This guide lays out what a test-and-learn framework actually consists of, how it differs from simply "running more tests," and how to wire it into the tools most experimentation teams already have.

What a test-and-learn framework is not

It is not a synonym for experiment velocity. A team that ships twenty tests a quarter but never looks back at test twelve when planning test twenty-one is not running a test-and-learn framework — it is running twenty disconnected trials. Velocity is an input to learning, not the same thing as it.

It is also not a tagline for "we're data-driven." That phrase describes a value, not a process. A framework has to specify who reviews a result, where the learning is recorded, how it gets surfaced to the next person planning a related test, and what happens when a result contradicts something the team believed six months ago. Without those mechanics, "test and learn" is aspirational language sitting on top of the same ad hoc process it was meant to replace.

The three loops a real framework closes

A working test-and-learn framework closes three separate feedback loops, and most teams that stall out have built only the first one.

The per-test loop. Hypothesis, launch, result, decision. This is the loop most teams already have, because it's the one a testing platform's dashboard makes visible by default: did the metric move, was it significant, do we ship it. It answers "was this specific idea right," and stops there.

The knowledge loop. This is the loop that's usually missing. A single test result becomes a durable, searchable finding — "urgency messaging on the PDP increased add-to-cart by 6% for mobile visitors, no effect on desktop" — that gets tagged, stored somewhere other than the testing tool's own history, and surfaces automatically when someone drafts a related hypothesis later. Without this loop, the same idea gets re-tested by a different team eighteen months later because nobody remembers, or could find, that it was already answered.

The prioritization loop. Findings from the knowledge loop should change what gets scored highly in your next planning cycle. If urgency messaging worked on mobile PDP, that's a signal the mechanism generalizes — it should raise the confidence score of related ideas in your prioritization framework, not just sit in an archive. A framework where confidence scores never update based on accumulated evidence is not actually learning from its own history.

Per-test loop:        hypothesis -> launch -> result -> ship/kill decision
Knowledge loop:        result -> tagged finding -> searchable archive -> surfaces on related hypotheses
Prioritization loop:   accumulated findings -> updated confidence scores -> next cycle's ranking

Most teams have built the first loop well and the other two barely at all. That imbalance is usually why "we've been testing for two years and don't feel any smarter about our users" is a common complaint even at teams with healthy experiment counts.

Where wins and losses both have to count

A framework that only stores winning tests is building a biased archive, and it will eventually mislead the team that relies on it. A losing test — the urgency messaging that moved nothing — is exactly as valuable a finding as a winning one, because it rules out a mechanism the team might otherwise retry with a different label six months later.

The discipline this requires is uncomfortable in a way that pure velocity is not: it means writing up the null results with the same rigor as the wins, including what confidence interval the null result actually had — a wide interval spanning zero means "inconclusive, worth retesting with more power," while a narrow interval spanning zero means "the mechanism doesn't work, stop proposing variants of it." Conflating those two kinds of null result is one of the most common ways a knowledge archive quietly degrades into noise; a genuinely closed loop keeps the distinction visible in the finding itself.

Where the framework lives operationally

The knowledge loop fails most often not because nobody wants to write things down, but because the writing happens somewhere nobody revisits — a slide in a quarterly deck, a Slack thread, a spreadsheet tab that gets abandoned after the person who made it changes teams.

If your experimentation platform supports it, the finding belongs as close to the test record itself as possible. Optimizely Feature Experimentation's native Hypothesis and Test plan fields on a rule are one option — they keep the original claim and the eventual result in the same place a future team member will actually look when they're deciding whether to relaunch something similar, rather than in a document they'd need to know exists. See the hypothesis template guide for how to structure the claim itself so the eventual finding is easy to compare against it.

Whatever storage you use, the test for whether it's actually part of the framework — rather than a compliance step nobody reads — is simple: when someone drafts a new hypothesis, does anything surface the related prior finding automatically, or does it depend on that person remembering to go looking? If it depends on memory, the knowledge loop isn't closed yet.

Building the review cadence

The prioritization loop needs a recurring point where accumulated findings actually change scores, not just a backlog grooming meeting that re-ranks the same list by gut feel. A simple, sustainable cadence:

After every test, the finding gets tagged with the mechanism it tested (not just the metric it moved) and stored where the next planner will see it.
Before every prioritization cycle, whoever scores confidence on new ideas is expected to check for related prior findings first — not from memory, but by searching the archive.
Quarterly, the team reviews which mechanisms have accumulated multiple confirming or disconfirming results, and retires ideas that keep failing the same way across different surfaces.

This cadence is intentionally light. A test-and-learn framework that requires a heavyweight ritual to maintain will get skipped the first time the team is busy, and a skipped framework teaches the same lesson as no framework at all.

Common mistakes

Measuring the framework by test count alone. A high-velocity team with no knowledge loop is not learning faster than a slower team that closes the loop — it's just accumulating more untracked history.
Archiving only wins. A knowledge base without documented losses will eventually recommend re-testing ideas that were already ruled out.
Storing findings somewhere disconnected from where hypotheses get written. If the archive requires a separate lookup step nobody remembers to take, it will not get used.
Letting confidence scores stay static regardless of accumulated evidence. If a mechanism has been confirmed three times across different surfaces, related ideas should score higher in prioritization than they did before any of that evidence existed.

Key takeaways

A test-and-learn framework is not the same as experiment velocity — it requires a knowledge loop and a prioritization loop on top of the basic per-test loop most teams already have.
Null results need the same documentation discipline as wins, including whether the confidence interval was wide (inconclusive) or narrow (the mechanism doesn't work).
Findings belong as close to the original hypothesis as operationally possible, so the next planner encounters them without having to remember to look.
The real test of a closed loop is whether accumulated findings change prioritization scores over time, not whether a slide exists documenting past results.

Frequently asked questions

How is a test-and-learn framework different from just running more A/B tests?

Running more tests closes only the per-test loop — hypothesis to result to ship/kill decision. A framework additionally closes a knowledge loop, where each result becomes a durable, searchable finding, and a prioritization loop, where accumulated findings change how future ideas get scored.

Should losing tests be documented as thoroughly as winning tests?

Yes, and with more care about the confidence interval specifically. A win is straightforward to record; a loss needs to distinguish between "inconclusive — the interval was wide" and "the mechanism doesn't work — the interval was narrow around zero," because those two outcomes should lead to different next steps.

Where should test findings be stored so they actually get reused?

As close to the point where the next hypothesis gets written as possible. If your platform supports native hypothesis and result fields on the experiment record itself, that beats a separate document, because it removes the step of remembering a separate archive exists.

Post Hoc Power Analysis: Why Recalculating Power After a Null Result Fails

David Sert — Fri, 21 Aug 2026 05:05:26 +0000

An A/B test comes back inconclusive — no significant difference between variation and control — and someone on the team asks a reasonable-sounding question: "was the test even powered enough to detect a real effect?" The instinct is to run a power calculation using the effect size the test actually observed, plug it back into a power formula, and report the number. That calculation has a name, post hoc power analysis, and a well-documented problem: it is mathematically redundant with the p-value you already have, and treating it as new evidence is one of the most common ways a null result gets quietly reinterpreted as "probably a win we couldn't detect." This guide covers why the calculation doesn't do what it appears to do, what to check instead, and how to size a test correctly the first time so you never need the retrospective version.

What post hoc power analysis actually calculates

Prospective power — the kind you calculate before a test launches — asks: "if the true effect is X, and I collect N visitors, what is the probability I detect it as significant?" That calculation depends on three things you choose in advance: the minimum effect worth detecting, your sample size, and your significance threshold. It is a planning tool, and it is exactly what a sample size calculation is for.

Post hoc power analysis takes the same formula and swaps one input: instead of the minimum detectable effect you decided was worth chasing, it plugs in the effect size the experiment actually observed. Then it reports the probability that a test of this size would have detected an effect of this exact magnitude — the one already sitting in your results.

Why the observed-effect version is circular

The problem is not the arithmetic; it is what the output means. Prospective power tells you something you did not already know — whether your planned sample size is big enough for an effect you care about. Post hoc power computed from the observed effect tells you nothing you cannot already read off the p-value, because the two are mathematically tied together: for a fixed sample size, post hoc power is a strictly decreasing function of the p-value. A high p-value (a null result) will always produce a low post hoc power calculation, and a low p-value (a significant result) will always produce a high one. The calculation cannot come back and say "actually, this null result had high power" — that combination is not mathematically possible.

That means a stakeholder who asks for post hoc power on a null result and receives a low number has learned nothing beyond "the result was not significant," restated in a more technical-sounding unit. The number feels like independent confirmation that "we just needed more data," but it is an algebraic reflection of the p-value you already had, not a second measurement.

The retrospective-power fallacy in practice

The fallacy shows up in a specific, recognizable pattern: a test ends without significance, the team is reluctant to call the idea a loss, and a post hoc power calculation comes back low — say, 40%. That 40% gets reported as "the test was underpowered," which sounds like a design flaw rather than a null result, and the conclusion drawn is "the effect is probably real, we just need to run it longer or on more traffic."

That conclusion does not follow from the number. A low post hoc power figure is a restatement of "not significant," and it carries no information about whether the true effect exists at all. The actual data available for that judgment is the confidence interval around the observed effect, not a recomputed power figure — and the confidence interval is the tool that should be doing this work.

What to check instead of retrospective power

If a null result raises the question of whether the test just needed more data, the honest way to answer it is to look at the confidence interval around the effect estimate, not to recompute power.

A wide confidence interval that spans zero — say, a 95% CI of −1.5% to +4.0% on relative lift — genuinely is consistent with "we don't know yet, and a bigger sample might resolve it." That is real information: the interval is telling you the experiment did not have enough precision to distinguish a meaningful effect from no effect. This is functionally the same conclusion a well-designed prospective power calculation would have flagged before you ever launched, which is why sizing the test properly the first time — using an honestly chosen minimum detectable effect — avoids the whole situation.

A narrow confidence interval that spans zero — say, −0.3% to +0.4% — tells a different story: the test had plenty of precision, and it precisely located the effect at approximately zero. Rerunning that test with more traffic will not turn up a real effect, because the data already ruled out anything but a trivial one.

Wide CI spanning zero:    [-1.5%, +4.0%]  -> underpowered for this effect size; more data could resolve it
Narrow CI spanning zero:  [-0.3%, +0.4%]  -> well-powered; the effect is precisely estimated near zero

Both examples are "not significant." Only the confidence interval tells you which kind of "not significant" you are looking at, and that distinction is the actual answer to the question post hoc power analysis is being asked to provide.

How this connects to peeking and effect size

Post hoc power analysis is a close cousin of two other retrospective mistakes covered elsewhere on this site. Effect size reporting already explains why comparing the observed lift against your original MDE — rather than recalculating anything — is the correct way to judge whether a result was meaningfully sized. And the temptation to keep collecting data until the picture improves is exactly the optional-stopping problem: if you extend a test because a retrospective power number felt low, you are running an unplanned, uncontrolled extension of an experiment whose original stopping rule you have already broken.

The common thread across all three is the same: decisions made by looking backward at data you already have, dressed up as if they were new evidence, inflate your real error rate without showing up anywhere in the reported statistics.

When a retrospective power calculation is legitimate

There is one legitimate retrospective use of a power calculation, and it does not use the observed effect size at all. Before deciding whether to run a follow-up experiment, you can calculate what sample size would be needed to detect a specific, externally chosen effect size — for example, the smallest lift that would be worth the engineering cost of shipping the change permanently. That is a forward-looking calculation about a new, hypothetical test, using an effect size you chose for a reason unrelated to what the last test happened to observe. It looks similar to post hoc power analysis on the surface, but the input is a decision, not a readout, which is what keeps it from being circular.

Common mistakes

Reporting post hoc power as if it were independent evidence. It is a deterministic function of the p-value you already had; it does not add information.
Using the observed effect size as the input, then acting surprised the number is low on a null result. That outcome is guaranteed by the math, not a discovery about the experiment.
Extending a test's runtime because a retrospective power figure "felt low." This breaks your stopping rule and reintroduces the peeking problem under a different name.
Skipping a confidence-interval read entirely. The CI is doing the real diagnostic work a post hoc power calculation is being asked to do — check it first.

Key takeaways

Post hoc power analysis recomputes power using the effect size a test actually observed, and that number is mathematically tied to the p-value you already have — it is not new evidence.
A null result will always produce a low post hoc power figure; that is guaranteed by the algebra, not a finding about whether the effect is real.
The confidence interval around the observed effect — not a recalculated power figure — is the correct tool for judging whether a null result reflects "no effect" or "not enough precision to tell yet."
The fix that avoids the question entirely is sizing the test honestly before it launches, using a minimum detectable effect you actually intend to act on.

Frequently asked questions

Is post hoc power analysis ever useful?

Only when it uses an externally chosen effect size to plan a genuinely new test — for example, sizing a follow-up experiment around the smallest lift worth shipping. Using the just-observed effect size from the test you're analyzing is the circular version to avoid.

My test came back with low post hoc power. Does that mean I should run it longer?

Not on its own. Look at the confidence interval around the observed effect instead. A wide interval spanning zero suggests genuine imprecision that more data could resolve; a narrow interval spanning zero means the effect is precisely estimated at approximately zero, and more data will not change that conclusion.

How is this different from a normal, prospective power calculation?

A prospective power calculation, done before the test launches, uses an effect size you chose because it is the smallest one worth detecting — that is what a sample size calculator does. Post hoc power analysis instead plugs in the effect size the test happened to observe, which makes the output a restatement of the p-value rather than independent planning information.

One-Tailed vs Two-Tailed A/B Tests: Which Should You Run

David Sert — Fri, 21 Aug 2026 05:04:51 +0000

Every hypothesis makes a prediction about direction — "this will increase revenue," not "this will change revenue somehow." That prediction has a statistical consequence most teams never examine: whether the test behind it should be one-tailed or two-tailed. Get this choice wrong, or worse, make it after the data arrives, and you can turn a 5% false-positive rate into something closer to 10% without changing a single line of test code. This guide covers what each test actually claims, why Optimizely's Stats Engine defaults to two-tailed, the narrow cases where one-tailed is legitimate, and the much more common case where it is p-hacking wearing a statistics textbook's clothes.

What a two-tailed test actually claims

A two-tailed test's null hypothesis is that the variation and the control produce the same result. Its alternative hypothesis is simply that they differ — in either direction. The test's rejection region is split across both tails of the distribution: half the alpha budget covers "variation beats control," half covers "control beats variation." At a 95% confidence level, that means roughly 2.5% of the probability mass sits in each tail.

This is the conservative, symmetric default, and it is conservative on purpose. It treats "the button color change made things worse" as just as real and just as reportable a finding as "the button color change made things better." A two-tailed test does not presuppose which way the world will move; it only asks whether it moved at all.

What a one-tailed test actually claims

A one-tailed test narrows the alternative hypothesis to a single direction — "the variation is better than control," full stop. The rejection region lives entirely in one tail, so the same 5% alpha budget that was split across two tails in the two-tailed case is now concentrated in one. Practically, that means a one-tailed test reaches significance with a smaller observed effect than a two-tailed test would need, for the identical data and the identical alpha.

That extra sensitivity is the entire appeal, and it is also the entire danger. A one-tailed test is a bet: you are declaring, before you see any data, that a result in the "wrong" direction is not just unlikely but uninteresting — that you would take no action on it either way. If that premise is false, the test is miscalibrated, because it is silently discarding evidence of harm as if it could not exist.

Why Optimizely's Stats Engine defaults to two-tailed

Optimizely's Stats Engine runs two-tailed tests by design, and that is not an arbitrary default — it is required for the false discovery rate control the engine is built around. FDR control needs symmetric tails to keep its error-rate guarantee honest across every metric in a test, including the ones you did not expect to move. A one-tailed override would quietly break that guarantee for the metric it was applied to.

There is a practical reason to like the default even if you ignore the FDR mechanics: most experiments are run precisely because you do not know which way a change will move the metric. If you already knew the direction with certainty, you would not need to test it. Two-tailed testing keeps the platform honest about that uncertainty instead of assuming it away.

When a one-tailed test is legitimate

There is a narrow, defensible case for a one-tailed test, and it has nothing to do with statistical power. It applies only when a result in the "wrong" direction is genuinely inactionable — when you would ship the same decision regardless of whether the metric moved down or stayed flat, because either way the answer is "do not ship."

A migration or performance-safety check is the clearest example: you are testing whether a new checkout implementation is at least as fast as the old one before a full rollout you already intend to do for other reasons. A result showing it is dramatically faster changes nothing about the decision; a result showing it is the same speed changes nothing either. Only "meaningfully slower" would change what you do next, so the interesting region really is one-sided.

Outside cases like that, a one-tailed test is difficult to justify honestly, because most product and marketing decisions are exactly the kind where a surprising result in the "wrong" direction would change what you do.

The failure mode: switching tails after you've seen the data

The far more common way a one-tailed test shows up in practice is not principled design — it is a rescue attempt. A two-tailed test comes back with a p-value of 0.08, just short of significance. Someone notices the effect is in the "expected" direction and proposes re-running the analysis as one-tailed, which roughly halves the p-value and pushes it under 0.05.

This is p-hacking with a statistical vocabulary. The test's design was chosen after seeing the outcome it was meant to be judged against, which means the "5% error rate" claimed for the result was never actually in effect — the true error rate for a decision rule of "run two-tailed, and switch to one-tailed if it doesn't reach significance" is close to double the stated alpha, because you get two chances to declare a win from the same data. The fix is procedural, not statistical: decide the tail before the test launches, write it into the hypothesis, and do not revisit the decision once results are in.

Worked example: the same data, two verdicts

The table below shows why the distinction is not academic. Both rows use the identical observed data — a 4.2% relative lift with a two-sided p-value of 0.062 — and differ only in which test was declared before the experiment ran.

Declared before launch:  Two-tailed
Observed lift:           +4.2%
Two-sided p-value:       0.062
Verdict at alpha=0.05:   Not significant

Declared before launch:  One-tailed ("variation > control")
Observed lift:           +4.2%
One-sided p-value:       0.031
Verdict at alpha=0.05:   Significant

Nothing about the underlying data changed between the two rows — only the pre-registered claim did. That is exactly why the decision has to be made before launch and recorded somewhere a reviewer can check it, rather than chosen after the fact to produce the answer you wanted.

How this interacts with your stopping rule

Tail choice and stopping rule are two separate decisions that compound if you get both wrong. A one-tailed test that is also peeked at repeatedly during the run — checked daily, stopped the moment it crosses significance — stacks a directional shortcut on top of the optional-stopping problem, and the two inflate the same underlying quantity: how often you declare a winner that is not real. Fix the stopping rule with a proper sequential method or a fixed horizon, and fix the tail by writing it into the hypothesis before you launch. Neither substitutes for the other.

Common mistakes

Deciding the tail after seeing a "close" result. This is the single most common misuse, and it is invisible in a results dashboard — the platform cannot tell the difference between a principled one-tailed test and a rescued one, because the giveaway is when the decision was made, not what number came out of it.
Treating "one-tailed" as a way to get more power for free. It is not free. You are trading the ability to detect and act on harm in the opposite direction for a smaller required effect size. That trade is sometimes worth making, but it is a trade, not a discount.
Applying a one-tailed override to only the primary metric. If a guardrail metric moves sharply in the "wrong" direction, a one-tailed primary-metric test gives you no formal signal to catch it — you are relying entirely on eyeballing the guardrail dashboard.
Assuming Stats Engine has a one-tailed mode to switch on. It does not, by design; the FDR guarantee depends on the two-tailed default holding for every metric in the experiment.

Key takeaways

A two-tailed test asks whether variation and control differ at all; a one-tailed test asks only whether variation is better, and gives up the ability to detect "worse" as a formal result.
Optimizely's Stats Engine runs two-tailed tests by design — it is a requirement of the false discovery rate control the engine provides, not an arbitrary setting.
A one-tailed test is only legitimate when a result in the unexpected direction is genuinely inactionable, decided and documented before the experiment launches.
Choosing the tail after seeing the data — even informally, even just noticing the direction matches your prediction — silently doubles your real false-positive rate relative to the one you're reporting.

Frequently asked questions

Can I run a one-tailed test in Optimizely Web Experimentation or Feature Experimentation?

Stats Engine's significance calculation is two-tailed by design across both products, as part of its false discovery rate guarantee. You can still write a directional hypothesis and interpret a two-tailed result through that lens at the analysis stage — you are simply not changing the underlying test statistic to do it.

Is a one-tailed test ever more "correct" than a two-tailed test?

Neither is more correct in the abstract; they answer different questions. A one-tailed test is the right tool only when a result in the unexpected direction is truly inactionable and that has been decided in advance. Outside that narrow case, a two-tailed test is the safer default because it does not assume away the possibility of harm.

Why does a one-tailed test need a smaller effect to reach significance?

Because the same alpha budget is concentrated in one tail of the distribution instead of split across two, so a smaller observed deviation is enough to cross the threshold. That is the source of the extra sensitivity — and also the source of the risk, since it comes from narrowing what counts as evidence, not from collecting more of it.

Google Content Experiments vs Optimizely: What Actually Changed

David Sert — Fri, 21 Aug 2026 05:04:15 +0000

Google Content Experiments is the testing tool most people searching for it are not actually thinking of anymore, and that confusion is worth clearing up before comparing anything else. It was Google's original multivariate and A/B testing feature, built into classic Google Analytics, and Google deprecated it back in 2017 — years before Google Optimize, its more famous successor, was retired in 2023. If your team's testing history goes back far enough to remember Content Experiments by name, or you've simply found old documentation referencing it while researching a move to Optimizely, this guide covers what it actually was, how it differs from Optimizely, and which of two entirely different migration paths applies to you.

What Google Content Experiments actually was

Content Experiments shipped as a feature of classic Google Analytics, reachable from the Behavior section of the Analytics interface. It let you test up to ten full-page variations against a control, using a simple redirect-based mechanism: visitors were routed to one of several distinct URLs, and Analytics tracked which URL produced better performance against a chosen objective. It replaced an even earlier tool, Google Website Optimizer, and it was itself replaced by Google Optimize in 2017 — Optimize added the visual editor, more targeting options, and eventually a Google Analytics 4 integration that Content Experiments never had.

That lineage matters because it explains why searches for "Content Experiments" and searches for "Google Optimize" turn up overlapping but not identical results, and why some migration guidance written for one does not cleanly apply to the other.

How Content Experiments compares to Optimizely

The comparison is less about feature parity and more about what generation of testing tool each one represents.

Content Experiments was redirect-only: every "variation" was a distinct, separately hosted page, and Analytics split traffic across the URLs and reported which one won on the metric you configured. It had no visual editor, no client-side DOM manipulation, and no concept of audience targeting beyond what you could encode into which visitors received which redirect.

Optimizely Web Experimentation supports that same redirect-based pattern for full-page and URL-based tests, but it is one option among several rather than the only one. It also supports client-side DOM changes through a visual editor without requiring a second hosted page, audience targeting rules that go well beyond a simple traffic split, and integration with Optimizely's Stats Engine for significance testing that accounts for multiple metrics and repeated looks at the data — none of which Content Experiments offered.

Google Content Experiments (2012-2017):  redirect-only, up to 10 variations, basic Analytics objective, no visual editor
Optimizely Web Experimentation (today):  redirect-based OR visual-editor DOM changes, audience targeting, Stats Engine significance

For a team that only ever used Content Experiments for simple landing-page redirect tests, the redirect-based workflow in Optimizely — covered in the landing page A/B testing guide — is the closest like-for-like starting point, even though the platform underneath it is considerably more capable.

Two different migration paths, depending on where you're coming from

If your team's most recent testing tool was Content Experiments itself — meaning you never adopted Google Optimize before it, too, was retired — you are migrating across two deprecations at once, and there is no automated importer for either leg of that journey. Google never published an export format for live Content Experiments configurations, and the platforms differ enough structurally that a mechanical translation would not produce working tests regardless. The realistic path is the same structured rebuild any migration off a legacy Google testing tool requires: inventory what you were testing, decide whether each experiment maps most naturally to a redirect-based test or a visual-editor DOM change in Optimizely, and rebuild from there.

If instead you moved from Content Experiments to Google Optimize at some point before Optimize's 2023 sunset, you are migrating from Optimize, not from Content Experiments — and that is a more common, better-documented path. The Google Optimize to Optimizely migration guide covers that transition in full: mapping Optimize's objectives, audiences, and visual-editor changes to their Optimizely equivalents, and what does not carry over automatically.

Why there's no automated importer for either tool

The absence of a migration tool is not an Optimizely limitation specifically — it reflects a gap upstream. Google never published a machine-readable export format for either Content Experiments or Optimize configurations, so no third-party platform, Optimizely included, has anything to build an importer against. Even if one existed, the underlying models differ enough — Content Experiments' pure-redirect approach versus Optimizely's combination of visual-editor DOM changes and redirects, different audience-targeting concepts, different significance calculations — that a mechanical translation would likely produce broken or subtly wrong tests rather than a clean lift-and-shift.

That makes the rebuild a deliberate, if mechanical, exercise rather than a data-migration problem: most concepts from either legacy tool have a clear Optimizely counterpart, so the work is mapping and recreating rather than inventing a new testing program from scratch.

What actually carries over conceptually

Even without an automated path, the underlying testing logic from a Content Experiments era program is not wasted. A redirect-based page comparison — page A's URL against page B's URL, split by traffic — maps directly onto a URL-redirect test in Optimizely Web Experimentation, and any historical learnings about which page structure won are still valid evidence worth feeding into your next hypothesis, even though the original test predates the platform you're rebuilding in.

What does not carry over is the objective configuration itself — Content Experiments' Analytics-goal-based objectives have no direct equivalent field in Optimizely, since Optimizely defines primary and guardrail metrics natively rather than borrowing them from an Analytics goals configuration. Those need to be redefined as part of the rebuild, not imported.

Common mistakes

Assuming "Google Optimize migration" guidance covers Content Experiments too. The two tools have different underlying mechanics — Content Experiments is redirect-only with no visual editor — so guidance written for an Optimize migration will skip steps a Content Experiments migration actually needs.
Looking for an export file that does not exist. Neither Content Experiments nor Optimize ever published one; budget time for a structured rebuild rather than searching for an importer.
Treating old Content Experiments results as historical noise instead of evidence. A redirect-test result from years ago is still a real finding about what worked on that page — it belongs in your knowledge base, not the trash, even if the tooling that produced it is long gone.
Recreating every legacy test exactly as it ran. Some Content Experiments-era tests existed because the tool had no visual-editor option; in Optimizely, several of those may be better redesigned as DOM-change tests rather than reproduced as redirects out of habit.

Key takeaways

Google Content Experiments (deprecated 2017) and Google Optimize (deprecated 2023) are two different, sequential Google testing tools — check which one your team actually used before following migration guidance.
Content Experiments was redirect-only with no visual editor; Optimizely supports that same pattern plus client-side DOM changes, audience targeting, and Stats Engine significance testing.
Neither Google tool ever published an export format, so the migration path for either is a structured rebuild — inventory, map concepts, recreate — not a data import.
If your path ran through Google Optimize before Optimizely, the dedicated Google Optimize migration guide is the more specific resource to follow.

Frequently asked questions

Is Google Content Experiments the same thing as Google Optimize?

No. Content Experiments was Google's earlier testing tool, built into classic Google Analytics and deprecated in 2017. Google Optimize was its successor, with a visual editor and GA4 integration, and was itself retired in 2023. They are two different products in the same lineage.

Can I import my old Content Experiments configuration directly into Optimizely?

No automated importer exists, because Google never published an export format for either Content Experiments or Google Optimize. The migration is a structured rebuild: map each concept to its Optimizely equivalent and recreate the experiments you still want to run.

Which Optimizely product replaces what Content Experiments did?

Optimizely Web Experimentation covers the same redirect-based, full-page comparison pattern Content Experiments used, and adds a visual editor, audience targeting, and Stats Engine significance testing on top of it. See the landing page A/B testing guide for the closest equivalent workflow.

Champion/Challenger Testing vs A/B Testing: When the Older Method Still Wins

David Sert — Fri, 21 Aug 2026 05:03:39 +0000

Champion/challenger testing predates A/B testing as most web teams practice it today, and it never went away — it just moved into places like fraud scoring, credit models, and recommendation systems, where it is still the default. It resurfaces in web and product experimentation whenever a team needs something an A/B test does not naturally give them: a live, defensible incumbent that keeps running while a challenger earns its way past it, one comparison at a time. This guide covers what champion/challenger actually is, how it differs from a standard A/B test and from a multi-armed bandit, and when reaching for it is the right call instead of a reflex from a different industry.

What champion/challenger actually means

In a champion/challenger setup, there is always a "champion" — the current best-known approach, running in production and serving the majority of traffic — and one or more "challengers," each getting a smaller, deliberately limited slice. A challenger that beats the champion by a pre-agreed margin gets promoted to be the new champion. One that doesn't gets retired, and a new challenger takes its slot.

The defining feature is that there is no fixed end date and no single terminal readout. The champion is always the reigning answer to "what do we run right now," and the question the process answers is not "which of these two options is better," but "is there currently anything that beats what we're running, and if so, promote it." That framing is closer to a standing tournament than to a bounded scientific test.

How this differs from a standard A/B test

A standard A/B test is symmetric and time-bounded: control and variation split traffic roughly evenly, the test runs until it reaches a pre-computed sample size, and it ends with a single decision — ship the variation, keep the control, or call it inconclusive. Once the decision is made, the test is over. There is no "champion" that persists into the next test by default; each new test starts from a fresh 50/50 split unless you deliberately structure it as a sequel.

Champion/challenger drops the symmetry and the fixed endpoint on purpose. The champion typically gets the large majority of traffic — commonly 80-90% or more — because it is a known quantity you don't want to put unnecessarily at risk, while the challenger gets just enough to accumulate evidence without meaningfully denting overall performance if it turns out to be worse. And rather than one comparison with a hard stop, the process continues indefinitely: today's promoted challenger becomes tomorrow's champion, facing the next challenger in line.

A/B test:                control (50%) vs variation (50%) -> fixed sample size -> one decision -> test ends
Champion/challenger:     champion (80-90%) vs challenger (10-20%) -> ongoing -> promote or retire -> repeats indefinitely

How this differs from a multi-armed bandit

Champion/challenger and multi-armed bandit allocation look similar on the surface — both give the current best option more traffic than the alternatives — but the mechanism and the guarantee behind them are different.

A bandit reallocates traffic continuously and automatically based on observed performance, adjusting the split many times over a run in pursuit of maximizing cumulative reward across the whole traffic stream. It optimizes for earning during the test itself, and it explicitly trades away the clean significance machinery of a fixed-split test to do it — see the fuller comparison in Stats Accelerator vs MAB vs contextual bandit for the mechanics.

Champion/challenger's traffic split is typically set once by a person, not continuously re-optimized by an algorithm, and the promotion decision at the end is usually a discrete, human-reviewed comparison against a stated threshold rather than a live reallocation curve. It trades the bandit's automatic optimization for something a bandit does not offer: an easily explainable, auditable record of exactly when and why each promotion happened — which matters a great deal in domains like credit and fraud decisioning, where a regulator or an internal risk committee needs to be able to reconstruct the reasoning behind a model change.

When champion/challenger is the right choice

The clearest case for champion/challenger over a standard A/B test is when there genuinely is no natural end date to the comparison — recommendation ranking, fraud-scoring thresholds, and pricing models are all domains where "the current best model" is a permanent, evolving role rather than a question with a final answer. In those settings, running a fresh symmetric A/B test for every candidate improvement means repeatedly re-litigating a decision that champion/challenger treats as continuous by design.

It is also the better fit when the cost of a wrong answer at scale is asymmetric and high, and you want the safety of keeping the bulk of traffic on a known-good option while a challenger proves itself gradually rather than immediately splitting evenly. A fraud-scoring model that gets it wrong on 50% of transactions during a symmetric test is a materially worse outcome than one that gets it wrong on 10-20% while a challenger is evaluated.

And it fits a governance requirement an A/B test's single readout does not naturally satisfy: an auditable trail of exactly which model was live when, and what threshold each promotion cleared. Web experimentation platforms report a result; champion/challenger produces a lineage.

When it is the wrong choice

For a typical web or product experiment — a landing page redesign, a checkout flow change, a new onboarding sequence — a standard A/B test is almost always the better tool, not the weaker one. These changes usually do have a natural end state: you either ship the winner or you don't, and there is no ongoing "model" that needs a permanent incumbent role. Reaching for champion/challenger here mostly adds complexity — an indefinite runtime, an ambiguous promotion threshold, traffic sitting on a "challenger" slice long after a fixed-horizon test would have already reached a clean answer — without adding anything the situation actually needed.

It is also the wrong tool when you need the formal significance guarantees a platform like Optimizely's Stats Engine provides. Because champion/challenger's traffic split and promotion threshold are typically set by policy rather than derived from a significance calculation, it does not carry the same false-discovery-rate guarantee a properly sized, symmetric test does. If your organization needs a defensible "we know this is real" answer for a single, bounded decision, that is what a standard test is built to provide.

A decision framework

Ask three questions before choosing between the two:

Is there a natural end date to this comparison, or is "the current best option" a permanent, evolving role? A permanent role points toward champion/challenger; a bounded decision points toward a standard test.
Do you need a rigorous significance guarantee for a single decision, or an ongoing, auditable lineage of incremental promotions? The former points toward a fixed-horizon A/B test; the latter points toward champion/challenger.
Is the downside of a bad challenger large enough that you want it capped at a small traffic share indefinitely, rather than accepted at 50% for a bounded window? If yes, that argues for champion/challenger's asymmetric, ongoing split over a standard test's symmetric one.

Common mistakes

Using champion/challenger for a one-off decision with a real end state. This just adds indefinite runtime and an ambiguous stopping point to a question that a fixed-horizon test would answer more cleanly.
Confusing it with a bandit and expecting automatic reallocation. Champion/challenger's split is typically fixed by policy; if you want continuous, algorithmic reallocation toward the better-performing option, that is a bandit, not this.
Skipping a stated promotion threshold. Without one, "is the challenger good enough" becomes a judgment call made after the fact, which reopens exactly the kind of post-hoc reasoning a designed comparison is supposed to prevent.
Applying it to web page tests out of habit from a fraud or risk background. The governance need that makes champion/challenger valuable in those domains usually isn't present for a landing page test, and the standard A/B test's cleaner significance guarantee is the better fit.

Key takeaways

Champion/challenger keeps an incumbent "champion" running on most traffic indefinitely, promoting a challenger only when it clears a pre-agreed margin — there is no fixed end date and no single terminal readout.
It differs from a bandit in that the traffic split and promotion decision are typically set by policy rather than continuously optimized by an algorithm, trading automatic reallocation for an auditable lineage.
It fits domains with a permanent "current best" role and asymmetric downside risk — fraud, credit, ranking — better than it fits a typical web experiment with a real end state.
For a bounded product decision, a standard A/B test with a proper significance guarantee is almost always the better tool.

Frequently asked questions

Is champion/challenger testing the same as an A/B test with unequal traffic splits?

Not quite. An unequal split alone doesn't capture the defining feature, which is the lack of a fixed end date — champion/challenger is an ongoing process where today's winner becomes tomorrow's incumbent, not a single bounded comparison with an unusual split.

Does Optimizely support champion/challenger testing directly?

Optimizely's Stats Engine is built around bounded, significance-driven comparisons rather than an indefinite promotion lineage. Teams that need champion/challenger's governance model typically build it as a repeated series of fixed-horizon tests with a documented promotion rule between them, rather than relying on a single ongoing experiment configuration.

How is champion/challenger different from a multi-armed bandit in practice?

A bandit continuously reallocates traffic toward whichever arm is performing best, optimizing for cumulative reward during the run itself. Champion/challenger's split is typically fixed by policy, and the promotion decision is a discrete, auditable comparison against a stated threshold rather than a live reallocation curve.

Sample Ratio Mismatch: Is Your A/B Test Broken?

David Sert — Thu, 20 Aug 2026 05:10:45 +0000

Your experiment finished, the dashboard shows a winner, and something still feels off. The traffic split you set to 50/50 is sitting at 47/53. Before you write up the result or ship the change, you need to answer one question: is the split you observed close enough to the split you designed, or is the test itself broken? That question has a precise, statistical answer, and getting it wrong is one of the most common ways a team ships a change based on a number that was never real. This is sample ratio mismatch, and it is the first thing a senior experimenter checks before looking at any lift.

What sample ratio mismatch is

A sample ratio mismatch (SRM) occurs when the number of visitors actually assigned to each variation differs from the split you intended by more than random chance can explain. You designed a 50/50 test; the engine delivered something that a fair coin, flipped that many times, would essentially never produce.

The important word is chance. You should never expect a perfectly even split. If you assign 20,000 visitors with a fair 50/50 rule, landing on exactly 10,000/10,000 is rare — the count wobbles around the target every time, the same way 10,000 coin flips almost never give exactly 5,000 heads. SRM is not "the split isn't perfect." SRM is "the split is so far off that randomness is no longer a believable explanation."

That distinction is why you cannot judge SRM by eye. Looking at 47/53 tells you the direction and size of the gap but nothing about how surprising it is, because surprise depends on how many visitors produced that gap. The same 47/53 is unremarkable at 400 visitors and a five-alarm fire at 400,000. You need a test that accounts for sample size. The tool for that is the chi-square goodness-of-fit test, and OptiPilot's free SRM Checker runs it for you — paste in your observed counts and expected split, and it returns the p-value and verdict.

Why an SRM invalidates your results

Randomized assignment is the entire reason an A/B test can claim causality. When you split traffic randomly, the two groups are, on average, identical in every way — same mix of devices, same proportion of loyal customers, same intent — except the change you are testing. That balance is what lets you attribute any difference in outcomes to the variation rather than to a difference in who was in each group.

An SRM is direct evidence that the randomization did not work as designed. If a mechanism pushed extra visitors into one variation, that same mechanism almost certainly pushed a non-random kind of visitor. Redirects that time out drop the impatient and the slow-connection users. Bot filters that fire unevenly strip out a specific traffic source. In every case the groups are no longer comparable, so the lift you measured is a mix of the real treatment effect and a selection bias you cannot separate out.

This is the point that trips up teams under deadline pressure: you cannot adjust your way out of an SRM. There is no correction factor, no reweighting, no "the split was 45/55 so I'll scale the numbers." You do not know which visitors are missing or why, so you cannot model the bias. An SRM means the data is untrustworthy at the source. The only valid responses are to find and fix the cause, then rerun — or, if you can prove the cause is benign and non-differential, to document exactly why. Reading the Optimizely results page is meaningless until the split checks out.

How SRM is detected with a chi-square test

The standard test is Pearson's chi-square goodness-of-fit. It compares the visitor counts you observed in each variation against the counts you expected under your intended split, and returns a p-value: the probability of seeing a gap at least this large if assignment were truly random.

The statistic is:

chi-square = sum over each variation of (observed - expected)^2 / expected

degrees of freedom (df) = number of variations - 1

For a two-arm test, df = 1. For a three-arm 34/33/33 test, df = 2, and so on. You compute the statistic, look up the p-value for that value and df, and compare it to a threshold.

The industry alarm threshold is p < 0.001. If the p-value falls below it, you treat the experiment as having an SRM and stop trusting the results until you find the cause. Some teams run stricter: Microsoft's experimentation platform uses 0.0005, and a few use 0.0001. The threshold is deliberately far tighter than the 0.05 you use for a metric result, for two reasons. First, you run this check on every experiment, so a loose threshold would drown you in false alarms. Second, SRM is a binary data-integrity gate, not an effect you are trying to measure — you want it to fire only when something is genuinely wrong. Note that a real SRM usually produces a p-value orders of magnitude below any of these thresholds, not a marginal miss.

A worked chi-square example

Take a 50/50 test. You intended an even split; you observed 10,200 visitors in A and 9,800 in B, for 20,000 total. Under a fair split you expect 10,000 in each.

Observed:  A = 10,200   B = 9,800     Total = 20,000
Expected:  A = 10,000   B = 10,000    (50% of 20,000 each)

chi-square = (10,200 - 10,000)^2 / 10,000 + (9,800 - 10,000)^2 / 10,000
           = (200)^2 / 10,000 + (-200)^2 / 10,000
           = 40,000 / 10,000 + 40,000 / 10,000
           = 4 + 4
           = 8.0

df = 2 variations - 1 = 1
p-value for chi-square = 8.0, df = 1  ->  p ~= 0.0047

How to read that: p ≈ 0.0047 means a gap this large would happen about 5 times in 1,000 by pure chance. That clears the conventional 0.05 bar, so a naive check would panic. But it is above the 0.001 SRM threshold, so by the standard you do not declare an SRM. It sits in the uncomfortable middle — worth a glance, not worth halting for. This is exactly why the strict threshold exists: a 2% wobble at this sample size is not yet strong enough evidence of a broken split.

Now keep the same 51/49 ratio but run 10x the traffic:

Observed:  A = 102,000   B = 98,000    Total = 200,000
Expected:  A = 100,000   B = 100,000

chi-square = (2,000)^2 / 100,000 + (-2,000)^2 / 100,000
           = 40 + 40
           = 80.0

df = 1
p-value for chi-square = 80.0, df = 1  ->  p ~= 3.7e-19  (effectively zero)

Identical proportions, wildly different verdict. At 20,000 visitors the 51/49 split is plausibly noise; at 200,000 it is a certainty that something is wrong. This is the single most important intuition about SRM: the ratio alone tells you nothing — the mismatch is a function of the ratio and the sample size together. It is also why staring at the percentages on a dashboard will never substitute for the test.

What causes sample ratio mismatch

An SRM is a symptom, like a fever — it tells you something is wrong without telling you what. The KDD 2019 taxonomy from Microsoft and Booking.com researchers organizes the causes by where in the pipeline the visitor count gets distorted. Walking that pipeline is how you narrow the search.

flowchart LR
    A[Assignment] --> B[Exposure]
    B --> C[Logging]
    C --> D[Analysis]
    A -.->|"faulty hash, bad IDs,<br/>uneven ramp, carryover"| A
    B -.->|"redirect timeouts, flicker,<br/>variation errors, latency"| B
    C -.->|"bot filters, dropped events,<br/>caching, ad blockers"| C
    D -.->|"wrong trigger point,<br/>bad joins, segment filters"| D

Assignment-stage causes

The split is wrong at the moment of bucketing. A faulty randomization or hash function, corrupted or reassigned visitor IDs, or carryover effects from a previous experiment that still tag returning users. Uneven ramping belongs here too: if you launch variation B a day after A, or ramp them to different allocations, the counts diverge for a completely mundane reason.

Exposure-stage causes

Assignment was fair, but visitors are lost between being bucketed and being counted. Redirect tests are the classic offender — a visitor sent to a separate URL can close the tab before the redirect completes, so the event never fires and that visitor is never counted. Because the delay is not symmetric across variations, the counts drift. Client-side flicker and load races drop a variation when the page renders before the experiment code runs, latency in one variation lets impatient users leave first, and a variation that errors for a subset of browsers silently loses those visitors.

Logging and analysis-stage causes

The split was fine; the measurement distorts it. Bot and spam filtering applied unevenly strips a traffic source from one arm. Ad blockers and tracking-prevention block the logging beacon for some visitors more than others. Caching or CDN behavior serves stale assignments. On the analysis side, choosing the wrong starting point — counting from a downstream event that already differs between variations rather than from first exposure — manufactures an SRM out of clean data. Bad table joins and delayed log arrival do the same.

The most useful directional clue: excess visitors in the treatment arm often points to logging or engagement changes (the new variant generates more events, so more one-time visitors get recorded), while missing visitors in treatment points to exposure failures like redirects, errors, or latency.

How to diagnose an SRM

Once the check fails, resist the urge to guess. Diagnosis is a process of localization: find the segment or stage where the mismatch lives, and the cause usually becomes obvious.

flowchart TD
    S[SRM detected] --> Q1{Is it in every segment<br/>or just some?}
    Q1 -->|Isolated to one<br/>browser/device/source| Seg[Suspect that segment:<br/>rendering bug, bot source,<br/>ad blocker, redirect on that platform]
    Q1 -->|Spread evenly<br/>across all segments| Q2{Does the split break<br/>at assignment or later?}
    Q2 -->|Wrong at first exposure| Assign[Assignment layer:<br/>hashing, IDs, ramp timing,<br/>overlapping experiments]
    Q2 -->|Fine at exposure,<br/>off downstream| Q3{Redirect or heavy<br/>client-side render?}
    Q3 -->|Yes| Exp[Exposure loss:<br/>redirect timeouts, flicker,<br/>variation errors]
    Q3 -->|No| Log[Logging/analysis:<br/>bot filter, wrong trigger point,<br/>join error, caching]

Work through it in this order:

Segment the mismatch. Break the counts down by browser, device, operating system, day, traffic source, and new-vs-returning. If the SRM collapses into one segment — say, only Safari, or only mobile, or only the day of a deploy — you have found it. A rendering bug, a browser that rejects redirects, or a bot wave from one source will all show up as a single hot segment.
Find the stage where the ratio breaks. Compare the split at assignment, at exposure, and at analysis. If bucketing is even but exposure is skewed, the loss is happening after assignment (redirects, errors, latency). If assignment itself is off, the problem is upstream in the hashing or ID logic.
Check the analysis trigger. Confirm you are counting from first exposure, not from a downstream event that the treatment itself changed. This alone explains a large share of "SRMs" that are really analysis artifacts.
Line up the timeline. Map the onset of the mismatch against deploys, ramp changes, and marketing sends. SRM that starts precisely at a code push has a precise cause.

One caution specific to how detection works: run the SRM check on the whole experiment population, not on filtered segments as a decision input. Segmenting is a diagnostic tool for finding the cause once an SRM is flagged — it is not a place to hunt for fresh imbalances to act on, because slicing enough ways will always turn up a low p-value by chance.

How to prevent SRM

Prevention is cheaper than diagnosis. The teams that rarely see SRM have built these habits in:

Assign server-side where you can. Server-side or edge assignment removes the flicker, redirect, and client-render failure modes that cause most exposure-stage mismatches. Client-side tests are more exposed and need tighter QA.
Bucket at the point of exposure. Assign the visitor at the moment they are actually eligible to see the change, not at some earlier step that a variation may reach at a different rate. Assign-at-exposure keeps the counted population aligned with the treated population.
Run an [A/A test**](https://optipilot.com/data/aa-testing) first.** An A/A test — two identical variants — should show no SRM. If it does, your instrumentation is broken before you have tested anything real. It is the cheapest possible way to catch a pipeline problem.
Put an SRM check in CI and pre-analysis. Make the chi-square check an automated gate that runs continuously, not a manual step someone remembers at the end. Add it to your pre-launch experiment QA checklist so it is verified every time.
Be careful with redirects and uneven ramps. Prefer single-page variation over redirects when you can; when you cannot, expect and monitor a small imbalance. Ramp all variations together, and never adjust traffic mid-flight to "rebalance" counts — that corrupts the results further.

How Optimizely detects SRM automatically

Optimizely does not make you run this check by hand. Its Stats Engine A/B tests include automatic SRM detection with an Experiment Health indicator on the results page: a green Good status means no imbalance, and an SRM detected status means the test has failed a traffic check and needs investigation before you trust it.

One design detail matters for practitioners. Optimizely's detector does not use a one-shot chi-square test. It uses a sequential SRM (SSRM) method that checks traffic counts continuously as data arrives, so it can flag a problem within the first days of a launch — when stopping the test actually limits the damage — rather than only at the end. This is the same reasoning behind sequential testing and the peeking problem: a fixed-horizon test is only valid if you look once, whereas a sequential test is built to be monitored continuously. A retroactive end-of-experiment chi-square check has the opposite problem — by the time the totals even out, a transient mid-experiment SRM can be hidden entirely.

A few operational limits are worth knowing. Optimizely's automatic detection applies to Stats Engine A/B tests with Manual traffic distribution (not Stats Accelerator), running 45 days or less, with at least 1,000 visitors, and it evaluates only first decisions. It does not run on segmented results, and it does not check paused or archived experiments. So the manual chi-square check and the SRM Checker still earn their place: for Stats Accelerator tests, for segment-level diagnosis once an alarm fires, and for any experiment run on a platform without built-in detection.

Trust the split before you trust the lift

Sample ratio mismatch is the smoke detector of experimentation. It does not tell you what is on fire, but it tells you — reliably, and before you make an expensive decision — that you should not trust the numbers yet. The discipline is simple and non-negotiable: check the split with a chi-square test against a strict threshold, and if it fails, find and fix the cause rather than reasoning around it. A test with an SRM is not a weaker result; it is not a result at all.

Make it the first thing you look at, not the last. Before you read a single metric, run your observed counts through the SRM Checker. If the split holds, everything downstream — your lift, your confidence interval, your variance-reduced estimates — rests on solid ground. If it doesn't, you have saved yourself from shipping a decision built on a broken experiment.

Prevent Optimizely from being blocked by ad-blockers using AWS

David Sert — Thu, 20 Aug 2026 05:10:10 +0000

Category: Privacy

Some ad-blockers might block Optimizely from running. In this tutorial, we’ll see how we can prevent this from happening.

Ad-blockers will often look at the domain the request is originating from and if it happens to be from a list of known analytics tools (such as Optimizely.com), the ad blocker will block the network request from happening.

The key solution is to proxy requests via an API gateway which will forward all requests back to Optimizely.com.

Here’s how.

Step 1: Create an AWS API Gateway

flowchart TB
    subgraph Browser
        A[Your Website] --> B[Proxy Request]
    end

    subgraph AWS["AWS CloudFront"]
        B --> C[Your Domain cdn.yourdomain.com]
        C --> D[CloudFront Distribution]
    end

    subgraph Optimizely
        D --> E[cdn.optimizely.com]
        E --> F[Optimizely Snippet]
    end

    F --> A

    style AWS fill:#ff9900,color:#fff
    style Optimizely fill:#0037ff,color:#fff

Head the AWS Management Console then head to the API gateway section

Then, click on Create New API

Select HTTP API as the API type:

Step 2: Create the required routes

Now we are going to create our API routes and where the API will forward the requests to.

We’ll need 3 routes, depending on the Optimizely product you use:

One that allows us to retrieve the Optimizely snippet (if you are using Web). The snippet is fetched via a GET request done to https://cdn.optimizely.com. You can find the full snippet URL inside your Project settings.
One that allows us to send decision & conversion events (logx.optimizely.com). This route will be fetched via a POST request to https://logx.optimizely.com/v1/event.
One that allows us to retrieve the Optimizely datafile (if you are using Full Stack). This route will be fetched via a GET request to https://cdn.optimizely.com/datafiles. You can find your datafile URL inside your Project settings.

Here’s how it should look like once properly filled out:

AWS will ask you for a confirmation:

Change the ANY to be exactly the same as the method on the right-hand side. (POST, GET & GET)

Congrats you now have a working API which will proxy requests to Optimizely.com.

Step 3: Fetch Optimizely from your newly-created API gateway

Now that we’ve got a working API, it’s time to update our website to start fetching from this API.

Optimizely Web

Update your script tag that contains Optimizely to no longer fetch the file from cdn.optimizely.com but from your AWS API Gateway. You’ll find the invoke URL on the main API page, as such:

Now to ensure the Optimizely snippet sends events to the API gateway instead of the default Optimizely endpoint, this is a custom snippet setting that can’t be configured by a customer. You’ll need to ask your account manager about it. They can amend your snippet to ensure the snippet dispatches events to your newly-created API.

Optimizely Full Stack

For Full Stack, you’ll need to customize the SDK’s createInstance method to include a new datafile URL pointing to your API, as such:

const optimizely = require('@optimizely/optimizely-sdk');

const optimizelyClientInstance = optimizely.createInstance({
  sdkKey: '',
  datafileOptions: {
    autoUpdate: true,
    urlTemplate: 'https:///datafiles/%s.json',
  },
});

You’ll also need to provide a custom event dispatcher which will dispatch the events back to your newly AWS API Gateway. You can use this built-in event dispatcher and change line 39 to indicate the POST endpoint of your newly created API.

Related guides

Optimizely vs Amplitude: Experimentation Compared

David Sert — Thu, 20 Aug 2026 05:09:34 +0000

"Optimizely vs Amplitude" is a slightly unfair framing, because the two products start from opposite ends of the same problem. Optimizely is an experimentation platform that added analytics; Amplitude is a product analytics platform that added experimentation. Both can run an A/B test and tell you which variation won, but the path each takes — and the org each is built for — is different. This guide compares them honestly so you can pick the right tool for your situation, or decide to run both.

TL;DR of the difference

Optimizely leads with experimentation. Its core products — Web Experimentation (visual, client-side testing) and Feature Experimentation (server-side feature flags and SDK-based tests) — are purpose-built to run a high volume of rigorous experiments across marketing pages, product surfaces, and backend logic, with the Stats Engine making continuous monitoring statistically safe.

Amplitude leads with product analytics. Amplitude Experiment is an experimentation layer bolted onto a best-in-class behavioral analytics engine, so every test is automatically connected to deep funnel, retention, and cohort analysis on the same event data.

If your primary question is "what should we test, where, and how do we govern a large testing program," Optimizely is the stronger center of gravity. If your primary question is "how do users behave, and can I experiment without leaving my analytics," Amplitude is compelling. They are not mutually exclusive — many teams run both.

The decision comes down to two questions: what is the primary job, and who runs the tests?

flowchart TD
  A{What is the primary job?} -->|Run and govern experiments| B{Who builds most of the tests?}
  A -->|Deep behavioral analytics| C{Is Amplitude already your<br/>analytics system of record?}
  B -->|Marketers, in a visual editor| D[Optimizely Web Experimentation]
  B -->|Engineers, behind feature flags| E[Optimizely Feature Experimentation]
  C -->|Yes, flag-based product tests| F[Amplitude Experiment]
  C -->|No, and you test backend + marketing pages| G[Optimizely for experiments,<br/>Amplitude for analysis]

What each tool is built for

Optimizely

Optimizely splits experimentation into two complementary products. Web Experimentation uses a WYSIWYG Visual Editor and a JavaScript snippet, letting marketers and optimization teams build A/B tests, redirect tests, and multivariate tests on a live site without engineering for most changes (custom HTML/JS is available when needed). Feature Experimentation is developer-first: server-side and client-side SDKs (Java, Python, Go, C#, JavaScript/Node, Swift, Android, and more) wrap experiments in feature flags, so you can roll features out, roll them back instantly, and test backend logic such as algorithms, pricing, or APIs. Both share Optimizely's Stats Engine for results.

The platform is built for experimentation breadth and program scale: visual marketing tests and deep server-side tests under one roof, audience targeting, mutual exclusion groups, multi-armed bandits, and a free Rollouts tier for teams starting with feature flags.

Amplitude

Amplitude's foundation is its analytics graph: every event a user fires is stored and queryable through funnels, retention curves, pathfinder, and behavioral cohorts. Amplitude Experiment sits on top of that data, offering feature flags, server-side and client-side evaluation, and both feature-flag-based experiments and a web experiment capability. Its defining advantage is that experiment results read from the same event stream as the rest of your analytics — so a winning variation can immediately be sliced by any cohort or downstream behavior you already track.

Amplitude is built for product teams who live in their analytics and want experimentation to be a natural extension of behavioral analysis rather than a separate discipline.

A note on the moving market: OpenAI acquired the experimentation vendor Statsig in September 2025, and in May 2026 Amplitude announced it was taking over the Statsig brand, platform, and customer base. That leaves Amplitude with two experimentation lineages (its native Experiment product and the Statsig platform), and some analysts have flagged near-term uncertainty about how the overlapping capabilities will consolidate. Treat any roadmap promises in this area as provisional.

Head-to-head comparison

Dimension	Optimizely	Amplitude
Primary use case	Experimentation across web and product/backend	Product analytics, with experimentation layered on
Experimentation model	Visual (Web Experimentation) + feature flags / server-side SDKs (Feature Experimentation)	Feature-flag-based experiments + web experiment capability
Visual/no-code testing	Yes — mature WYSIWYG Visual Editor	Limited; beyond simple changes, work tends to need engineering
Server-side / SDK testing	Strong, broad SDK coverage	Yes, via Experiment SDKs
Statistics engine	Stats Engine (always-valid, sequential, controls false positives)	Sequential testing, t-tests, CUPED, multi-armed bandits
Analytics depth	Solid experiment analytics; warehouse-native analytics available	Best-in-class behavioral analytics, cohorts, retention, funnels
Integration model	Connects to external analytics (including Amplitude); CMS/DXP ecosystem	Experimentation tied directly to native event data
Pricing model	MAU-based (free Rollouts tier for flags)	Event-volume based
Ideal team	Optimization, growth, and engineering teams running a structured testing program	Product teams who already standardize on Amplitude analytics

Experimentation capabilities compared

The clearest difference is where and how you can test.

Optimizely covers two distinct surfaces well. For marketing and front-end teams, the Web Experimentation Visual Editor makes it straightforward to change copy, layout, and styling and ship an A/B test without a deploy. For engineering and product teams, Feature Experimentation runs tests behind feature flags directly in application code — including server-side paths where there is no DOM to manipulate, such as recommendation algorithms, checkout logic, or API behavior. Flags double as a kill switch, so a bad variation can be turned off remotely without redeploying. Optimizely also supports targeted rollouts, mutual exclusion, and multi-armed bandit optimization.

Amplitude Experiment is primarily flag-based: you gate a feature behind a flag, evaluate it client- or server-side, and measure the result against your event data. This is a clean model for product experimentation and progressive rollouts. Its web experimentation capability exists, but for anything beyond simple changes, practitioners generally report that meaningful variations require engineering involvement rather than a marketer-friendly visual workflow. If a large share of your testing is on marketing pages built by non-engineers, that distinction matters.

On the statistics, both are credible. Optimizely's Stats Engine is designed for "always-valid" inference — you can monitor results continuously without inflating false-positive rates, which suits teams that watch dashboards daily. Amplitude likewise uses sequential testing for valid-anytime results, and supports t-tests, CUPED for variance reduction, multi-armed bandits, mutual exclusion groups, and holdouts. Neither team should feel they are settling for weak statistics.

Analytics and data compared

This is where Amplitude's genuine strength shows. Because Experiment runs on the same event pipeline as Amplitude Analytics, a result is never a dead end: you can immediately ask "did the winning variation help retention at day 30," "how did it perform for this behavioral cohort," or "where in the funnel did the lift come from" without exporting data or stitching IDs across tools. For teams whose core competency is behavioral analysis, that tight loop is the main reason to consider Amplitude Experiment over a standalone testing tool.

Optimizely's analytics are good for reading experiment results and, increasingly, offers warehouse-native analytics that let you analyze decision and event data alongside the rest of your data in Snowflake, BigQuery, or Databricks. But product analytics is not its historical center of gravity the way it is Amplitude's. If your team's daily home is rich cohort and retention analysis, Amplitude will feel more native; if your team's daily home is running and governing experiments, Optimizely will.

A practical caveat on cost: Amplitude's event-volume-based pricing can scale quickly for high-traffic products, whereas Optimizely prices experimentation on monthly active users and offers a free Rollouts tier for feature flags. Model both against your actual traffic before deciding — the cheaper option depends heavily on your event volume versus user count.

When to choose Optimizely

Optimizely is the stronger choice when:

Experimentation is the primary job, not a feature of your analytics tool. You want one platform that handles both marketing-page tests and deep server-side product tests.
Non-engineers run a meaningful share of tests. The Visual Editor lets growth and marketing teams ship experiments without a deploy.
You test in the backend — algorithms, pricing, infrastructure changes — where flag-based, server-side experimentation and an instant kill switch are essential.
You run a high-volume, governed program and want mutual exclusion, bandits, audience targeting, and always-valid statistics in one place.
You are starting with feature flags and want a free on-ramp via Rollouts before scaling up.

When Amplitude makes sense

Amplitude Experiment is the better fit when:

Amplitude is already your analytics system of record and most of your team's decisions start in its funnels, cohorts, and retention reports.
Your experimentation is product-led and flag-based — gating features, progressive rollouts, and measuring impact on behavioral metrics.
The analysis loop matters more than test authoring breadth. You value being able to slice any result by behavioral cohort instantly, on the same data, without integration work.
You are not relying on a visual, marketer-driven editor for the bulk of your tests.

Be deliberate here given the current consolidation around the acquired Statsig platform: confirm which experimentation product a vendor is steering you toward and what its supported roadmap looks like.

How they work together

These tools are not an either/or for many organizations, and the most pragmatic answer is often "both." A common pattern is to run experiments in Optimizely and analyze them in Amplitude: Optimizely decides which variation a user sees and provides the statistical results, while the variation a user was bucketed into is sent into Amplitude as a user property or event, so you can analyze experiment impact against your full behavioral dataset.

This is a supported, documented integration. Optimizely Feature Experimentation can forward decision data to Amplitude via a decision notification listener (setting an [Optimizely] <flagKey> user property and an optional impression event), and Optimizely Web Experimentation integrates through custom analytics extensions, with the connector built and maintained by Amplitude. If your team has standardized on Amplitude for analytics but wants Optimizely's experimentation breadth and visual testing, this combination gives you the best of both rather than forcing a single choice.

The bottom line

Pick based on where your center of gravity sits. If experimentation is the discipline you are investing in — across marketing, product, and backend — Optimizely is built for that job and gives non-engineers and engineers a shared platform with rigorous statistics. If best-in-class behavioral analytics is your foundation and you want experimentation as a tightly coupled extension of it, Amplitude Experiment earns its place. And if you have both needs, the supported integration means you do not have to choose: experiment in Optimizely, analyze in Amplitude.

This site covers both Optimizely products in implementation depth. Browse the Feature Experimentation guides and the Web Experimentation guides, or start from the integration catalog. Not sure which platform fits your stack? Contact us.

Related guides

How the Optimizely Stats Engine Works

David Sert — Thu, 20 Aug 2026 05:08:58 +0000

Optimizely's Stats Engine is the statistical framework behind every winner, loser, and "inconclusive" verdict on the Experiment Results page. It exists to solve a specific, expensive problem: practitioners watch experiments in real time and make decisions the moment a result "looks significant," and traditional statistics punishes that behavior with a flood of false positives. Understanding how Stats Engine works is the difference between trusting a result and acting on noise.

What Stats Engine Is and the Problem It Solves

Stats Engine is Optimizely's proprietary statistical methodology for evaluating A/B tests. It is a frequentist engine built on sequential testing combined with false discovery rate** (FDR) control**. Those two mechanisms are the whole story, and each addresses a distinct failure mode of classical A/B testing.

The first failure mode is the peeking problem. Traditional fixed-horizon statistics (the t-test being the canonical example) are only valid if you commit to a sample size in advance, wait until you reach it, and look exactly once. Every time you check results early and react, you give random noise another chance to cross the significance threshold. A test that peeks repeatedly can report a false "winner" far more often than its stated error rate suggests. But waiting passively for a pre-computed sample size is exactly what real experimentation teams do not do.

The second failure mode is the multiple comparisons problem. Real experiments rarely track one metric on one variation. Add more variations and more metrics and the chance of at least one false positive climbs quickly, even though each individual test holds its error rate. Worse, the rate that actually matters to a decision-maker, the proportion of false positives among the results you act on, is higher still.

Stats Engine is engineered so that both problems are handled automatically. As Optimizely's documentation states, results are always valid: you can monitor a test continuously and stop as soon as you have a clear winner, without invalidating it, because the engine controls the false discovery rate throughout.

How Sequential Testing Solves the Peeking Problem

Sequential testing is the mechanism that makes continuous monitoring safe. Rather than computing a single p-value at one fixed endpoint, Stats Engine evaluates the experiment as evidence accumulates over time and produces inferences that remain valid no matter when you look.

Optimizely's own framing uses a baking analogy that is worth keeping in mind:

Fixed Horizon : Set a timer before baking. You may only open the oven
                when the timer ends. Open it early and the result is unreliable.
Sequential    : Put the cake in without committing to a time. Open the oven
                whenever you like to check; looking never ruins the result.
                When it looks done, it is done.

Mechanically, Stats Engine does not compute one confidence interval. It computes a series of 100 successive confidence intervals across the experiment's lifetime, each with its own significance value. The numbers you see on the Results page are deliberately conservative summaries of that series:

The statistical significance shown is the smallest significance value observed across those sequential intervals, not the average and not the latest.
The confidence interval shown is the running intersection of all prior intervals: it tracks the smallest upper limit and the largest lower limit seen during the run.

Because of this, the displayed significance and interval may not exactly match the currently observed conversion rates. That is intentional. It is what makes the result robust to having been observed many times.

Why Significance Climbs (and Occasionally Drops)

In a stable environment, significance should rise in a stepwise, generally increasing fashion as evidence accumulates. Two forms of evidence move it: larger differences between conversion rates, and differences that persist across more visitors. Early on, when the sample is small, large swings are treated conservatively, so you often see a flat line that later rises sharply once real evidence accrues.

Significance can also fall, though Optimizely's analysis indicates this happens in only about 4% of experiments. Small dips of a few percentage points come from time bucketing: Optimizely divides the experiment's runtime into 100 equal buckets that grow as the test runs, reshuffling visitors among them and recomputing as it goes, which produces minor fluctuations. Larger drops, potentially all the way to 0%, come from a stats reset, a protective mechanism that triggers when the engine detects that the underlying environment has changed (the assumption that data is identically distributed has been violated). A reset is the engine refusing to stand behind a conclusion the new evidence no longer supports.

False Discovery Rate Control vs Traditional Significance

The second pillar is what makes Stats Engine trustworthy when you run many metrics and variations. The naive error metric is the false positive rate: out of all the comparisons where there is truly no effect, what fraction are wrongly flagged? Optimizely controls something more decision-relevant, the false discovery rate: out of the results you would actually act on (the declared winners and losers), what fraction are wrong?

The distinction matters enormously. Optimizely's worked example: an experiment with ten comparison opportunities reports two winners, one of which is a false winner. Measured as a false positive rate, that is 1 in 10, about 10%, which sounds acceptable. But you do not implement the eight inconclusive results; you implement the two winners. Among those, your error rate is 1 in 2, or 50%. The false discovery rate captures the risk that actually reaches your roadmap.

False positive rate = false positives / all null comparisons   = 1/10 = 10%
False discovery rate = false positives / declared discoveries   = 1/2  = 50%

To control FDR across many hypotheses, Stats Engine uses a tiered version of the Benjamini-Hochberg procedure. The tiering reflects that not all metrics deserve equal weight:

Primary metric (rank 1) — evaluated independently of all others, so it reaches significance as fast as possible and is unaffected by how many other metrics you track.
Secondary metrics (ranks 2-5) — their significance threshold is adjusted for the number of metrics and variations. Adding more secondary metrics can slow each of them to significance, but never slows the primary metric.
Monitoring metrics (rank 6+) — each given a fractional weight of 1/n, so they have minimal impact on secondary metrics and none on the primary metric.

The practical payoff: Optimizely keeps the false discovery rate low (approximately 10%) while still letting your most important metric reach significance quickly. This is also why Stats Engine uses two-tailed tests, which are required for FDR control.

One caveat the documentation is explicit about: FDR control is not maintained when you segment results. The deeper you slice, the higher your chance of finding a spurious "significant" segment. Use segments for exploration, not for decisions, and only inspect the most meaningful ones.

How to Read Significance and Confidence Intervals

Statistical significance answers a precise question: how unusual would these results be if the variation and baseline truly performed identically? At 90% significance, you are accepting roughly a 10% false-positive risk on that call. The confidence interval is the estimated range that likely contains the true effect (the true uplift), and Optimizely sets its confidence level to match your project's significance threshold (90% by default).

The single most useful rule for reading the Results page:

A variation reaches significance exactly when its confidence interval stops crossing zero.

Confidence interval entirely above 0% means a winning variation.
Confidence interval includes 0% means inconclusive (you cannot yet rule out "no effect").
Confidence interval entirely below 0% means a losing variation.

Before declaring anything, Stats Engine enforces minimum data thresholds. For binary metrics, it requires at least 100 visitors or sessions and at least 25 conversions in both the baseline and the variation. For numeric metrics such as revenue, it requires at least 100 visitors or sessions but no fixed conversion count. Until those are met, the page reports that more visitors are needed and estimates the wait.

A useful judgment heuristic from the documentation: if the observed mean (the tick mark) sits near the edge of the confidence interval, the engine may be accumulating evidence against its own conclusion, so consider waiting. If the observed mean sits near the center, you can be more confident the call will hold.

For revenue-per-visitor goals, be aware that revenue distributions are heavily skewed, which reduces statistical power. Stats Engine applies skew correction to recover some of that power and to shape the confidence interval correctly, but skewed metrics still generally need more data than binary ones.

Stats Accelerator: A Separate Feature, Not the Engine

Stats Accelerator is frequently conflated with Stats Engine. They are different things. Stats Engine is the statistical methodology that evaluates results. Stats Accelerator is a traffic-allocation feature that sits on top of it and uses a multi-armed-bandit-family algorithm (a variation on the Upper Confidence Bound strategy) to shorten the time to statistical significance.

Stats Accelerator monitors a running experiment and routes more traffic toward the variation showing the most significant difference from the baseline, regardless of whether that difference is positive or negative, because its goal is to minimize time, not regret. Once a variation reaches significance, it is removed from consideration and traffic is redistributed to the rest. It requires at least three variations (a baseline plus two). It still produces statistical significance, because the underlying engine is still doing the inference.

This is distinct from a true multi-armed bandit** (MAB) optimization** (formerly "Accelerate Impact"), which minimizes regret by funneling traffic to whichever variation currently performs best on the primary metric. MABs are for temporary, value-maximizing scenarios such as a Black Friday promotion, and crucially MAB optimizations do not generate statistical significance at all.

Feature	Goal	Produces significance?	Use when
Stats Accelerator	Minimize time to significance	Yes	You want a reliable winner faster
Multi-armed bandit	Maximize reward / minimize regret	No	Short-lived campaigns; no permanent decision needed

Because Stats Accelerator changes traffic allocation mid-flight, it risks a sampling bias called Simpson's Paradox when conversion rates vary over time (for example, weekday-vs-weekend seasonality). Optimizely addresses this with the Epoch Stats Engine, which produces a stratified, weighted improvement estimate, comparing baseline and variation within each interval between allocation changes, then combining those intervals by visitor count. This is also why, with Stats Accelerator enabled, the Results page may report both absolute (in percentage points) and relative improvement. For Feature Experimentation, use a user profile service (sticky bucketing) so frequent reallocation does not expose one visitor to multiple variations.

Common Misconceptions

Is the Optimizely Stats Engine Bayesian?

No. This is the most common misconception, and it is worth correcting precisely. Stats Engine is a frequentist sequential method. Optimizely does offer a separate, explicitly Bayesian A/B testing mode (which expresses results as direct probabilities like "90% chance B beats A"), and a separate Frequentist Fixed Horizon mode. But the classic Stats Engine, the one that powers sequential testing, is frequentist. It reports statistical significance and frequentist confidence intervals, not posterior probabilities. The reason a search for "Optimizely Bayesian" surfaces Stats Engine at all is that both Bayesian and sequential methods let you peek and stop early. That shared behavior does not make them the same methodology.

Stats Engine results disagree with my t-test, so they must be wrong

They can legitimately disagree, and Stats Engine is the more trustworthy of the two when you have been monitoring continuously. A t-test uses only the currently observed mean and difference, so if evidence looked strong early and weakened later, a t-test can still report the stale, strong result. Stats Engine's intersected intervals are more conservative, less likely to declare a false winner, and less likely to reverse a call later.

I can keep slicing segments until something is significant

You can, but you will be manufacturing false discoveries. FDR control does not extend across segments. Repeated segment-hunting inflates false positives exactly like peeking does.

A stats reset means the tool is broken

The opposite. A reset means the engine detected that the environment changed and is protecting you from standing behind a conclusion the new data no longer supports.

Practical Guidance for Trusting Your Results

Rank your metrics deliberately. Put the metric that defines success as the primary metric, ideally measured close to the change in the funnel. It gets independent, fastest-to-significance treatment; everything else is secondary or monitoring.
Let the test run to its planned duration even though you can peek. Sequential validity means peeking will not break your stats, but a result that has barely cleared the threshold on thin data is fragile. Treat experimentation as a standardized process, not a dashboard you babysit.
Read the confidence interval, not just the significance number. Width tells you precision; position relative to zero tells you direction; the tick mark's position warns you whether a call is at risk of reversing.
Use segments to explore, never to decide. If a segment looks interesting, treat it as a hypothesis for a new, properly powered experiment.
Match the method to the intent. Use a standard A/B test (sequential Stats Engine) when you need a trustworthy decision; add Stats Accelerator to reach that decision faster; use a multi-armed bandit only for temporary value-maximization where you do not need a statistically defensible winner.

For how the three inference families differ in practice, see Bayesian vs frequentist vs sequential testing.

Related guides

Feature Flag Best Practices for Production Systems

David Sert — Thu, 20 Aug 2026 05:07:11 +0000

Your flag count has quietly crossed a threshold. What started as a clean way to ship a risky feature behind a toggle is now a few hundred conditionals scattered across services, half of them permanently on, nobody quite sure which are safe to delete. Someone flips the wrong one during an incident. A test passes locally and fails in CI because a flag defaulted differently. This is the predictable midlife of feature flagging: the mechanism is trivial, but the discipline around it is what separates teams who ship faster from teams who have simply moved their risk somewhere less visible.

The good news is that the disciplines are well understood and largely independent of any one vendor. This article lays them out, then shows how they map onto Optimizely Feature Experimentation specifically, so that the practices have a concrete home in tooling rather than living only in a wiki page nobody reads.

Classify every flag by type and lifespan

The single most useful idea in feature flagging is that not all flags are the same thing. Martin Fowler's taxonomy splits them into four categories, and the reason it matters is that each has a different owner, a different lifespan, and a different failure mode when you ignore it (Fowler, Feature Toggles).

Release toggles hide in-progress work so it can merge to trunk and ship dormant. They are the shortest-lived flags you own, meant to survive days or weeks, and they should be deleted the moment the feature is fully rolled out.
Experiment toggles split traffic between variations to measure an outcome. They live exactly as long as the test needs to reach a decision, then collapse to the winning variation. This is the layer beneath A/B testing.
Ops toggles give operators control over system behavior: degrade an expensive feature under load, disable a flaky dependency, or trip a kill switch during an incident. Some are short-lived, but a genuine kill switch may live indefinitely as a safety control.
Permissioning toggles decide who gets access to what, such as gating a feature to premium accounts. These can legitimately live for years and are effectively part of your product, not temporary scaffolding.

Fowler frames these along two axes: longevity (how long the flag should exist) and dynamism (whether the decision is fixed per deployment or must be re-evaluated per request). The categories fall at different points, and conflating them is the root of most flag debt. A release toggle that gets treated like a permanent ops switch never gets cleaned up, because nobody remembers it was supposed to be temporary.

The practical takeaway is to make the type explicit at creation. Decide, before you write the flag, whether it is a release, experiment, ops, or permissioning flag, and record that alongside an owner and an expected removal or review date. In Optimizely, flag rules make the distinction concrete: an A/B test rule is an experiment flag, while a targeted delivery is a release or ops flag. If you are weighing a fixed split against an adaptive one for an experiment flag, the trade-offs are covered in Stats Accelerator vs multi-armed bandit vs contextual bandit.

flowchart TD
    A[New flag] --> B{What is its job?}
    B -->|Hide in-progress work| C[Release toggle<br/>lifespan: days-weeks<br/>delete after rollout]
    B -->|Split traffic to measure| D[Experiment toggle<br/>lifespan: the test<br/>collapse to winner]
    B -->|Operator control / kill switch| E[Ops toggle<br/>lifespan: short, or<br/>indefinite for safety]
    B -->|Gate access by plan/role| F[Permissioning toggle<br/>lifespan: years<br/>part of the product]
    C --> G[Book its removal now]
    D --> G
    E --> H[Owner + review date]
    F --> H

Name flags so they stay searchable at scale

At ten flags, naming is a matter of taste. At three hundred, it is the difference between a flag you can find and a flag you are afraid to touch. A good flag key is greppable across your codebase, self-describing, and encodes enough context that someone who did not create it can guess its purpose and owner.

A convention worth adopting encodes, in a consistent order, the owning team or domain, the flag's intent, and where it applies. Something like checkout_new-address-form_release or search_semantic-ranking_experiment reads cleanly, sorts sensibly, and tells a reader the type without opening a dashboard. Keep to one delimiter, avoid spaces and punctuation, and never reuse a retired key for a new purpose. OptiPilot's free feature flag naming convention generator turns a flag's type, team, and feature into a consistent key across dot, kebab, and snake styles, so a whole team names flags the same way.

Naming discipline pays off precisely because flag keys are the join between your code and your configuration. Optimizely makes this contract literal: a rule's key cannot be changed after creation, and if the key referenced in your decide call does not match a key in the platform, no traffic is served to that rule (Manage rules). A typo is not a loud error; it is a silent no-op that quietly serves the fallback. Consistent, reviewed names make that class of mistake far easier to catch in code review.

Roll out gradually and keep a kill switch

The reason flags earn their keep in production is that they decouple deploy from release. You ship code dark, then turn it on for a slice of traffic, watch, and widen the slice, or reverse instantly if something breaks. Two properties make this safe: a gradual ramp and a fast reversal.

A gradual ramp means starting small. Optimizely's own guidance for feature rollouts recommends 10% as a starting allocation before widening (Run Feature Rollouts). You can also target the ramp: deliver to internal users first, then beta customers, then the general population, using audience conditions rather than a blunt percentage. Optimizely's targeted delivery supports exactly this, letting you roll out to a percentage of a specific audience and increase it over time (Flag delivery overview).

Fast reversal is the kill switch. Because flag changes take effect through configuration rather than a code deploy, you can roll a feature back in seconds. In Optimizely, deliveries take effect without a code deployment, and you can toggle a flag on and off without re-bucketing the users already assigned, as long as you do not change the traffic allocation at the same moment (Run flag deliveries). That last caveat matters operationally: an incident kill switch should flip the flag off, not simultaneously re-slice traffic, or you will disturb bucketing for everyone else.

There is a subtle trap in ramping that shows up most in experiment flags. Bucketing is deterministic: Optimizely hashes the user ID to assign a variation, so the same user gets the same experience as long as the distribution has not changed (Core concepts). Change the traffic allocation on a running experiment and users can be re-bucketed, which both disturbs their experience and can skew results. If you need to change allocation mid-flight, use a user profile service to keep bucketing sticky, and watch for sample ratio mismatch, the canary that tells you your split is not landing the way you configured it.

One decision to make deliberately: whether a given flag needs measurement. Optimizely separates a plain targeted delivery (free, no analytics, dispatches no decision events) from a feature rollout (full analytics, consumes monthly active users and impressions). Use a targeted delivery for a simple release or kill switch where you only need on/off control, and a rollout or A/B test when you actually want to measure impact. Reaching for the analytics-bearing rule on a flag you will never analyze is a common and needless cost.

Environments and the datafile: where flags actually live

Flags are configuration, and configuration needs somewhere to live per stage of your pipeline. In Optimizely, a project contains one or more environments, typically staging and production, and each environment holds its own ruleset for a flag. You can run a flag at 100% in development and 10% in production, or configure entirely different rules per environment, with separate permissions for who can change each (Core concepts).

At runtime the SDK does not call back to Optimizely on every decision. It downloads a datafile, a JSON snapshot of your flags and rules for one environment, and evaluates decisions locally against it. When you edit a rule, the datafile updates within a few seconds, and how quickly that reaches production depends on how often your app fetches it; webhooks let you push updates in near real time (Manage rules). This architecture is why flag evaluation is fast and why it keeps working during a brief network blip: decisions are computed from an in-memory config, not a synchronous network round trip.

It also defines your failure mode, which is the next thing to get right.

Test safely with flags and sensible fallbacks

Flags multiply the states your system can be in, and untended they make tests flaky and behavior unpredictable. Three habits keep that under control.

First, always have a safe default. Design so that the flag being off, or the configuration being unavailable, yields a working, conservative experience. Optimizely's decide method returns an OptimizelyDecision object whose enabled field is false until a rule turns the flag on, and on a critical error, such as the SDK not being ready or an unknown flag key, it returns a nullvariationKey and populates reasons (Decide methods). Your code should treat that state as "off" and fall back to a value you control, rather than assuming a variable will be present. For resilience against the SDK not having a datafile yet, you can also bundle a datafile with the application so the client can initialize synchronously from a known-good snapshot.

Second, decouple the decision from the decision point. Fowler's advice is to avoid scattering raw flag checks throughout the codebase, because changing the logic then means hunting down every call site. Instead, funnel decisions through a single abstraction, a FeatureDecisions object or equivalent, so the toggle point (where behavior branches) is separated from the toggle router (how the decision is made). This keeps business logic testable without the flagging system present, and confines flag removal to one place.

Third, test the states you actually ship. At minimum, exercise the flag-on and flag-off paths, since both are real production states. Wrap decisions so tests can inject a decision directly rather than standing up the SDK, which removes network dependence and the flakiness that comes with it. For the human side of pre-launch verification, a structured pass like the experiment QA checklist catches the targeting, bucketing, and event-tracking mistakes that unit tests miss.

Pay down flag debt before it compounds

Every flag is inventory, and inventory has a carrying cost. Fowler's framing is blunt: "savvy teams view their feature toggles in their codebase as inventory which comes with a carrying cost and seek to keep that inventory as low as possible." Each live flag is an extra branch to reason about, an extra combination to test, and an extra thing that can be misconfigured. Stale flags are the single most common feature-flag problem, and they accumulate because deleting a flag is nobody's urgent job.

The fix is to make cleanup a scheduled, owned activity rather than a good intention:

Book the removal when you create the flag. For every release toggle, add a removal task to the backlog at the moment it is introduced. The work of retiring it is then already tracked, not rediscovered months later.
Give short-lived flags an expiry. Record an expected removal date on release and experiment flags. Some teams go further and build "time bombs" that fail a test, or even refuse to start the application, once a flag outlives its expiration date, converting silent debt into a loud, actionable failure.
Cap the inventory. A Lean-style limit on the total number of live flags forces a retirement before a new flag can be added, keeping the pile bounded.
Review the whole set on a cadence. Optimizely's flags dashboard gives you the raw material: every flag with its status, creator, and last-modified date, filterable by environment and audience (Custom Flags Dashboard). A monthly sweep for flags that are fully rolled out, concluded, or untouched for a quarter turns "someone should clean these up" into a concrete list.

When a flag is genuinely done, retire it in both places. Remove the code branch and the flag definition together, so you never leave a dangling key or an orphaned conditional. The code side of that sweep can be automated: find unused feature flags walks through locating stale flag references and dangling keys in your codebase before they calcify. In Optimizely, archiving a flag removes it from the datafile and requires that it not be active in your highest-priority environment, usually production, which is a useful guardrail against archiving something still serving traffic; the flag's data is retained so you can unarchive later if needed (Custom Flags Dashboard).

How Optimizely maps flags to your code

Pulling the pieces together, here is the shape of the integration. A flag is, in Optimizely's own words, "a place in code where a decision is made." You create a user context from a user ID and any targeting attributes, then call decide with the flag key. The returned decision tells you whether the flag is enabled and carries any configuration variables and the variation key (Key concepts).

The following is illustrative JavaScript, showing the decision routed through a single wrapper with an explicit fallback. Method names are verified against current SDK docs, but exact syntax varies by language and SDK major version.

// Illustrative: one place that owns the decision, with a safe default.
const FALLBACK_SORT = 'alphabetical'; // conservative, always-works behavior

async function getProductSort(optimizely, userId, attributes) {
  // Wait until the SDK has a datafile; on failure, fall back.
  try {
    await optimizely.onReady();
  } catch (err) {
    return { enabled: false, sortMethod: FALLBACK_SORT };
  }

  const user = optimizely.createUserContext(userId, attributes);
  const decision = user.decide('product_sort');

  // Critical error: variationKey is null and reasons are populated.
  // Treat as "off" and use our own fallback rather than trusting variables.
  if (decision.variationKey === null) {
    console.warn('flag decision failed:', decision.reasons);
    return { enabled: false, sortMethod: FALLBACK_SORT };
  }

  return {
    enabled: decision.enabled,
    sortMethod: decision.enabled
      ? decision.variables['sort_method']
      : FALLBACK_SORT,
  };
}

Two SDK details are worth knowing for production hygiene. First, decide dispatches a decision (impression) event by default so the exposure shows up in results; when you only want to read a flag's state without recording an exposure, the OptimizelyDecideOption.DISABLE_DECISION_EVENT option suppresses it (Decide methods). This is the right tool when you evaluate a flag early to pre-render content and will record the real exposure later at the point of interaction. Second, remember the silent-failure rule from earlier: a flag key in your code with no matching, running rule serves no traffic, so keep code and configuration in lockstep.

The through-line across all of this is modest but powerful. Treat each flag as a small, owned, dated commitment rather than a permanent fixture. Name it so it can be found, ramp it so a mistake is cheap, keep a switch that reverses it fast, and schedule its removal the day you create it. Do that consistently and feature flags go back to being what they were supposed to be: a mechanism for shipping with confidence, not a second codebase you are quietly maintaining by accident.

To see a flag decision run end to end, open the demo lab. The JavaScript v6+, React and Next.js SSR harnesses each make a live decision and show its events. Teams still on v5 can use the JavaScript Browser v5 harness.

False Discovery Rate Control in A/B Testing

David Sert — Thu, 20 Aug 2026 05:06:35 +0000

Track one metric on an A/B test and a false positive is a 1-in-20 event at the usual 95% confidence threshold. Track twenty metrics and, under the null hypothesis, you should expect one of them to look like a winner purely by chance. Add a few variations and a segment breakdown, and the number of simultaneous comparisons climbs into the dozens. This is the multiple comparisons problem, and left uncontrolled it turns a rich results page into a machine for manufacturing phantom wins. The false discovery rate is the modern statistical answer to it — a way to keep testing many things at once without letting your rate of false claims run away from you.

This guide explains what the false discovery rate is, how it differs from older corrections like family-wise error control and Bonferroni, how the Benjamini-Hochberg procedure works with a fully worked example, and how Optimizely's Stats Engine applies false discovery rate control automatically across every metric and variation on the results page.

What Is the False Discovery Rate?

The false discovery rate (FDR) is the expected proportion of your "discoveries" — the results you flag as statistically significant — that are actually false positives. If you declare 10 winning metrics and your FDR is controlled at 5%, then on average you expect about 0.5 of those 10 to be a mistake. FDR is a property of the set of significant results, not of any single test.

That framing is the whole point. A traditional p-value controls the error rate of one hypothesis in isolation: a 5% significance level means that if this one variation has no real effect, there is a 5% chance you wrongly call it significant. But an experiment rarely tests one hypothesis. It tests a primary metric, several secondaries, a stack of guardrails, maybe two or three variations, and often a few segments. Every one of those cells is a hypothesis test. The false discovery rate asks a more useful question: across everything I flagged as a discovery, what fraction should I expect to be wrong?

Controlling the FDR lets you keep a wide, informative results page — which you want, because catching harm requires watching many metrics — while bounding how often you chase something that was never really there.

The Multiple Comparisons Problem

The mechanism is pure arithmetic. Suppose every metric on your experiment is truly null (no real difference between variation and control), and you test each at a 5% significance level. The probability that a single test does not produce a false positive is 0.95. The probability that none of m independent tests produces one is 0.95^m, so the probability of at least one false positive is:

P(at least one false positive) = 1 - 0.95^m

m = 1   metric   -->  5.0%
m = 5   metrics  -->  22.6%
m = 10  metrics  -->  40.1%
m = 20  metrics  -->  64.2%
m = 50  metrics  -->  92.3%

At twenty metrics you are more likely than not to see at least one "significant" result even when nothing is happening. This is why a naive results page that runs every metric at a flat 95% threshold is untrustworthy the moment it carries more than a handful of metrics: the headline significance numbers no longer mean what they appear to mean. The problem compounds with variations and segments, because each comparison is another draw from the same lottery.

The instinct to fix this by simply "watching fewer metrics" is the wrong trade. You need guardrail metrics to catch changes that win on conversion but quietly raise refunds or slow the page — that is a deliberately wide net. The right fix is not to look at less, but to adjust the significance bar to account for how much you are looking at.

Family-Wise Error Rate vs False Discovery Rate

There are two different error rates you might try to control, and the choice shapes everything downstream.

The family-wise error rate (FWER) is the probability of making even one false positive across the entire family of tests. Controlling FWER at 5% means there is at most a 5% chance that any of your significant results is wrong. This is a strict, conservative standard — appropriate when a single false claim is catastrophic, such as a drug-approval trial.

The false discovery rate (FDR) is the expected proportion of false positives among the results you called significant. Controlling FDR at 5% means you accept that some individual discoveries may be wrong, as long as, on average, no more than 5% of them are.

	Family-wise error rate (FWER)	False discovery rate (FDR)
Controls	Probability of any false positive	Expected fraction of false positives among discoveries
Stance	One mistake is unacceptable	A bounded share of mistakes is tolerable
Power	Low — misses many real effects	High — detects far more real effects
Typical procedure	Bonferroni, Holm	Benjamini-Hochberg
Best fit	Small number of critical tests	Many exploratory tests, as in A/B testing

For online experimentation, FDR is almost always the better target. You run many tests, most effects are small, and the cost of missing a real winner (low power) is a genuine business cost. FDR control gives you dramatically more statistical power than FWER control while still keeping false claims on a leash. That trade — accept a known, bounded fraction of errors in exchange for detecting many more true effects — is why FDR has become the default for large-scale testing.

The Benjamini-Hochberg Procedure

The Benjamini-Hochberg (BH) procedure, introduced by Yoav Benjamini and Yosef Hochberg in 1995, is the standard method for controlling the false discovery rate. It is refreshingly mechanical.

Given m p-values and a target FDR level q (say 0.10):

Sort the p-values in ascending order, so p(1) <= p(2) <= ... <= p(m).
For each rank k, compute the BH critical value (k / m) * q.
Find the largest rank k for which p(k) <= (k / m) * q.
Declare significant every hypothesis from rank 1 up to and including that k — even any whose raw p-value sits above the naive threshold.

The key idea is the sloped threshold. Instead of comparing every p-value to one fixed cutoff, BH compares each ranked p-value to a line that rises with its rank. A p-value only needs to beat (k / m) * q, which is more forgiving for higher ranks when there are many small p-values supporting them. The procedure adapts: when the data contain many strong effects, the effective cutoff loosens; when almost nothing is real, it tightens toward the strict Bonferroni bound.

flowchart TD
    A[Collect m p-values from every metric and variation] --> B[Sort ascending: p1 to pm]
    B --> C[For each rank k compute BH critical value k over m times q]
    C --> D{Find largest k where p_k is at or below its BH critical value}
    D --> E[Reject ranks 1 through k: declare significant]
    D --> F[Ranks above k: not significant]
    E --> G[Expected false positives among rejections stays at or below q]

A Worked Example

Take 10 metrics on an experiment and a target FDR of q = 0.10. Sort the p-values ascending, compute each BH critical value (k / 10) * 0.10 = k * 0.01, and compare:

Rank k   p-value    BH critical (k/m)*q    p <= critical?
  1      0.001         0.010                yes
  2      0.008         0.020                yes
  3      0.012         0.030                yes
  4      0.021         0.040                yes
  5      0.030         0.050                yes
  6      0.041         0.060                yes
  7      0.060         0.070                yes   <-- largest passing rank
  8      0.200         0.080                no
  9      0.350         0.090                no
 10      0.600         0.100                no

Largest k with p(k) <= (k/m)*q  is  k = 7.
Benjamini-Hochberg rejects ranks 1 through 7  ->  7 discoveries.

Notice rank 7: its p-value is 0.060, which is above the naive 0.05 threshold, yet BH still declares it significant. That is the procedure working as designed — the weight of six even-smaller p-values below it earns rank 7 a place. Now compare the same data under a strict Bonferroni correction, which would test every metric against 0.10 / 10 = 0.01: only ranks 1 (0.001) and 2 (0.008) clear that bar, for 2 discoveries instead of 7. Same data, same 10% error budget — BH finds more than three times as many real effects because it spends that budget across the whole set instead of on each test in isolation.

FDR Control vs Bonferroni Correction

The Bonferroni correction is the best-known multiple-comparisons fix: divide your significance level by the number of tests, so with 20 metrics at 5% each metric must clear 0.05 / 20 = 0.0025. It controls the family-wise error rate, and it is trivially simple. It is also badly suited to A/B testing.

	Bonferroni	Benjamini-Hochberg (FDR)
Controls	Family-wise error rate	False discovery rate
Threshold	Fixed: `alpha / m` for every test	Sloped: `(k / m) * q`, rank-dependent
As metrics grow	Cutoff shrinks fast; power collapses	Cutoff adapts; power preserved
False negatives	Many — misses real winners	Far fewer
Right context	A few make-or-break tests	Many metrics, exploratory testing

Bonferroni's flaw in this setting is that it treats every additional metric as equally threatening and pays for it with a cutoff that shrinks toward zero. With 50 metrics, each must beat 0.001 — so real, moderate effects get thrown out alongside the noise, and your false-negative rate soars. FDR control accepts a small, known fraction of false positives in exchange for keeping the power to detect the effects that matter. For an experimentation program that lives or dies by finding true wins across a broad metric set, that is the correct trade.

How Optimizely's Stats Engine Controls the False Discovery Rate

Optimizely's Stats Engine applies false discovery rate control automatically. Per Optimizely's documentation, the Stats Engine "provides a data-rich view of visitor interactions, includes confidence intervals, and applies false discovery rate control" across the metrics on the Experiment Results page. You do not configure it, sort p-values, or apply a correction by hand — it is built into how significance is reported. For the broader mechanics of how significance and always-valid inference are computed, see How the Optimizely Stats Engine Works.

Two documented properties are worth internalizing:

Testing more does not raise your false-positive risk. Optimizely states that the "Stats Engine adjusts statistical significance for every metric and variation, so testing more does not raise the risk of false positives." That is the FDR guarantee in plain language — the correction scales with how many comparisons the results page carries, so adding a guardrail does not silently inflate your chance of a phantom winner.
The primary metric is protected. On the results page, "the primary metric's statistical significance is calculated independently from all other metrics and monitoring goals." The tiering matters: your headline result reaches significance as fast as possible, while secondary and monitoring metrics absorb the multiple-comparisons adjustment. Adding many low-improvement secondary metrics slows their time to significance, but never the primary's.

This is why the Change the statistical significance setting documentation can promise that carrying a wide metric set — including a full complement of guardrail metrics — does not cost you in false alarms. FDR control is what makes a broad, honest results page statistically safe.

Do Not Change Metrics Mid-Experiment

There is one operational rule that follows directly from how FDR control is computed. Optimizely controls the false discovery rate over "metrics currently appearing on the Experiment Results page." Its Edit a metric documentation warns that changing the metrics during a running experiment "alters the number and nature of the hypotheses being tested, which can affect the false discovery rate control calculations," and that "excessive metric changes after an experiment or campaign has started may invalidate Optimizely's guarantees regarding false detection."

The reason is exactly the BH mechanism above: the critical values depend on m, the number of hypotheses. Add or swap metrics after data has started arriving and you retroactively change m, which shifts every threshold and reopens the door to the cherry-picking FDR control was meant to close. Decide your metric set before launch and freeze it — the same discipline that protects you from the winner's curse and from sequential testing peeking traps.

Practical Implications for Your Experiments

Understanding FDR control changes a few concrete habits:

Keep the metrics you need, not fewer. Because the Stats Engine controls FDR across the results page, you can carry a healthy set of guardrails and secondaries without inflating false positives. Trim metrics for clarity, not out of statistical fear.
Lock the metric set at launch. Write your primary, secondary, and guardrail metrics into the experiment brief before traffic starts, and do not edit them mid-flight. Changing m mid-experiment undermines the FDR guarantee.
Read the primary metric first. It is computed independently and reaches significance fastest. Treat secondary and monitoring metrics as supporting evidence whose significance the FDR adjustment deliberately makes more conservative.
Power the experiment for the metrics that matter. FDR control makes false positives rare, but it cannot rescue an underpowered test from false negatives. Size the experiment properly — see A/B test sample size and statistical power — so you can actually detect the effects you care about.
Validate the traffic split. No error-rate control matters if the experiment is broken at the instrumentation level. Check for sample ratio mismatch before you trust any significance number on the page.

Frequently Asked Questions

What is a good false discovery rate to target?

For most online experimentation, controlling the FDR at 5% to 10% is standard — the same range as conventional significance levels. A 10% FDR means you accept that, on average, up to one in ten of your flagged discoveries may be a false positive, in exchange for substantially more power to detect real effects. Optimizely ties this to your project-wide statistical significance setting rather than asking you to specify an FDR directly.

Is the false discovery rate the same as the false positive rate?

No, and the distinction is important. The false positive rate (or Type I error rate) is the probability of a false positive among all truly null tests — it looks backward from the ground truth. The false discovery rate is the expected fraction of false positives among the results you declared significant — it looks forward from your discoveries. FDR is the more actionable quantity because it describes the reliability of the conclusions you actually act on.

Why not just use Bonferroni correction?

Bonferroni controls the family-wise error rate by dividing your significance level across all tests, which makes the threshold shrink rapidly as you add metrics. With many metrics it becomes so conservative that it discards real effects along with the noise, driving up false negatives. Benjamini-Hochberg controls the false discovery rate instead, adapting the threshold to the data and preserving far more power — a better fit for the many-metric reality of A/B testing.

Does tracking more metrics in Optimizely increase my false positive risk?

No. Optimizely's documentation states that the Stats Engine "adjusts statistical significance for every metric and variation, so testing more does not raise the risk of false positives." The false discovery rate control scales with the number of comparisons on the results page. Adding metrics can lengthen the time for secondary metrics to reach significance, but it does not inflate your overall false-positive risk.

Why does changing metrics mid-experiment break FDR control?

Benjamini-Hochberg critical values depend on the number of hypotheses being tested. When you add, remove, or edit metrics on a running experiment, you change that number retroactively, which shifts the thresholds the Stats Engine already applied and can invalidate Optimizely's guarantees against false detection. Decide your metric set before launch and keep it fixed for the life of the experiment.

Experiment Design for A/B Testing: A Practical Framework

David Sert — Thu, 20 Aug 2026 05:05:59 +0000

An A/B test can be implemented perfectly and still answer the wrong question. Experiment design is the written contract that connects a business decision to a causal comparison: who is eligible, what is randomized, what changes, how outcomes are measured, and what evidence will trigger action.

This guide turns those choices into a one-page specification, then maps the same design to Optimizely Web Experimentation and Feature Experimentation. It complements the deeper guides to an A/B testing hypothesis, experiment metrics, sample size and statistical power, test duration, and pre-launch QA.

Experiment design is the blueprint for a causal answer

The output of experiment design is not a list of variation ideas. It is a precise description of the comparison that will support a decision. Optimizely's own basic experiment plan asks why the experiment runs, where and when variations run, who sees them, what changes, and how success is measured. A rigorous design adds the unit of randomization, estimand, power assumptions, and decision rule.

flowchart LR
  A[Business decision] --> B[Causal question]
  B --> C[Unit and eligibility]
  C --> D[Control and treatment]
  D --> E[Metrics and estimand]
  E --> F[MDE, sample, and duration]
  F --> G[Optimizely configuration]
  G --> H[QA and launch]
  H --> I[Ship, iterate, stop, or conclude]

Start with the decision the result must support

Write the action before the hypothesis. A useful decision is specific enough that a result can change it:

Ship the checkout recommendation module to all eligible customers if it produces a practically valuable increase in completed orders without materially increasing refund rate; otherwise iterate or stop.

Turn that decision into one falsifiable causal question:

Among eligible signed-in shoppers, what is the effect of showing the recommendation module during checkout, compared with the current checkout, on completed orders within seven days of assignment?

This order prevents a common failure: designing an attractive variation, choosing an available metric, and only later deciding what the test was meant to prove.

Define the estimand before the dashboard metric

The estimand is the effect the team wants to estimate. Define five elements:

Element	Checkout recommendation example
Population	Signed-in shoppers who begin checkout in supported markets
Treatment	Checkout with the recommendation module enabled
Comparator	Current production checkout with the module absent
Outcome	Orders completed within seven days of assignment
Time window	Seven days from the first eligible checkout

Use an intention-to-treat estimand by default: analyze people according to the variation to which they were assigned, even if rendering fails or they do not engage with the module. This preserves the benefit of random assignment and estimates the effect of the policy as deployed.

An exposure-only analysis answers a different question because it conditions on behavior or technical events that may happen after assignment. It can be useful as a diagnostic, but it should not silently replace the primary analysis.

Know when an experiment is the wrong tool

Do not randomize merely because the platform makes it possible. An experiment is a poor choice when:

No plausible outcome would change the decision.
Available traffic cannot detect an effect worth acting on within a useful timeframe.
The treatment cannot be assigned independently enough to create a credible control.
Random assignment would be unethical, unlawful, or operationally unacceptable.
Instrumentation cannot distinguish assignment, exposure, and outcomes reliably.

When assignment cannot be randomized but a decision still needs evidence, use a quasi-experimental design and state its assumptions explicitly. That design should not be presented as equivalent to a randomized test.

Choose the experimental unit and assignment mechanism

The experimental unit is the entity independently assigned to control or treatment. It determines what must remain stable, where interference can occur, and which observations are independent in the analysis.

User, session, account, device, or cluster

Match the unit to how the treatment is delivered and how people can influence one another:

Unit	Appropriate when	Main risk
User	A durable identity receives an individual experience	The same person appears under multiple IDs
Session	The treatment is intentionally temporary and session-contained	Repeat sessions receive inconsistent experiences
Account	Members share settings, workflows, or outcomes	Too few accounts for adequate power
Device	The experience is device-specific and identity is unavailable	Cross-device contamination
Cluster	Locations, teams, or networks can affect one another	Correlated outcomes reduce effective sample size

For a B2B account-level recommendation feature, randomizing individual members can contaminate the comparison: one member may configure the experience for colleagues, or team members may discuss it. Assign the account, then analyze with methods that respect account-level assignment.

Keep assignment stable

Use a durable identifier for the unit of randomization. If the treatment must persist across browser sessions and services, a transient session ID is not sufficient. Document how anonymous identities merge with authenticated identities and what happens when a user changes devices.

Optimizely Web Experimentation uses deterministic bucketing: it hashes a user ID or cookie with MurmurHash, so the same identifier maps to the same variation while the relevant configuration remains stable. Optimizely also warns that some traffic changes can rebucket visitors; the official bucketing guide recommends avoiding allocation changes after results accumulate.

The randomization unit is not necessarily the metric denominator. A user-randomized experiment may measure revenue per eligible user, orders per user, or average order value among orders. Each denominator answers a different question and must be fixed in the measurement plan.

Detect spillovers and network effects

Random assignment does not prevent treated units from changing control units' outcomes. Before launch, ask:

Can users communicate about or share the treatment?
Does treatment change inventory, marketplace liquidity, queue time, or prices for everyone?
Can one account member alter settings seen by other members?
Does a recommender learn from behavior pooled across variations?

If interference is plausible, assign clusters that contain the spillover, use a switchback or marketplace-specific design, or redefine the estimand. Record the remaining interference risk rather than assuming it away.

Define control, treatment, and eligibility precisely

A reproducible comparison needs versioned experiences and frozen entry rules. “Business as usual” and “all users” are not adequate specifications.

Make the control a real production baseline

Describe the control as thoroughly as the treatment:

User interface and copy.
Code path and service dependencies.
Default feature-flag values.
Existing personalization or recommendation logic.
Analytics events expected from the baseline.

For the checkout example, the control is “the current checkout with no recommendation component and no recommendation API call,” not simply “original.” This matters if production changes while the experiment is being prepared.

Change one causal idea at a time

One hypothesis can require several implementation edits. Adding a recommendation module may change layout, make an API request, add loading behavior, and introduce new tracking. Those edits express one causal idea if the intended decision is whether to deploy the module as a package.

If the decision is which component causes the effect—placement, copy, image, or algorithm—use separate experiments or a multivariate test. A bundled treatment cannot identify the effect of each ingredient.

Freeze inclusion and exclusion rules

Define eligibility before anyone inspects results:

Target pages, routes, or flag call sites.
Geography, language, device, browser, and app version.
Account tier, authentication state, and product entitlement.
Prior exposure and concurrent-experiment exclusions.
Employee, bot, fraud, and QA-traffic handling.
The exact moment a unit enters the experiment.

Changing eligibility after viewing outcomes creates a new analysis population. If a safety issue forces a change, record the time and configuration, then decide whether to restart rather than blending incompatible cohorts.

Design the measurement plan

Metrics operationalize the estimand. They should be specified as formulas and event rules, not only selected by name in a dashboard.

Choose one primary metric

The primary metric must be affected by the treatment, interpretable by stakeholders, and tied to the ship decision. Define:

Numerator and denominator.
Aggregation unit.
Attribution or conversion window.
Event source and deduplication key.
Direction of improvement.
Practical effect threshold.

For the checkout example, “conversion” is ambiguous. A better definition is “distinct eligible users with at least one completed order within seven days divided by distinct assigned eligible users.”

Optimizely Web Experimentation treats the first metric added as the primary metric; its experiment setup guide recommends choosing an event directly affected by the change. Put the decision metric first deliberately.

Add restrained secondary and guardrail metrics

Secondary metrics explain mechanism or downstream impact. For the recommendation module, useful secondary measures might include recommendation clicks, units per order, and revenue per eligible user.

Guardrails describe harms that can veto a nominal win: refund rate, checkout latency, payment errors, support contacts, or gross margin. State each veto rule in advance. A dashboard containing dozens of metrics invites selective interpretation; include only metrics with a defined role.

Instrument assignment and outcomes separately

An outcome event is not proof that assignment or exposure worked. Validate three layers:

Assignment: the unit was bucketed into a variation.
Exposure: the intended experience was actually delivered.
Outcome: the conversion or guardrail event occurred and was attributed correctly.

Specify how the pipeline handles duplicate events, bots, missing identifiers, delayed events, identity merges, and events arriving after the experiment stops. Run an A/A or staged validation when the cost of a broken decision pipeline is high.

Design for enough information, not merely significance

A design should collect enough information to make the business decision. Statistical significance alone does not establish that an effect is large enough, safe enough, or measured without bias.

Pick an MDE worth acting on

The minimum detectable effect (MDE) should reflect the smallest effect that would justify implementation and opportunity cost. Derive it from baseline performance, expected value, engineering cost, risk, and competing roadmap items.

Do not inflate the MDE simply to make the required sample fit available traffic. That creates a faster test by declaring smaller—but potentially valuable—effects invisible.

Calculate sample size and calendar duration

Use the baseline rate, MDE, significance threshold, desired power, and number of variations to estimate the required sample. Then convert eligible sample into calendar time using:

The share of traffic satisfying eligibility.
Total experiment allocation.
Variation traffic distribution.
Weekly and seasonal cycles.
Ramp time and planned outages.
Conversion delay and data-processing latency.

The sample-size guide covers the calculation; the design document should retain the inputs and result. The duration guide explains why reaching a nominal sample early does not erase calendar-cycle or delayed-conversion requirements.

Predefine the stopping and decision rules

Write the complete rule before launch:

Statistical method and confidence policy.
Minimum runtime and information requirement.
Practical-effect threshold or acceptable interval.
Guardrail vetoes.
Safety-stop conditions.
Treatment of inconclusive results.

Do not stop because a preferred variation briefly looks significant, change the primary metric after seeing results, or segment repeatedly until a favorable result appears. Optimizely Stats Engine supports ongoing results monitoring, but operational visibility is not permission to rewrite the design after outcomes are known.

Map the experiment design into Optimizely

The same causal design can be implemented in either Web Experimentation or Feature Experimentation. The platform configuration enforces parts of the design; it does not choose the question, unit, estimand, MDE, or decision threshold for the team.

Start from one design card

Use a one-page design card as the source of truth:

Field	Checkout recommendation design
Decision	Ship, iterate, or stop the module
Causal question	Effect of offering recommendations on seven-day order completion
Population	Signed-in checkout starters in supported markets
Randomization unit	Durable user ID
Control	Current checkout without recommendation call or component
Treatment	Checkout with recommendation module
Primary metric	Seven-day completed-order users per assigned user
Secondary metrics	Recommendation clicks, units per order, revenue per user
Guardrails	Refund rate, checkout errors, p95 latency
Allocation	100% of eligible traffic
Distribution	50% control, 50% treatment
Runtime	Precomputed minimum sample plus complete weekly cycles
Decision rule	Practical lift and statistical evidence with no guardrail veto

flowchart TB
  D[Shared experiment design card]
  D --> W1[Web: URL or saved page]
  D --> W2[Web: audience]
  D --> W3[Web: original and variation]
  D --> W4[Web: allocation, distribution, metrics]
  D --> F1[Feature: flag and variables]
  D --> F2[Feature: attributes and audience]
  D --> F3[Feature: A/B test rule]
  D --> F4[Feature: decide, events, metrics]

Web Experimentation mapping

The current official Web Experimentation workflow maps the design as follows:

Design field	Web Experimentation object
Where treatment can occur	Target By URL or Target By Saved Pages; page triggers and conditions where needed
Eligible population	Audience conditions
Control and treatment	Original and variation under Design > Variations
Random assignment	Design > Traffic Allocation
Primary and supporting outcomes	Track > Metrics
Pre-launch evidence	Preview, testing, Summary, and pre-launch review

Set total traffic allocation to 100% when every eligible visitor should enter the experiment, then distribute that traffic 50/50 between original and treatment. Optimizely distinguishes the eligible share entering the test (allocation) from the share assigned to each variation (distribution) in its traffic documentation. Freeze both before launch.

Feature Experimentation mapping

Feature Experimentation implements the same design in application code and a flag ruleset:

Design field	Feature Experimentation object
Where treatment can occur	Flag evaluation call site
Control and treatment	Flag variations and variables
Eligible population	User-context attributes and audience
Random assignment	A/B Test flag rule, allocation, and distribution mode
Exposure	SDK `decide` call and its decision event
Outcomes	Tracked events configured as metrics

Optimizely's official Feature Experimentation getting-started guide describes the sequence: create a flag, implement it with decide, define an audience and flag rule, then analyze the Experiment Results page. The newer A/B test configuration guide adds the hypothesis, test plan, traffic allocation, metrics, distribution mode, and flag variations.

const user = optimizelyClient.createUserContext(
  durableUserId,
  {
    market: checkout.market,
    account_tier: checkout.accountTier,
  },
)

const decision = user.decide('checkout_recommendations')

if (decision.variationKey === 'treatment') {
  renderCheckout({ recommendations: decision.variables })
} else {
  renderCheckout({ recommendations: null })
}

The durable user ID implements user-level assignment. Attributes implement declared eligibility. The flag key defines the decision point, while the returned variation selects the control or treatment code path. Track the completed-order event with the same user context so the result can be attributed to the decision.

Preserve the analysis contract in Collaboration

If the organization uses Optimizely Collaboration, store the structured design alongside the work. The official hypothesis documentation provides a Brief tab for a structured plan, a Variations tab for images or links, an Experiment tab for linking the implementation, and History for an activity trail.

Attach the design card, metric definitions, MDE inputs, QA evidence, risks, and assumptions. Collaboration can preserve the contract and make changes visible; it does not repair a design that was never specified.

Run a pre-launch design review

Review the design before launch, when weaknesses can still be fixed without contaminating results. Separate validity from feasibility so “we can build it” is not mistaken for “it will answer the question.”

Validity review

Challenge the causal comparison:

Selection: Are eligibility rules evaluated before treatment, and are exclusions symmetrical?
Spillover: Can treated units alter control outcomes?
Attrition: Can missing outcomes differ by variation?
Novelty: Could a temporary reaction dominate the intended long-run effect?
Instrumentation: Can assignment, exposure, and outcomes be reconciled?
Concurrent tests: Can another experiment interact with this treatment or metric?
Identity: Can one randomization unit receive multiple IDs or variations?

Record mitigations and residual risks. Randomization balances expected pre-treatment differences; it does not fix differential loss, broken event collection, or post-assignment filtering.

Feasibility review

Confirm the operating plan:

Required sample and realistic calendar duration.
Engineering, design, analytics, and QA ownership.
Event latency and conversion window.
Rollout and rollback procedure.
Safety monitoring and incident owner.
Browser, device, and environment coverage.
Production baseline freeze and change control.

If the design cannot be executed as written, revise it before exposure rather than improvising after launch.

One-page experiment-design checklist

Decision, causal question, and hypothesis are explicit.
Population, randomization unit, and analysis unit are distinct.
Control and treatment are versioned production specifications.
Eligibility, exclusions, and prior-exposure rules are frozen.
One primary metric has a complete formula and conversion window.
Secondary metrics explain mechanism; guardrails have veto rules.
Assignment, exposure, and outcome events can be reconciled.
MDE, sample, duration, and stopping rule are recorded.
Optimizely pages or flags, audiences, traffic, variations, and metrics match the design.
QA evidence, rollback owner, and launch approval are documented.

Use the Optimizely experiment QA checklist to turn the approved design into a production launch review.

Frequently asked questions

What is experiment design in A/B testing?

Experiment design is the specification that connects a decision to a causal comparison. It defines the population, randomization unit, eligibility, control, treatment, metrics, power assumptions, duration, and decision rule before traffic is exposed.

Is the randomization unit the same as the analysis unit?

Not always. A test can randomize accounts and observe users or transactions, but the analysis must account for dependence within each randomized account. Treating correlated observations as independently randomized overstates the amount of information in the test.

Should an A/B test change only one page element?

It should test one causal idea, which may require several coordinated implementation changes. If the decision requires isolating the contribution of individual elements or their interactions, use separate experiments or a multivariate design.

Can traffic allocation change after an experiment starts?

Operationally, Optimizely permits traffic changes, but they can affect bucketing and interpretation. Predefine allocation and distribution, avoid changing them after results accumulate, and document any safety-driven intervention.

Does Optimizely replace an experiment design document?

No. Optimizely implements pages or flags, audiences, variations, traffic rules, and metrics. The team must still define the causal question, unit, estimand, MDE, duration, risks, and action threshold. The configuration should be a faithful implementation of that prior design.

Use a multivariate test when one decision depends on the separate effect of multiple components.