A/B testing was built for a web that no longer exists

A/B testing is one of the most successful ideas in web development. It replaced opinion with evidence. It made "I think users prefer the green button" into a testable claim. Nobody sane wants to go back to shipping on vibes.

So this isn't a "A/B testing is dumb" post. A/B testing is a good tool with a narrow set of assumptions baked into it — and those assumptions came from the web of 2010. Three of them are quietly breaking. One of them broke this year.

Let's look at the assumptions, because once you see them, you can't unsee them.

Assumption 1: there is an "average visitor" worth optimizing for

An A/B test asks: which variant wins? Singular. You run A against B, you get a winner, you ship the winner to everyone.

But "the winner" is the variant that performed best averaged across your entire traffic mix. And your traffic isn't one audience — it's a pile of them stacked on top of each other:

• The developer who came from Hacker News and reads every word
• The buyer who came from a LinkedIn ad and wants the pricing in 4 seconds
• The returning user who already knows what you do
• The person on a 3G phone in a different timezone

A/B testing collapses all of them into one number and picks the variant with the best mean. That's optimizing for the average visitor — and the average visitor doesn't exist. It's the Norden bomber cockpit designed for the average pilot that fit nobody. You can win the A/B test and still show every real segment a worse experience than they could have had.

The moment you accept "different visitors want different things," a single winner stops being the goal. What you actually want is the right variant conditioned on who showed up.

Assumption 2: you have enough traffic to reach significance

Here's the part nobody likes to say out loud: most A/B tests never reach statistical significance, and teams ship anyway.

The sample size you need scales brutally with how small the effect is. Detecting a lift from a 3% baseline conversion rate to 3.3% (a real, respectable 10% relative lift), at 80% power and 95% confidence, needs roughly:

n ≈ 16 * p(1-p) / (MDE)²
  ≈ 16 * 0.03 * 0.97 / (0.003)²
  ≈ ~51,000 visitors per variant

That's ~100k visitors for one test of one change. Most sites don't get that in a month. So what actually happens:

• The test runs for two weeks
• The dashboard shows B is "up 12%"
• The p-value is 0.34
• Someone says "close enough, ship it"

Congratulations, you just shipped noise and wrote it down as data. And you'll do it again next sprint, because the ritual feels rigorous even when the math isn't there. Underpowered testing isn't better than intuition — it's intuition wearing a lab coat.

Assumption 3: the web moves slowly enough to test episodically

A/B testing is a batch process. Set up the experiment, freeze the variants, wait for the calendar to grant you significance, pick a winner, tear it down. Weeks per decision.

Two problems with batch in 2026:

The regret problem. During the entire test window you keep sending ~50% of traffic to the variant you increasingly suspect is worse. Even after the evidence is piling up, the design of the test forces you to keep paying that tax until the end date. That wasted traffic has a name in the literature — regret — and A/B testing maximizes it by construction.

The staleness problem. The winner you crown is optimal for the traffic mix you had during the test. Then your Product Hunt launch lands, or a campaign shifts your source mix, or seasonality hits — and your "winner" is now tuned for an audience you no longer have. You won't find out until the next quarterly test.

The alternative here isn't new, it's just underused on the frontend: multi-armed bandits. Instead of a fixed split, you shift traffic toward what's winning as you learn, keeping a little exploration alive:

// A/B test: fixed 50/50 the whole way, decide at the end
const variant = Math.random() < 0.5 ? "A" : "B";

// Bandit: allocation follows the evidence, continuously
function pickVariant(stats: Record<string, {wins: number; trials: number}>) {
  // Thompson sampling: draw from each variant's Beta posterior, take the best draw
  return Object.entries(stats)
    .map(([name, s]) => [name, sampleBeta(1 + s.wins, 1 + (s.trials - s.wins))] as const)
    .sort((a, b) => b[1] - a[1])[0][0];
}

Same evidence, far less regret, and it never "ends" — it just keeps adapting. The batch experiment becomes a continuous process.

The assumption that broke this year: everyone visiting is a human

Every assumption above still lives in a world where a person lands on your page, forms an impression, and decides. A/B testing is fundamentally a theory of human behavior over sessions.

Then a meaningful slice of your traffic stopped being human.

If you've looked at your logs lately you've seen it: GPTBot, ChatGPT-User, Claude-User, PerplexityBot, and a rising tide of agents fetching your pages — often without running your JavaScript — to answer a question on someone's behalf. They don't scroll. They don't A/B into a funnel. They don't come back on visit two and convert. They read, extract, and leave. Your carefully-tested hero headline may never be rendered for them at all.

This breaks A/B testing in a way you can't patch:

• You can't measure them the same way. No session, no funnel, often no JS, so no client-side experiment framework even fires.
• You can't optimize the same thing. "Which button converts better" is meaningless to an agent extracting facts. The question becomes "is my content legible and answerable?" — a completely different objective.
• They're invisible in your current tooling. Most analytics silently drop no-JS agent traffic, so the fastest-growing segment of your visitors doesn't even show up in the data you're A/B testing against.

You cannot A/B test your way to good agent experiences, because the entire premise — a human making a decision you can nudge — isn't there. You need to (a) see agent traffic as a first-class segment and (b) optimize for a different goal for it than you do for humans.

So what replaces it?

Not "nothing" — and not "one clever tool." The shift is from testing (find the single winner, ship it to all) to adapting (serve the right experience conditioned on who — or what — showed up):

• Optimize per segment, not per average. Different personas, sources, and devices can converge on different variants.
• Allocate traffic continuously toward what's working, instead of freezing a split for three weeks.
• Treat agents as their own segment with their own objective — legibility, not clicks.

Concretely, that looks less like "set up experiment #47" and more like declaring what's adaptive and letting the system converge:

// You mark what can vary and what "good" means.
// The system decides per visitor and keeps learning — humans and agents included.
<Adaptive goal="signup">
  <Hero variants={["proof-first", "speed-first", "developer-first"]} />
</Adaptive>

The honest version of the argument

A/B testing isn't obsolete. If you have huge traffic and one high-stakes, all-human decision — a checkout flow on a site doing millions of sessions — a clean A/B test is still the right, rigorous call. Keep it there.

But if you're like most teams — not enough traffic to reach significance, multiple audiences hiding inside one funnel, and a growing share of visitors who are software, not people — then the tool is being asked to hold up assumptions it was never built for. That's not a knock on A/B testing. It's just that it was designed for a web that is quietly, permanently, changing underneath it.

The web got heterogeneous, continuous, and non-human. Our optimization tools should too.

---

I'm building SentientUI around exactly this shift — adaptive interfaces instead of episodic tests, with agent traffic as a first-class segment. The SDKs are open source if you want to see how the adaptation and the agent-legibility parts actually work under the hood. Curious what your logs look like — how much of your traffic is already non-human? Drop a number in the comments.