DEV Community: SentientUI

A/B testing was built for a web that no longer exists

SentientUI — Tue, 04 Aug 2026 10:50:01 +0000

A/B testing is one of the most successful ideas in web development. It replaced opinion with evidence. It made "I think users prefer the green button" into a testable claim. Nobody sane wants to go back to shipping on vibes.

So this isn't a "A/B testing is dumb" post. A/B testing is a good tool with a narrow set of assumptions baked into it — and those assumptions came from the web of 2010. Three of them are quietly breaking. One of them broke this year.

Let's look at the assumptions, because once you see them, you can't unsee them.

Assumption 1: there is an "average visitor" worth optimizing for

An A/B test asks: which variant wins? Singular. You run A against B, you get a winner, you ship the winner to everyone.

But "the winner" is the variant that performed best averaged across your entire traffic mix. And your traffic isn't one audience — it's a pile of them stacked on top of each other:

The developer who came from Hacker News and reads every word
The buyer who came from a LinkedIn ad and wants the pricing in 4 seconds
The returning user who already knows what you do
The person on a 3G phone in a different timezone

A/B testing collapses all of them into one number and picks the variant with the best mean. That's optimizing for the average visitor — and the average visitor doesn't exist. It's the Norden bomber cockpit designed for the average pilot that fit nobody. You can win the A/B test and still show every real segment a worse experience than they could have had.

The moment you accept "different visitors want different things," a single winner stops being the goal. What you actually want is the right variant conditioned on who showed up.

Assumption 2: you have enough traffic to reach significance

Here's the part nobody likes to say out loud: most A/B tests never reach statistical significance, and teams ship anyway.

The sample size you need scales brutally with how small the effect is. Detecting a lift from a 3% baseline conversion rate to 3.3% (a real, respectable 10% relative lift), at 80% power and 95% confidence, needs roughly:

n ≈ 16 * p(1-p) / (MDE)²
  ≈ 16 * 0.03 * 0.97 / (0.003)²
  ≈ ~51,000 visitors per variant

That's ~100k visitors for one test of one change. Most sites don't get that in a month. So what actually happens:

The test runs for two weeks
The dashboard shows B is "up 12%"
The p-value is 0.34
Someone says "close enough, ship it"

Congratulations, you just shipped noise and wrote it down as data. And you'll do it again next sprint, because the ritual feels rigorous even when the math isn't there. Underpowered testing isn't better than intuition — it's intuition wearing a lab coat.

Assumption 3: the web moves slowly enough to test episodically

A/B testing is a batch process. Set up the experiment, freeze the variants, wait for the calendar to grant you significance, pick a winner, tear it down. Weeks per decision.

Two problems with batch in 2026:

The regret problem. During the entire test window you keep sending ~50% of traffic to the variant you increasingly suspect is worse. Even after the evidence is piling up, the design of the test forces you to keep paying that tax until the end date. That wasted traffic has a name in the literature — regret — and A/B testing maximizes it by construction.

The staleness problem. The winner you crown is optimal for the traffic mix you had during the test. Then your Product Hunt launch lands, or a campaign shifts your source mix, or seasonality hits — and your "winner" is now tuned for an audience you no longer have. You won't find out until the next quarterly test.

The alternative here isn't new, it's just underused on the frontend: multi-armed bandits. Instead of a fixed split, you shift traffic toward what's winning as you learn, keeping a little exploration alive:

// A/B test: fixed 50/50 the whole way, decide at the end
const variant = Math.random() < 0.5 ? "A" : "B";

// Bandit: allocation follows the evidence, continuously
function pickVariant(stats: Record<string, {wins: number; trials: number}>) {
  // Thompson sampling: draw from each variant's Beta posterior, take the best draw
  return Object.entries(stats)
    .map(([name, s]) => [name, sampleBeta(1 + s.wins, 1 + (s.trials - s.wins))] as const)
    .sort((a, b) => b[1] - a[1])[0][0];
}

Same evidence, far less regret, and it never "ends" — it just keeps adapting. The batch experiment becomes a continuous process.

The assumption that broke this year: everyone visiting is a human

Every assumption above still lives in a world where a person lands on your page, forms an impression, and decides. A/B testing is fundamentally a theory of human behavior over sessions.

Then a meaningful slice of your traffic stopped being human.

If you've looked at your logs lately you've seen it: GPTBot, ChatGPT-User, Claude-User, PerplexityBot, and a rising tide of agents fetching your pages — often without running your JavaScript — to answer a question on someone's behalf. They don't scroll. They don't A/B into a funnel. They don't come back on visit two and convert. They read, extract, and leave. Your carefully-tested hero headline may never be rendered for them at all.

This breaks A/B testing in a way you can't patch:

You can't measure them the same way. No session, no funnel, often no JS, so no client-side experiment framework even fires.
You can't optimize the same thing. "Which button converts better" is meaningless to an agent extracting facts. The question becomes "is my content legible and answerable?" — a completely different objective.
They're invisible in your current tooling. Most analytics silently drop no-JS agent traffic, so the fastest-growing segment of your visitors doesn't even show up in the data you're A/B testing against.

You cannot A/B test your way to good agent experiences, because the entire premise — a human making a decision you can nudge — isn't there. You need to (a) see agent traffic as a first-class segment and (b) optimize for a different goal for it than you do for humans.

So what replaces it?

Not "nothing" — and not "one clever tool." The shift is from testing (find the single winner, ship it to all) to adapting (serve the right experience conditioned on who — or what — showed up):

Optimize per segment, not per average. Different personas, sources, and devices can converge on different variants.
Allocate traffic continuously toward what's working, instead of freezing a split for three weeks.
Treat agents as their own segment with their own objective — legibility, not clicks.

Concretely, that looks less like "set up experiment #47" and more like declaring what's adaptive and letting the system converge:

// You mark what can vary and what "good" means.
// The system decides per visitor and keeps learning — humans and agents included.
<Adaptive goal="signup">
  <Hero variants={["proof-first", "speed-first", "developer-first"]} />
</Adaptive>

The honest version of the argument

A/B testing isn't obsolete. If you have huge traffic and one high-stakes, all-human decision — a checkout flow on a site doing millions of sessions — a clean A/B test is still the right, rigorous call. Keep it there.

But if you're like most teams — not enough traffic to reach significance, multiple audiences hiding inside one funnel, and a growing share of visitors who are software, not people — then the tool is being asked to hold up assumptions it was never built for. That's not a knock on A/B testing. It's just that it was designed for a web that is quietly, permanently, changing underneath it.

The web got heterogeneous, continuous, and non-human. Our optimization tools should too.

I'm building SentientUI around exactly this shift — adaptive interfaces instead of episodic tests, with agent traffic as a first-class segment. The SDKs are open source if you want to see how the adaptation and the agent-legibility parts actually work under the hood. Curious what your logs look like — how much of your traffic is already non-human? Drop a number in the comments.

The Interface Is Not The Product. It's The Output

SentientUI — Thu, 02 Jul 2026 10:13:20 +0000

There is an assumption buried so deep in how we build software that it rarely gets questioned. The assumption is that an interface is something you make. You sit down, you make decisions. What goes where, what colour, what size, what order... Those decisions get encoded into a codebase that serves them to everyone who arrives, identically, until someone decides to make different decisions.

This is not how most things that learn work. A teacher doesn't deliver the same lesson to every student in the same order with the same emphasis regardless of what they already know. A good doctor doesn't give every patient the same examination in the same sequence regardless of what they came in for. But software has always worked this way, one interface, authored once, served to everyone, and we've never really interrogated whether that's the best we can do or just the best we knew how to do.

We think it's the latter. And we think that's changing.

The Logic Knows Everything. The Visuals Know Nothing About You

The most interesting thing about how interfaces are built today is what they don't separate.

A React component is a function that contains both. It knows what it does: what state it holds, what events it fires, what data it needs. And it knows what it looks like: the markup, the styling, the layout. These two things are compiled together into the same artifact. You can't have one without the other. The logic and the visuals are inseparable by design.

This coupling made sense when interfaces were simple and when the people building them were also the people using them. When you're building for yourself, your decisions about how something looks and how it behaves can live in the same place without friction. But as interfaces scaled to billions of users across thousands of different contexts, the coupling started to create a problem that nobody quite named.

The logic of an interface, what it does, what it knows, what it responds to, is stable across people. A checkout flow adds items to a cart for everyone. A search bar takes queries from everyone. A dashboard displays data to everyone.

The visuals, what those things look like, in what order they appear, how much prominence each element gets, what the flow emphasises and what it minimises, are not stable across people. They're stable across decisions made by one person, at one time, for a user they imagined. Not for you.

The question we keep returning to is: what happens when you pull those two things apart? When the logic is complete and self-contained, and the visuals are derived separately, specifically, for the person looking?

The Interface Doesn't Know What It Looks Like Until It Knows Who's Looking

Here's what we mean by that in practice.

Imagine a product page. The logic is fixed: there is a product, it has a price, it has images, there is an add-to-cart action, there are reviews, there is related product data. All of that is structural. It exists the same way for every visitor.

But the way that information is arranged, what you see first, how much of the price is emphasised versus buried, whether the reviews are above the fold or below it, whether the call to action is bold and urgent or quiet and inviting, none of that is structural. All of it is a visual decision. And right now, those decisions are made once, by a designer or a product team, and served identically to a first-time visitor arriving with low intent, a returning customer ready to buy, a researcher comparing options, and someone who arrived from a recommendation and is already sold.

The same page. For all of them. Because the logic and the visuals are coupled. Because the interface is authored as a single thing.

When you pull them apart, the logic remains fixed. The product exists. The price exists. The reviews exist. But the visual layer, the decisions about arrangement, emphasis, ordering and prominence gets derived from what is actually true about the person arriving. Not guessed in advance. Derived from evidence about this kind of person, at this kind of moment, with this kind of history.

The interface doesn't look the same to everyone. It looks like the right thing for who is looking.

Brand Doesn't Disappear. It Gets More Precise.

The first objection that comes to mind is usually about brand. If the interface is derived differently per person, does that mean every company loses the visual identity they've spent years building?

It's the wrong question. It assumes that brand is the specific arrangement of pixels on a specific page, rather than what brand actually is. We define it as a set of values, constraints, and expressions that define the space within which good design decisions live.

Nike's brand is not a specific homepage layout. It's a philosophy: bold, minimal, action-forward, high-contrast. Those properties define a space. Within that space there are many possible arrangements. Some work better for different people. The brand doesn't tell you which arrangement is best for the high-intent mobile visitor versus the first-time desktop visitor, it tells you what all arrangements must feel like, regardless of who they're for.

When the interface is derived rather than authored, brand becomes the constraint system rather than the output. It defines what can vary and what cannot. The primary colour is locked. The typography is locked. The core identity is inviolable. What's unlocked is arrangement, emphasis, density and ordering, the things that don't change what a brand is but do change how effectively it communicates to each specific person.

The brand becomes more itself for each person, not less itself for everyone.

And there's something else that becomes possible when brand is a constraint system rather than a fixed artifact: it can be expressed in language. Not a token file or a style guide PDF — a sentence. "Luxury minimal with warmth" or "technical and precise but approachable" or "bold and urgent, never quiet." The system derives the specific design decisions from the sentence. The sentence is the brand. Everything that flows from it is interpretation within the space the sentence defines.

This is not a smaller idea than it sounds. It means that the people who understand a brand best, the founders, the CMOs and the creative directors can express it directly, without it getting lost in translation through a design system and a component library and a handoff document and a developer who had six other things to build that week.

Generate Once. Serve Instantly. For Everyone Who Matches.

One of the things that seems hardest about this idea is performance. If the interface is derived specifically for each person, does that mean it has to be generated fresh for every visitor? Wouldn't that be impossibly slow?

Not if you think about it at the right level of abstraction.

People are not infinitely varied. They cluster. There are patterns of behaviour, patterns of intent, patterns of arrival that repeat across millions of visitors. A high-intent buyer arriving from a paid search ad to a product category they've explored before is a type of person. A first-time visitor scrolling slowly on a mobile device at 11pm is a different type. A returning customer who almost bought last week is another.

These clusters are stable enough that you can derive the right interface for each one in advance, cache the result, and serve it instantly when someone who matches arrives. The derivation happens once per cluster. The serving happens in milliseconds. The visitor never waits for anything to be generated, they receive a cache hit that happens to have been prepared for someone exactly like them.

This is the model that changes the economics of personalisation. It's not "generate something new for each person in real time." It's "understand the patterns that exist in your visitors, derive the right interface for each pattern, and serve it instantly to everyone who matches." The intelligence is in the pattern recognition. The serving is just caching.

And the more sites participate in this model, the better the pattern recognition gets. A visitor's pattern doesn't start from zero when they arrive somewhere new. If they've been seen elsewhere. If their behaviour has been observed in other contexts their cluster is already known. They arrive pre-understood. The interface they receive on the first visit is already calibrated to who they are, not to who a designer imagined might show up.

The Interface Follows The Person, Not The Device

The way we've handled multiple devices until now is a version of the same problem. A designer makes decisions for a desktop visitor. Then they make different decisions for a mobile visitor. Then someone notices the tablet experience is broken and those decisions get made too. Each time, someone is sitting down and authoring a version of the interface for a category they're imagining.

When the interface is derived from who is looking rather than authored for a category, device context becomes one more input to the derivation rather than a separate authoring exercise. The same system that knows you're a high-intent returning customer also knows you're on a small touchscreen arriving from a social link late at night. The interface it derives takes all of those facts into account simultaneously.

The experience across your phone, your tablet, your laptop isn't three separate designs that someone tried to keep consistent. It's one set of facts about you, rendering appropriately for whatever surface you're currently using.

This extends further than people usually think. TV interfaces, in-app webviews, kiosk screens — anywhere software runs on a screen, the same question applies: who is looking, and what does the right interface for them look like here? The authoring model has always required someone to go and make separate decisions for each new surface. The derived model carries the understanding of who is looking and applies it wherever the looking happens.

The Honest Tensions

None of this arrives without tradeoffs, and it's worth being clear about them.

The first is stability. People who use a tool every day build familiarity with it. An interface that keeps discovering better arrangements is an interface that keeps changing. For occasional visitors to marketing sites, constant discovery is fine, they have no established habits to disrupt. For daily users of internal tools and productivity software, stability matters as much as optimisation. The system has to know when it has found a good enough answer and hold it, rather than continuing to explore.

The second is legibility. When a page looks different to you than it does to someone else, trust can erode. Did they see something I didn't? Was I shown a less favourable version? The derived interface needs to be explainable. Not technically, but in terms a person can understand and feel good about. You saw this arrangement because of what we know about people in your situation. Here is what we know. Here is why this serves you. Without that transparency, personalisation feels like manipulation rather than service.

The third is creativity. Not every interface should converge on what works. Some of the most important experiences in the world are designed to challenge, disorient and expand, not to give you what you'd already have chosen. A museum doesn't adapt its curation to your preferences. A film doesn't rearrange itself around your attention span. Some interfaces should be stubbornly, deliberately static, because the friction is the point and the designer's intent matters more than the visitor's comfort. The derived interface is powerful for software that serves goals. It's wrong for experiences that pursue them.

Where We Are

The assumption that an interface is something you make, authored once, served to everyone, is not a permanent feature of software. It's a consequence of the economics and the tools available when the decisions were made. When specificity is expensive, you default to generality. When it gets cheaper, you start asking whether generality was ever really what you wanted.

We're building in the direction of something more specific. Not a different look for every single person — that's neither possible nor desirable. But a different understanding of who is looking, expressed through the decisions that actually vary: what gets prominence, what gets ordered first, what gets said to this person at this moment with this history.

The interface as output rather than artifact. Derived from what is true rather than authored from what is imagined.

The technology for this exists. The patterns are recognisable. The caching model makes it fast. What's missing is the layer that pulls it together, the system that observes, understands, derives and serves and the ecosystem of interfaces built to work with it rather than against it.

That's what we're working on.

SentientUI is building the personalisation layer for the web, starting with React and Next.js, pointed at something larger. If this is a problem you're thinking about, we'd like to talk.