System Design - Building an Experiment Service

Designing an experiment service: API design, data modeling, and how to scale it

Before any architecture, a shared vocabulary. An experiment is the thing under test — "does the new signup page convert better than the old one." Each version shown to visitors is a variant; the version you'd show if there were no experiment running at all is usually the control. Traffic doesn't have to split evenly across variants — each variant carries a weight, its configured share of traffic. When a visitor sees a variant, that's an exposure. When they complete the goal action — signing up, buying something — that's a conversion. Conversion rate — conversions divided by exposures, per variant — is the number that eventually tells you which variant won.

That vocabulary is simple. Building the service that makes it happen correctly, at real traffic volume, without slowing down the page it's embedded in, is where the actual design work lives.

Requirements

Good system design starts with requirements, not architecture. Here's what this service has to do — and, just as importantly, what it has to guarantee.

Functional requirements

• A client (a business — this is a multi-tenant service) can sign up and get credentials.
• A client can define an experiment: its variants and their weights.
• Given a visitor and an experiment, the system returns which variant they should see.
• Exposures and conversions are recorded as they happen.
• A client can view exposures, conversions, and conversion rate per variant.

Non-functional requirements

• Low latency on the read path. The variant-assignment call sits directly between a visitor and their page rendering. It has to respond in single-digit-to-low-double-digit milliseconds — a real browser is blocked on it.
• Consistency. The same visitor, on the same experiment, must always get the same variant, for as long as the experiment runs.
• Data integrity. Every exposure and conversion must be counted exactly once, even under retries and duplicate requests.
• Multi-tenancy. One client's experiments must never be visible to, or affected by, another client's.
• Scalability. Traffic should be able to grow by orders of magnitude without a redesign — by adding capacity, not by rewriting.
• Graceful degradation. If any dependency misbehaves, the system degrades to a safe default rather than failing visibly for a real visitor.

Every design decision below traces back to one of these — mostly the first three.

API design

POST /clients — sign up

No auth here — you can't require a key to get your first key.

curl -X POST https://api.yourexperimentservice.com/clients \
  -H "Content-Type: application/json" \
  -d '{"name": "Acme Labs"}'

Response:

{ "id": "cl_9f2a", "name": "Acme Labs", "api_key": "sk_live_..." }

POST /experiments — configure an experiment

Requires the API key from signup.

curl -X POST https://api.yourexperimentservice.com/experiments \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk_live_..." \
  -d '{
    "key": "signup_test",
    "status": "live",
    "variants": [
      { "key": "control", "weight": 70, "static_copy": "Welcome!" },
      { "key": "variant_b", "weight": 30, "ai_prompt": "Write a short signup headline" }
    ]
  }'

Response: the persisted experiment, with every variant's static_copy filled in — generated ones included.

GET /assignment — the hot path

curl "https://api.yourexperimentservice.com/assignment?user_id=visitor_123&experiment_key=signup_test" \
  -H "X-API-Key: sk_live_..."

Response:

{ "experiment_key": "signup_test", "assigned_variant": "control", "static_copy": "Welcome!" }

Same user_id + experiment_key always returns the identical result, indefinitely.

POST /events — record an exposure or conversion

curl -X POST https://api.yourexperimentservice.com/events \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk_live_..." \
  -d '{
    "visitor_id": "visitor_123",
    "experiment_key": "signup_test",
    "variant_key": "control",
    "event_type": "exposure"
  }'

Response:

{ "status": "success", "event_id": "ev_88b1" }

A repeated identical exposure returns this same shape with the same event_id — not an error.

GET /results — reporting

curl "https://api.yourexperimentservice.com/results?experiment_key=signup_test" \
  -H "X-API-Key: sk_live_..."

Response:

{
  "experiment_key": "signup_test",
  "results": [
    { "variant_key": "control", "exposures": 812, "conversions": 94, "conversion_rate": 0.116 }
  ]
}

The same data also powers a tenant-scoped dashboard view, so a client can check results without writing a script.

High-level design

Four independent write paths — signup, configuration, assignment, and events — all persist to one database. Reads for reporting go through the same store, for now. Of these four, only the assignment path carries a strict latency budget: it's invoked once per page view, synchronously, with a real visitor's browser waiting on the response. That single asymmetry — one latency-critical path, three that can afford to be slower — drives most of the design decisions below.

Design deep dive: keep expensive work off the hot path

The experiment API has one interesting requirement: a variant's copy can be written by hand, or generated by an AI model. The naive design calls the model at assignment time — whenever a visitor is due to see an AI-authored variant, generate fresh copy right then.

That breaks the latency requirement immediately. A third-party model call can take seconds, time out, or simply be unavailable — none of which the assignment path can tolerate.

The general principle here shows up across most systems with a strict-latency read path: push expensive computation to write time, and serve reads from something already computed. If a value doesn't need to be fresh at read time — and nobody actually needs bespoke copy generated in the few milliseconds between a page loading and a response — compute it once when it's cheap to be slow (at configuration time), persist the result, and let every subsequent read be a plain lookup. The assignment path never talks to the model at all. If generation fails outright, the system falls back to plain, default copy rather than blocking the one call that isn't allowed to fail.

This is the same idea behind materialized views, precomputed caches, and read-model patterns generally: decouple when a value is computed from when it's read.

Design deep dive: the assignment API — sticky and deterministic

This is the most interesting endpoint in the system, and it's worth separating the requirement from the mechanism, because the two get conflated easily.

The requirement is stickiness. A visitor must see the same variant on every visit, for the life of the experiment. If you've worked with load balancers, this should sound familiar — it's the same property that session affinity ("sticky sessions") provides: routing a client's subsequent requests to the same backend server it hit the first time.

The traditional way to get stickiness is stateful: the first time you see a visitor, make a decision and remember it — in a session store, a cookie plus a server-side lookup, or a load balancer's affinity table. Every later request has to consult that stored state.

The mechanism is determinism instead of memory. There's a cheaper way to get the same property, if one condition holds: if the decision is a pure function of information already available on every request, nothing needs to be stored — the identical answer can be recomputed every time. That's the route taken here.

Concretely: hash the visitor ID and the experiment key together, producing a number. Reduce it to a fixed range — say 0 to 99 — and partition that range according to the configured weights: the first 70 numbers to variant A on a 70/30 split, the rest to variant B. The same two inputs always hash to the same number, so the same visitor lands in the same partition, for the same experiment, forever — nothing written to disk, no state anywhere.

Two properties of the hash function are doing the actual work:

• Determinism — the same input always produces the same output. This is what gives stickiness without storage. Most hash functions are deterministic by construction; this isn't something you engineer, but it is the property being relied on here.
• Uniform distribution — across many different inputs, outputs spread evenly across the output range, without clustering. This is what makes a configured 70/30 split actually land close to 70/30 in practice, instead of skewing toward one variant because of some quirk in how visitor IDs happen to be generated.

Why hash the visitor ID and the experiment key together, rather than the visitor ID alone. This detail matters more than it looks. Hash the visitor ID alone, and a given visitor lands in the same percentile bucket for every experiment ever run — someone in the "top 30%" bucket for one experiment is in the top 30% for all of them. That's an unwanted correlation: it means experiments aren't actually independent of each other, and a visitor who happens to convert unusually well (or badly) skews every test in the same direction. Hashing the experiment key in as well gives each experiment its own, effectively independent bucketing for the same visitor — which experiment and which visitor stop leaking into each other.

A clarifying note, since the terminology overlaps. This is not the same thing as consistent hashing, a related but distinct technique for distributing data across a changing set of nodes — think shards in a distributed cache or database — while minimizing how much gets remapped when a node is added or removed. What's happening here is simpler: one hash function mapping (visitor, experiment) to a fixed bucket range that never changes shape. Worth knowing the distinction, since "hashing for consistent assignment" and "consistent hashing" sound alike but solve different problems.

One more subtlety worth internalizing. This design produces two different guarantees, and verifying one doesn't verify the other. For any single visitor, the assignment is completely deterministic — no randomness in an individual decision at all. Across a population of visitors, the outputs merely look randomly distributed, and it's that population-level randomness that makes a configured split hold up in aggregate. That second property is statistical, not deterministic, and it has to be checked empirically, with a genuinely large sample — a handful of test visitors can look meaningfully skewed purely by chance in a perfectly correct system, and can just as easily look fine when something's actually broken.

Design deep dive: idempotency and duplicate prevention

Recording an exposure or conversion looks trivial: a visitor saw a variant, write a row. It stops being trivial the moment real traffic replaces clean test data — double-clicks, retried page loads, back-and-forward navigation all produce the same logical event more than once. If every one of those writes a row, every conversion-rate number downstream is silently inflated.

The naive fix, and why it isn't enough. Check whether this exact event has already been recorded before writing a new one. The problem: "check" and "then write" are two separate operations with a gap between them. Under concurrent requests, two duplicate calls can both pass the check before either has written anything. This is a classic check-then-act race condition — sometimes called TOCTOU, time-of-check to time-of-use — and it shows up anywhere a read-then-write isn't atomic.

The actual fix: idempotency, enforced structurally. An operation is idempotent if performing it once and performing it five times leave the system in the same state. The reliable way to get there isn't detecting duplicates in application code — it's making a duplicate physically impossible to write, by pushing a uniqueness constraint down to the storage layer itself: here, a unique constraint on (visitor, experiment, variant, event type). A duplicate write is no longer something application code has to notice — the write itself is rejected by the database, no matter how many concurrent requests arrive. The calling code treats "rejected as duplicate" as a success, not an error, since the thing it wanted recorded already exists.

This generalizes well beyond one system — it's the same principle behind idempotency keys in payment APIs, exactly-once processing in message queues, and safe retries in any at-least-once delivery system. Wherever possible, move "detect and reject a duplicate" down into "make the duplicate structurally impossible," as close to the data as the storage layer allows.

Data model (low-level design)

Four entities, and the relationships between them are doing real work, not just organizing storage:

• Client: the tenant. Everything else exists only in the context of one of these.
• Experiment: belongs to exactly one client. Its name only has to be unique within that client, not globally — which is exactly what makes "scope every lookup to the tenant" a hard requirement rather than a nice-to-have. Two clients can legitimately name an experiment the same thing; any lookup that isn't scoped by client risks serving one tenant's config to another tenant's request.
• Variant: belongs to an experiment, carries a weight and either literal or AI-generated content.
• TrackingEvent: one row per exposure or conversion, referencing both the experiment and the specific variant it happened under, carrying the idempotency-enforcing uniqueness constraint from the section above.

Worth naming explicitly: this ledger is append-only, not a set of counters incremented in place. Exposures could be tracked as a single mutable number per variant, but in-place counters have their own race conditions under concurrent writes, and they throw away the ability to answer questions nobody thought to ask up front — when exactly did this happen, in what order did events actually arrive. A new row per event, nothing ever mutated, trades some storage growth for meaningfully better correctness and flexibility.

Scaling: where the real system design happens

Everything above works comfortably at low-to-moderate traffic. The interesting engineering is in what breaks first as volume grows, and matching each failure to the specific technique that fixes it — not reaching for every scaling tool that exists because it sounds thorough. Correctness and clean API design get you a working system; how it behaves at 100x the traffic is what makes it a system.

Caching

The first thing to strain is reads on the assignment path. The query itself stays fast and indexed, but at high enough concurrency, requests start queuing for a free database connection, because configuration data is read constantly and almost never changes. Read far more than written is exactly the case a cache is built for. A cache-aside layer in front of the database, invalidated whenever the underlying configuration actually changes, takes the database out of the hot path almost entirely except on a miss.

Background jobs, for the AI call

Once experiment configuration itself becomes frequent rather than occasional, the AI-generation step is worth revisiting: move it fully into a background job queue, so a slow model call can't tie up a web server thread at all, rather than merely being "synchronous but tolerable."

Message queues, for the write path

Further out, the write side starts to strain: every exposure and conversion is a synchronous insert, and the uniqueness constraint that provides idempotency becomes, at high enough volume, itself a point of contention among many concurrent writers. The fix is a queue in front of the write — accept the event, acknowledge immediately, let a pool of workers drain the queue and perform the actual writes, often batched together, which is far cheaper per row at real volume than one insert at a time.

Read replicas

This is also the point where reporting queries start meaningfully competing with write traffic on the same database — the natural moment to introduce a read replica dedicated to reporting and dashboard queries, leaving the primary free to focus on writes.

Partitioning

Further still, the event ledger itself becomes the bottleneck — large enough that its indexes stop comfortably fitting in memory, and even indexed queries degrade. The fix here is partitioning, most naturally by time, since "how long do raw events actually need to be kept" is a real, answerable retention question in a way that visitor or experiment distribution isn't. At true extremes, raw events stop being the right thing to query for reporting at all, and a separate streaming layer maintaining pre-aggregated counts becomes the actual read path instead.

Load balancing

The app tier scales horizontally with essentially no extra design work, provided it stays stateless — no server-side session storage anywhere, so any instance can serve any request and a load balancer can distribute purely by load, with no session-affinity rules to get wrong (a nice callback: the assignment API achieved that same statelessness one layer up, via hashing, for exactly this reason). The one thing that genuinely has to be right: the load balancer's health check needs to hit a dedicated, request-independent liveness endpoint, never a page whose response can legitimately vary by who's asking — otherwise the load balancer can misread normal behavior as an unhealthy instance and start cycling servers that are actually fine.

Rate limiting, for multi-tenant fairness

Nothing so far stops one tenant's traffic spike from degrading latency for every other tenant sharing the same infrastructure. Per-tenant rate limiting, applied before it's ever actually needed, is far cheaper than retrofitting it after a real incident where one customer's bad day became everyone's bad day.

Observability

None of the fixes above should be applied before the specific metric that motivates it is actually visible — cache hit rate, queue depth, replica lag, per-tenant volume. Introducing infrastructure speculatively, because it sounds like the mature thing to do, is its own kind of mistake — solving a problem that hasn't been confirmed to exist yet.

Closing thought

The organizing principle underneath almost everything above is worth stating plainly: on a system that sits directly on a page's critical path, the safe default when anything goes wrong — an experiment nobody's heard of, a failed external call, an unreachable cache — is to fail toward the plainest, safest experience, never to error, hang, or block. A real visitor should never be able to tell that an experimentation system is having a bad day. Getting the happy path fast is the easy half of this problem. Making every unhappy path invisible is the harder, more important half.

Coffee?

If this article saved you a few hours of architectural overthinking (or inspired a personal project of your own), consider buying me a coffee ☕
https://superprofile.bio/vp/buy-me-a-coffee-263

It helps me continue writing about software architecture, system design and, every now and then, the occasional poem.