Testing a button color and testing a pricing threshold both get called "A/B testing," and architecturally they have almost nothing in common. A UI test swaps a visual element for the duration of one page render and measures a click. A backend logic test runs two different versions of a decision — different eligibility criteria, different pricing rules — against real customers, potentially across multiple requests over days or weeks, where the "variant" is a rule that determines something consequential: what a customer pays, whether they qualify for something, what they're allowed to do. Treating the second kind like the first is how backend experiments produce results nobody can actually trust.
Assignment: it has to be sticky, and it has to be deterministic
A UI test can re-randomize on every page load without much consequence. A backend logic test cannot — if a customer gets eligibility variant A on one request and variant B on the next, purely due to non-deterministic assignment, that's not an experiment anymore, it's noise with a decision attached to it. Assignment needs to be sticky across every touchpoint for the life of the experiment, and it needs to be computed the same way every time without a lookup table that could drift or fail:
import hashlib
def assign_variant(user_id: str, experiment_id: str, split: float = 0.5) -> str:
hash_input = f"{user_id}:{experiment_id}".encode()
hash_value = int(hashlib.sha256(hash_input).hexdigest(), 16)
bucket = (hash_value % 10000) / 10000
return "treatment" if bucket < split else "control"
Deterministic hashing means the same user and the same experiment ID always produce the same bucket, with no state to store or synchronize across services — any service that needs to know a user's variant can compute it independently and get the same answer, which matters a lot once eligibility logic is being called from more than one place in a real system.
The variant is a rule, not a UI element
This is the part that actually differs structurally from UI testing: the rule engine itself needs to be experiment-aware, because there isn't a single "the eligibility rule" anymore — there are two, running simultaneously in production, and every evaluation needs to know which one applies to which user.
def evaluate_eligibility(applicant: Applicant) -> Decision:
variant = assign_variant(applicant.id, experiment_id="foir-threshold-test")
threshold = 0.45 if variant == "treatment" else 0.50 # the actual variant is the rule itself
decision = Decision(
approved=applicant.foir <= threshold,
variant=variant,
experiment_id="foir-threshold-test",
)
return decision
Notice variant and experiment_id are captured directly on the decision record, not inferred later. That's not optional bookkeeping — it's the only way to reconstruct, after the fact, exactly which rule version produced a specific customer's outcome, which matters both for measuring the experiment correctly and for answering the question a customer or a regulator might reasonably ask about why they got the result they got. Reading through what A/B testing backend systems actually requires makes this concrete: the infrastructure needed here is much closer to feature-flagged rule evaluation than to a front-end experimentation SDK, because the thing being varied is decision logic, not markup.
Measurement: the metrics and the timelines are different
UI tests typically measure something that happens in seconds — a click, a scroll, a conversion within the same session. Backend logic tests on pricing or eligibility usually measure something with a longer horizon and a smaller sample: approval rate, revenue per approved customer, downstream churn or default rate, none of which resolve in the same session the decision was made. That has real consequences for how long an experiment needs to run and how large a sample it needs before a difference is statistically meaningful — a lower base rate outcome, like default rate on a loan, needs a meaningfully larger sample to detect the same effect size than a metric with a higher base rate does. The core statistical discipline — proper randomization, pre-registered significance thresholds, avoiding peeking at results before the sample is adequate — is the same discipline covered in any solid guide to how to run an A/B test correctly; what changes for backend logic tests is almost entirely the metric's latency and variance, not the underlying statistical method.
The fairness wrinkle that UI tests don't have
This is worth naming directly, because it's specific to testing pricing and eligibility rather than testing a UI: two customers with genuinely identical relevant facts can get different outcomes purely because of which experiment bucket they landed in. That's a very different situation than two customers seeing a different button color — a pricing or eligibility experiment is, by construction, treating similar customers differently, and the justification for that has to be the experiment itself, documented and time-bounded, not an ad hoc decision nobody can point back to later.
This connects directly to the same defensibility requirement that applies to any dynamic pricing or eligibility system: every decision needs to be traceable back to exactly what produced it — which variant, which rule version, which experiment — the same way a non-experimental pricing decision needs to be traceable back to its rule version. An experiment doesn't remove that requirement. If anything, it raises the bar, because "why did these two similar customers get different outcomes" now has an answer that's true and legitimate — they were in different experiment arms — but only if that answer is actually recorded and retrievable, not reconstructed from memory after the fact.
When one experiment isn't the whole picture
Backend systems rarely run just one experiment in isolation, and that's where interaction effects become a real risk in a way they less often are for isolated UI tests: an eligibility threshold experiment and a separate pricing experiment, both touching the same applicant pool, can interact in ways that make each experiment's individual results misleading if they aren't accounted for. Understanding the actual difference between A/B testing and multivariate testing matters here specifically because backend logic tests are more likely to need a multivariate design than UI tests are — when multiple rule-level experiments are running against overlapping populations, testing them as if they were independent, isolated A/B tests can produce results that don't hold up once the interaction between them is accounted for.
Where this needs to live in the rule layer
The pattern that makes all of the above tractable is treating experiment variants as first-class, versioned rule states rather than ad hoc conditionals scattered through application code. Nected's rule builder supports exactly this: a rule can be configured with multiple active versions simultaneously, with assignment logic determining which version a given evaluation uses, and every decision's audit trail captures not just the outcome but the specific rule version and experiment context that produced it. That audit trail is what turns "why did this customer get this outcome" from a question requiring code archaeology into a direct lookup — which matters for measuring the experiment correctly, and matters just as much if that specific decision is ever questioned individually, experiment or not.
When a simple feature flag is enough
For a low-stakes backend change — an internal-facing threshold, something with no real fairness or compliance weight, a short-lived test with an obvious right answer — a basic feature flag and a spreadsheet to track results is completely adequate, and building experiment-aware rule infrastructure for that is more than the situation calls for. This level of rigor earns its cost specifically when the variant touches something customers experience directly and unevenly — pricing, eligibility, anything where "why did I get a different outcome than someone else" is a question worth being able to answer precisely.
FAQ
Why does variant assignment need to be deterministic instead of just randomized per request?
Because re-randomizing on every request means the same customer could get different treatment on different calls within the same experiment, which isn't a controlled test anymore — it's inconsistent behavior that happens to be tracked. Deterministic hashing guarantees the same user always lands in the same bucket for a given experiment.
How is testing a pricing rule different from testing a UI element in terms of risk?
A UI test's downside is usually a worse click-through rate. A pricing or eligibility test's downside can be a customer being charged incorrectly or denied something they should have qualified for, which is why the audit and traceability requirements are meaningfully higher for backend logic experiments.
Do backend logic experiments need larger sample sizes than UI experiments?
Often yes, particularly when the outcome metric has a low base rate (like default rate or a rare conversion event) — detecting the same effect size in a rarer outcome requires a larger sample than detecting it in a common one like a click.
What's the biggest risk of running multiple backend rule experiments at once?
Interaction effects — two experiments touching overlapping populations or related logic can produce misleading individual results if analyzed as if they were fully independent. This is more common in backend rule testing than in isolated UI tests because rule-level changes tend to affect broader, more overlapping segments of traffic.
Is it ethical to A/B test pricing or eligibility on real customers?
It can be, provided the experiment is well-justified, time-bounded, and every decision is traceable back to the specific variant that produced it — the same standard that applies to non-experimental pricing and eligibility decisions. The experiment itself needs to be a legitimate, documented reason for treating similar customers differently, not an untracked ad hoc variation.
Top comments (0)