A/B Testing Platform Architecture: Ensuring Consistency Across User Sessions
Running experiments at scale means making split-second decisions for millions of users, but there's a fundamental challenge lurking beneath the surface. How do you ensure that when Sarah returns to your app tomorrow, she sees the exact same variant she saw today, even when your infrastructure has scaled to handle thousands of concurrent experiments? This architectural problem sits at the heart of every modern A/B testing platform, and solving it elegantly determines whether your experiments produce reliable data or become a statistical nightmare.
Architecture Overview
An A/B testing platform needs to orchestrate four interconnected systems working in harmony. The Experiment Management Service handles the lifecycle of your experiments, from creation through analysis. It stores experiment definitions, control groups, variant configurations, and statistical thresholds. Meanwhile, the User Assignment Engine is where the magic happens, responsible for deterministically assigning users to variants based on a combination of factors like user ID, experiment ID, and a consistent hashing algorithm.
The Metric Tracking System captures every interaction users generate, whether that's clicks, conversions, page views, or custom events. These raw events flow into a time-series database or data warehouse where they can be aggregated and analyzed. Finally, the Statistical Analysis Service processes accumulated metrics to calculate conversion rates, confidence intervals, and p-values, determining when results reach statistical significance.
What ties these components together is a well-designed assignment strategy. Rather than randomly picking variants each time a user visits, the system uses a deterministic approach. By hashing the user's ID combined with the experiment's ID, you always get the same output. This hash determines which variant bucket the user falls into (variant A gets hashes 0-49, variant B gets hashes 50-99, for example). This approach is computationally cheap and eliminates the need to store every single assignment in a database.
The Assignment Flow
When a user loads your application, the client or server calls the Assignment Engine with the user ID and experiment ID. The engine performs the hash calculation, checks if the user is even eligible for this experiment (respecting targeting rules and exclusion criteria), and returns the assigned variant. This happens in milliseconds, making it suitable for real-time request handling.
Design Insight: Session Consistency Through Deterministic Hashing
The answer to ensuring users see the same variant across sessions lies in removing randomness from the equation. Instead of storing assignment decisions in a separate lookup table, deterministic hashing makes the assignment a pure function of user ID and experiment ID. Every time Sarah's browser calls the assignment endpoint, the same user ID plus the same experiment ID produces the same hash bucket, guaranteeing the same variant assignment.
This approach solves several problems simultaneously. It scales without requiring a massive assignment database, it handles new users without any lookup latency, and it naturally handles the "same variant across devices" problem if you're assigning based on a universal user ID rather than device ID. The trade-off is that you can't change user assignments retroactively without risking data corruption, so your experiment design must be finalized before running traffic through it.
Watch the Full Design Process
Want to see how this architecture comes together in real-time? We've documented the complete design process across multiple platforms:
Try It Yourself
Building a system design from scratch can feel overwhelming, but InfraSketch makes it intuitive. Head over and describe your A/B testing platform (or any system) in plain English. In seconds, you'll have a professional architecture diagram complete with a design document. Whether you're preparing for an interview, designing your company's next platform, or just exploring system architecture, this tool turns description into diagrams instantly.
This is Day 89 of our 365-day system design challenge. Start designing today.
Top comments (0)