A/B Testing Platform Architecture: Consistency Across Sessions
Running experiments at scale requires more than just dividing users into groups. You need to ensure every user sees the same experience every time they visit, track metrics reliably across millions of events, and determine statistical significance without months of waiting. A well-designed A/B testing platform is the backbone of data-driven product decisions, and getting the architecture right separates smooth experimentation from inconsistent chaos.
Architecture Overview
An A/B testing platform brings together four essential layers working in concert. The assignment layer determines which variant each user sees based on their ID and experiment configuration, using consistent hashing to guarantee the same assignment every single time. The data collection layer captures user events, metrics, and interactions in real-time, flowing into a high-throughput event stream. The analytics engine processes these events, calculates conversion rates, and computes statistical significance using established methodologies like chi-square tests. Finally, the experiment management layer provides the interface for creating experiments, configuring rules, monitoring performance, and eventually declaring winners.
The key insight here is separation of concerns. Assignment logic lives independently from metric tracking because they have different latency requirements, consistency guarantees, and scaling challenges. The assignment service must respond in milliseconds and be always consistent, while the analytics pipeline can tolerate slight delays in exchange for deep aggregations and statistical rigor. Between them sits an event streaming system like Kafka that decouples real-time assignments from batch analysis, allowing you to replay events and recalculate metrics without affecting active experiments.
Deployment considerations matter too. The assignment service should be deployed close to your application (often as a sidecar or library), minimizing latency and reducing dependencies. Your analytics infrastructure can live further downstream, processing data in batches or micro-batches depending on how quickly you need insights. Configuration for active experiments must be cached aggressively at the edge, with fallback mechanisms when the assignment service is unreachable.
Design Insight: Sticky User Assignment
The most common question we see: how do you ensure users always see the same variant across sessions? The answer is deterministic assignment using consistent hashing. When a user enters an experiment, you hash their user ID together with the experiment ID using an algorithm like MurmurHash or xxHash. This produces a pseudo-random but deterministic value between 0 and 1. You then map that value to a variant based on allocation percentages, for example: if you're testing variant A at 50% and variant B at 50%, any user whose hash falls between 0 and 0.5 always sees A, and those between 0.5 and 1 always see B.
This approach requires zero additional storage or lookup calls. The assignment is instant, consistent across all your services, and survives service failures. Even better, if a user's ID is the only input, the same user gets the same variant whether they're on mobile, desktop, or using a different browser, as long as you can identify them consistently. Some platforms add additional entropy like geographic region or user cohort to the hash input, creating "stratified randomization" that balances variants across important dimensions.
Watch the Full Design Process
Want to see how this architecture comes together in real-time? We built a complete A/B testing platform design live, exploring assignment mechanisms, metric tracking, and consistency strategies. Watch the full system design process on your preferred platform:
Try It Yourself
This is Day 89 of our 365-day system design challenge, and we're constantly exploring new architectures. Want to design your own A/B testing platform or tackle a different system design problem? Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document. No whiteboarding, no guesswork, just clear architecture aligned with your requirements.
Top comments (0)