DEV Community: Matt Frank

Day 92: Stream Processing Engine - AI System Design in Seconds

Matt Frank — Wed, 08 Jul 2026 20:00:14 +0000

Real-time data processing is everywhere, from stock market analytics to recommendation engines, yet most systems struggle with a fundamental challenge: how do you transform millions of events per second while maintaining correctness and handling the messy reality of distributed systems? Stream processing engines like Kafka Streams solve this by providing a unified framework for event transformation, aggregation, and windowing at scale. Understanding how these systems work—especially the tricky edge cases—gives you insight into building reliable, high-throughput data pipelines.

Architecture Overview

A stream processing engine sits between your data sources and sinks, orchestrating the flow of events through a series of processing stages. The core components include a source connector that ingests events from topics or queues, a topology that defines transformations and aggregations, state stores that maintain intermediate results, and a sink connector that publishes output. Events flow through the topology as streams, which are unbounded sequences of data, and tables, which are stateful views of those streams. The engine processes these records in order while distributing work across multiple instances for horizontal scalability.

Windowing is where things get interesting. The engine groups events into finite windows (tumbling, sliding, session-based, or custom) so you can aggregate data meaningfully. A tumbling window for "events per minute" or a session window for "user activity sessions" transforms an infinite stream into manageable chunks. Behind the scenes, the engine maintains separate state stores for each window, allowing parallel aggregations across multiple windows simultaneously. When a window closes, its final result is emitted downstream, and its state can be cleaned up to reclaim memory.

State management is critical to the architecture. The engine persists state to fault-tolerant storage (usually a changelog topic) so that if a processor crashes, it can recover its state from that log. This dual approach of in-memory state with persistent backups ensures both performance and reliability. The topology also tracks watermarks, which represent the engine's understanding of how far through time it has processed, helping it know when a window is truly complete.

Design Insight: Handling Late-Arriving Events

Here's where the architecture gets sophisticated. Late-arriving events are records that arrive after their window has already closed and emitted results. A naive approach would simply drop them, but that loses data. Instead, stream processing engines use a grace period and allowed lateness configuration. When you close a window, you keep it open for a configurable grace period (say, 10 seconds) to absorb late events. If an event arrives within this grace period, the window's state is reactivated, the aggregation is recalculated, and a corrected result is emitted downstream.

Beyond the grace period, different engines handle outliers differently. Some systems support side outputs or dead letter queues where extremely late events are routed for separate handling or alerting. The key design decision is balancing correctness (capturing all relevant data) against resource constraints (you can't keep windows open forever). This is why allowed lateness is a tunable parameter: domain knowledge about your data helps you set appropriate thresholds. Understanding this tradeoff is essential when designing systems where late data matters, like billing pipelines or compliance-critical aggregations.

Watch the Full Design Process

Want to see how these concepts come together visually? Check out the real-time architecture design in action:

Try It Yourself

Ready to design your own stream processing architecture? Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document. Whether you're building event pipelines, real-time analytics, or fault-tolerant aggregations, InfraSketch helps you visualize and iterate on your design before implementation.

This post is part of Day 92 of the 365-Day System Design Challenge. Each day explores a new architecture pattern or infrastructure concept. Stay tuned for more deep dives into distributed systems, databases, caching, and everything in between.

Day 93: Batch Processing Pipeline - AI System Design in Seconds

Matt Frank — Wed, 08 Jul 2026 13:04:17 +0000

Processing terabytes of data nightly sounds straightforward until something breaks halfway through. When a job fails processing 2TB of data after 12 hours of work, your team faces a critical question: restart from the beginning and miss deadlines, or recover from the failure point? This is where a well-designed batch processing pipeline becomes essential infrastructure, not just a nice-to-have.

Architecture Overview

A robust batch processing pipeline resembles an assembly line with built-in checkpoints. The system typically consists of four main layers: data ingestion, distributed processing, state management, and output delivery. Data flows from source systems into a staging area, then gets partitioned and distributed across a cluster of worker nodes. Each node processes its assigned data slice independently, which is crucial for both performance and reliability.

The distributed processing layer is where the magic happens. Think of it like dividing a massive book into chapters that multiple people read simultaneously. A coordinator node orchestrates the work, assigning partitions to workers and tracking progress. Workers execute the same processing logic on different data segments, whether that's transformations, aggregations, or filtering. The beauty of this design is that if one worker fails, only its partition needs reprocessing, not the entire dataset.

State management ties everything together. The pipeline maintains checkpoints at regular intervals, essentially snapshots of "we've successfully processed data up to this point." This is where tools like HDFS or cloud object storage become invaluable. Metadata gets written to a reliable database, recording which partitions completed successfully, which are in-progress, and which haven't started yet. Without this state tracking, you'd have no choice but to restart from scratch after any failure.

Key Design Decisions

Why distribute the work at all? Speed and resilience. A single machine processing 2TB sequentially could take days. Distributing across 100 nodes dramatically cuts the runtime. More importantly, distribution isolates failures. One node crashing affects 1% of your work, not 100%.

Why maintain detailed checkpoints? Because recovery depends on knowing exactly where you stopped. Without granular state tracking, you lose visibility into what actually completed.

Design Insight: Recovering from Failure Midway

When a job fails halfway through a 2TB dataset, intelligent pipeline design prevents catastrophic restarts. The system knows exactly which partitions completed successfully because that state was persisted to a durable store after each partition finished processing. Instead of reprocessing all 2TB, the pipeline reschedules only the failed partitions and those that were in-progress when the crash occurred.

Some pipelines implement idempotency, ensuring that reprocessing a partition that already completed produces the same result without corrupting data. Combined with deduplication logic in the output layer, this approach guarantees correctness even if a partition gets processed twice. The recovery mechanism typically includes exponential backoff for retrying failed partitions, preventing cascading failures when partial outages affect multiple nodes.

The real sophistication comes from knowing what caused the failure. Was it a worker node crash, a disk space issue, or a data quality problem? Modern pipelines emit detailed logs and metrics that help operators distinguish between transient failures (retry the partition) and permanent ones (alert the team, investigate the data). This intelligence prevents infinite retry loops while ensuring temporary blips don't cause operational headaches.

Watch the Full Design Process

Curious how these components come together visually? Watch the architecture evolve in real-time as we design this system from scratch:

Try It Yourself

This is Day 93 of our 365-day system design challenge, and the pattern is clear: every architecture decision serves a purpose. The best way to internalize these concepts is to design your own system from scratch.

Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document.

Day 91: Business Intelligence Dashboard - AI System Design in Seconds

Matt Frank — Tue, 07 Jul 2026 20:00:14 +0000

Building a business intelligence dashboard that serves thousands of users real-time metrics without melting your database is a classic scaling challenge. When your data volume grows from gigabytes to terabytes, traditional query-on-demand approaches become impossibly slow. This is where thoughtful pre-aggregation strategies transform a sluggish system into one that delivers insights instantly.

Architecture Overview

A production BI dashboard typically sits at the intersection of multiple data worlds. On one side, you have source systems (transactional databases, APIs, event streams) constantly generating raw data. On the other side, you have end users expecting sub-second dashboard loads with drill-down capabilities. The architecture that bridges these worlds consists of several key layers.

The ingestion layer captures data from diverse sources and normalizes it into a common format. This might include ETL pipelines pulling from relational databases, event streaming systems consuming real-time application logs, and batch jobs syncing from third-party APIs. This layer is intentionally decoupled from the query layer so you can scale collection and transformation independently.

The processing layer is where the magic happens. Raw events flow into a stream processor or batch scheduler that performs initial transformations, deduplication, and validation. From there, data flows into two critical destinations: a data warehouse (your single source of truth) and a metrics store (your aggregation powerhouse). The dashboard itself sits in the presentation layer, typically powered by a fast query engine that speaks to both the warehouse and metrics store depending on the query type.

Key Design Decisions

You'll make several important choices here. First, is your dashboard real-time or near-real-time? True real-time adds complexity but may not be necessary if hourly or 5-minute freshness meets your SLA. Second, how much history do you need? Keeping all raw data indefinitely is expensive. Third, which dimensions matter most for slicing and dicing data? These shape your pre-aggregation strategy.

Design Insight: Pre-Aggregation as Your Scaling Secret

Pre-aggregation is the difference between a dashboard that scales and one that buckles under load. Instead of computing "total revenue by region by product by hour" on every query, you compute these combinations once during a scheduled aggregation window and store the results in a specialized metrics store. When a dashboard user requests that exact slice, you're retrieving pre-computed numbers from fast-access storage rather than scanning terabytes of raw data.

The strategy involves identifying your most common query patterns (through logs and user instrumentation) and building materialized views or time-series rollups that match those patterns. You might pre-aggregate at multiple granularities: hourly, daily, and weekly. Users asking for daily trends hit the daily rollup. Those drilling into the last hour hit the hourly data. This hierarchical approach balances storage costs against query latency. Tools like data cubes, columnar stores, and purpose-built metrics databases excel at this. The key is accepting that you can't pre-aggregate every possible combination and instead focusing on the 80% of queries that matter.

Watch the Full Design Process

Want to see how this architecture comes together in real-time? I created a video walkthrough of designing a BI dashboard system, exploring these decisions live:

The video uses InfraSketch to generate the architecture diagram as I describe the system, so you'll see each component appear as we discuss how data flows and where the critical bottlenecks live.

Try It Yourself

Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document.

Day 92: Stream Processing Engine - AI System Design in Seconds

Matt Frank — Tue, 07 Jul 2026 13:34:05 +0000

Real-time data processing has become the heartbeat of modern applications, from fraud detection to recommendation engines. Yet building a system that ingests, transforms, and aggregates millions of events per second while maintaining correctness is notoriously complex. This is where stream processing engines like Kafka Streams shine, solving the challenge of turning unbounded data streams into actionable insights.

Architecture Overview

A stream processing engine is fundamentally different from batch systems because it must handle continuous, infinite data streams while producing results with minimal latency. The architecture typically consists of several interconnected layers: a source layer that consumes from message brokers, a stateful processing layer that performs transformations and aggregations, a windowing layer that groups events into time-based buckets, and a sink layer that writes results to downstream systems.

The source layer acts as the intake valve, continuously pulling events from Kafka topics, message queues, or other event streams. These events flow into the processing topology, a directed acyclic graph where nodes represent operations like filtering, mapping, or joining, and edges represent data flow between them. The beauty of this design is its composability, allowing developers to build complex pipelines from simple, reusable components.

The stateful processing layer is where the real magic happens. Unlike stateless transformations, stateful operations maintain internal state across events, enabling aggregations like counting, summing, or tracking distinct users. This state must be durable and recoverable, so stream engines typically back it with local stores or distributed state management systems. The windowing layer sits alongside this, partitioning the infinite stream into finite windows based on time, sessions, or event counts, which is essential for performing meaningful aggregations.

Handling Late-Arriving Events: A Critical Design Decision

One of the trickiest aspects of stream processing is dealing with out-of-order and late-arriving data. Events don't always arrive in the order they occurred, whether due to network delays, distributed system complexities, or source buffering. When an event arrives after its window has already closed and results have been emitted, the engine faces a choice: discard it, or emit a correction.

Most production stream engines use a combination of strategies to handle this gracefully. First, they maintain a grace period, an additional buffer window that stays open slightly longer than the main window, giving stragglers a chance to arrive. Events within this grace period trigger updates to already-emitted results, which downstream systems must be prepared to handle through idempotent writes or event versioning. Beyond the grace period, engines can route extremely late events to a separate sidecar topic for manual investigation or reprocessing.

This design decision reflects a fundamental trade-off in stream processing: latency versus completeness. Shorter grace periods mean faster results but risk missing late data. Longer grace periods increase accuracy but delay final answers. The engine must expose this as a configurable parameter, allowing different use cases to find their optimal balance.

Watch the Full Design Process

Want to see how these architectural decisions come together? Watch the real-time design session where we built this stream processing engine architecture, exploring the nuances of windowing, state management, and late event handling:

Try It Yourself

Ready to design your own stream processing system? Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document.

Whether you're building real-time analytics, event-driven workflows, or complex data pipelines, having a clear mental model of your stream processing architecture is the first step toward production-ready systems. Start sketching today, and see how quickly you can move from concept to deployable design.

Day 90: Data Lake Architecture - AI System Design in Seconds

Matt Frank — Mon, 06 Jul 2026 20:00:14 +0000

Data lakes promise to democratize data across your organization, but without the right architecture, they quickly become data graveyards where information goes to die. As companies collect more structured, semi-structured, and unstructured data than ever before, the line between a valuable analytics platform and a chaotic data swamp becomes razor-thin. This is Day 90 of our 365-day system design challenge, and today we're diving into how to architect a data lake that actually delivers insights instead of headaches.

Architecture Overview

A modern data lake architecture sits at the intersection of flexibility and governance. The system ingests data from multiple sources (APIs, databases, IoT devices, files) through a scalable ingestion layer that normalizes incoming data without forcing it into rigid schemas. This raw data lands in a staging zone, typically organized by source, where it's retained in its original format. From there, the architecture branches into processing layers: a bronze zone holds untransformed raw data, a silver zone contains cleaned and deduplicated data, and a gold zone features business-ready datasets optimized for analytics and machine learning workloads.

The real magic happens in the governance and cataloging layer that sits alongside these zones. Metadata management tools track data lineage, ownership, and usage patterns across the entire lake. A data catalog acts as a searchable index, allowing analysts and engineers to discover datasets without becoming archaeologists. Access controls and quality metrics are embedded throughout, not bolted on afterward. This layered approach, sometimes called the medallion architecture, ensures that each zone serves a specific purpose while maintaining a clear audit trail.

Security and scalability considerations shape every component. Data at rest uses encryption and is organized by sensitivity level, with separate storage tiers for different classification levels. Compute resources scale independently from storage, preventing expensive compute clusters from idling while waiting for large file transfers. Monitoring pipelines track data freshness, quality metrics, and pipeline failures in real-time, alerting teams before downstream dashboards show stale or incorrect information.

Design Insight: Preventing the Data Swamp

The difference between a data lake and a data swamp comes down to one word: governance. Without it, data accumulates faster than it's consumed, schemas drift without documentation, and nobody knows what datasets are reliable. The antidote involves implementing strict cataloging standards from day one, requiring data owners to document their datasets before they're considered discoverable, and enforcing automated quality checks that flag data anomalies before they propagate downstream.

Equally important is establishing clear ownership models and retention policies. Every dataset should have an assigned owner responsible for its accuracy and freshness, and data that's unused for a defined period should be archived or deleted rather than left orphaned. Tools like metadata management platforms and data observability solutions transform governance from a bureaucratic checkbox into a living, breathing system that teams actually use. When discovery, quality, and lineage are frictionless, your data lake stays clean.

Watch the Full Design Process

Want to see how this architecture comes together in real-time? Check out the full system design process where we built this data lake architecture using AI-powered diagram generation:

Try It Yourself

Ready to design your own analytics platform? Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document.

Day 91: Business Intelligence Dashboard - AI System Design in Seconds

Matt Frank — Mon, 06 Jul 2026 13:03:17 +0000

Business Intelligence Dashboard: Real-Time Analytics at Scale

Real-time dashboards sound simple until you're monitoring millions of events per day across dozens of data sources. The challenge isn't collecting data, it's making sense of it fast enough for decision-makers to act. This is where thoughtful architecture becomes critical.

Architecture Overview

A modern BI dashboard typically sits at the intersection of three architectural layers: data ingestion, aggregation, and presentation. On the ingestion side, you're pulling metrics from APIs, databases, event streams, and logs. These raw events flow into a central processing layer where they get normalized, validated, and enriched. Finally, the query layer serves pre-computed results to your frontend dashboards, where users see charts and KPIs update in near real-time.

The key insight is that you shouldn't compute aggregations on demand. Instead, your architecture should be built around the principle of write-time computation. Raw events land in a distributed message queue like Kafka or Pub/Sub, where streaming processors consume them immediately. These processors calculate hourly, daily, and weekly summaries and store them in an analytics database or data warehouse. When a user opens their dashboard, they're querying these pre-computed tables, not scanning billions of raw events.

The data flow matters too. Your system needs a time-series database for high-cardinality metrics, a columnar warehouse for dimensional analysis, and a cache layer to handle dashboard refresh spikes. Many teams use a medallion architecture, starting with raw bronze data, moving to refined silver tables with business logic applied, and finally serving gold tables optimized for specific dashboards. This separation keeps concerns clean and lets different teams own different layers.

Design Insight: Pre-Aggregation at Scale

Here's the uncomfortable truth: as your data volume grows, query performance degrades exponentially unless you architect for it from day one. Pre-aggregation is your answer. Instead of calculating monthly revenue totals when someone asks for them, you compute and store those totals continuously. This means maintaining multiple levels of aggregation, typically at 5-minute, hourly, daily, and monthly intervals depending on your business needs.

The trade-off is storage and eventual consistency. Yes, you'll use more disk space for all those pre-computed tables. Yes, there's a few seconds of lag before new events appear in aggregates. But the user experience is transformative. A dashboard that used to take 30 seconds to load now loads in under a second. More importantly, your infrastructure can handle 10x more concurrent users without scaling your query resources. Tools like InfraSketch make it easier to visualize how these aggregation layers interact and where bottlenecks might emerge as your system grows.

Watch the Full Design Process

Curious how this architecture came together? Check out the real-time design session where we built this system from scratch:

Try It Yourself

Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document.

Day 89: A/B Testing Platform - AI System Design in Seconds

Matt Frank — Sun, 05 Jul 2026 20:00:15 +0000

A/B Testing Platform Architecture: Consistency Across Sessions

Running experiments at scale requires more than just dividing users into groups. You need to ensure every user sees the same experience every time they visit, track metrics reliably across millions of events, and determine statistical significance without months of waiting. A well-designed A/B testing platform is the backbone of data-driven product decisions, and getting the architecture right separates smooth experimentation from inconsistent chaos.

Architecture Overview

An A/B testing platform brings together four essential layers working in concert. The assignment layer determines which variant each user sees based on their ID and experiment configuration, using consistent hashing to guarantee the same assignment every single time. The data collection layer captures user events, metrics, and interactions in real-time, flowing into a high-throughput event stream. The analytics engine processes these events, calculates conversion rates, and computes statistical significance using established methodologies like chi-square tests. Finally, the experiment management layer provides the interface for creating experiments, configuring rules, monitoring performance, and eventually declaring winners.

The key insight here is separation of concerns. Assignment logic lives independently from metric tracking because they have different latency requirements, consistency guarantees, and scaling challenges. The assignment service must respond in milliseconds and be always consistent, while the analytics pipeline can tolerate slight delays in exchange for deep aggregations and statistical rigor. Between them sits an event streaming system like Kafka that decouples real-time assignments from batch analysis, allowing you to replay events and recalculate metrics without affecting active experiments.

Deployment considerations matter too. The assignment service should be deployed close to your application (often as a sidecar or library), minimizing latency and reducing dependencies. Your analytics infrastructure can live further downstream, processing data in batches or micro-batches depending on how quickly you need insights. Configuration for active experiments must be cached aggressively at the edge, with fallback mechanisms when the assignment service is unreachable.

Design Insight: Sticky User Assignment

The most common question we see: how do you ensure users always see the same variant across sessions? The answer is deterministic assignment using consistent hashing. When a user enters an experiment, you hash their user ID together with the experiment ID using an algorithm like MurmurHash or xxHash. This produces a pseudo-random but deterministic value between 0 and 1. You then map that value to a variant based on allocation percentages, for example: if you're testing variant A at 50% and variant B at 50%, any user whose hash falls between 0 and 0.5 always sees A, and those between 0.5 and 1 always see B.

This approach requires zero additional storage or lookup calls. The assignment is instant, consistent across all your services, and survives service failures. Even better, if a user's ID is the only input, the same user gets the same variant whether they're on mobile, desktop, or using a different browser, as long as you can identify them consistently. Some platforms add additional entropy like geographic region or user cohort to the hash input, creating "stratified randomization" that balances variants across important dimensions.

Watch the Full Design Process

Want to see how this architecture comes together in real-time? We built a complete A/B testing platform design live, exploring assignment mechanisms, metric tracking, and consistency strategies. Watch the full system design process on your preferred platform:

Try It Yourself

This is Day 89 of our 365-day system design challenge, and we're constantly exploring new architectures. Want to design your own A/B testing platform or tackle a different system design problem? Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document. No whiteboarding, no guesswork, just clear architecture aligned with your requirements.

Day 90: Data Lake Architecture - AI System Design in Seconds

Matt Frank — Sun, 05 Jul 2026 13:03:13 +0000

Organizations generate data in every format imaginable, from structured database records to videos, sensor streams, and logs. A data lake promises to centralize this chaos, but without proper architecture, it quickly becomes a data swamp where data sits forgotten, undocumented, and impossible to find. Getting this right is critical for analytics platforms that need to process petabytes of heterogeneous data while maintaining governance and discoverability.

Architecture Overview

A modern data lake architecture centers on three key layers: ingestion, storage, and governance. The ingestion layer uses connectors and streaming services to pull structured, semi-structured, and unstructured data from various sources, normalizing them into standardized formats where possible. Data flows into a distributed storage layer, typically built on object storage systems, where it's organized into bronze, silver, and gold zones. Bronze holds raw, unprocessed data exactly as it arrived. Silver contains cleaned, deduplicated, and validated datasets ready for analytics. Gold provides curated, business-ready tables optimized for specific use cases and dashboards.

What ties this system together is metadata and governance. Every dataset entering the lake gets cataloged with lineage information, schema details, ownership, and compliance tags. A data catalog acts as the single source of truth, allowing data engineers and analysts to discover what's available, understand its quality, and know who to contact with questions. This isn't optional infrastructure, it's the foundation that separates a functional data lake from a data swamp.

Compute separation is another key design principle. Rather than coupling storage and processing, the architecture isolates them so you can scale independently. Multiple processing engines, query engines, and machine learning platforms can access the same storage layer without contention. This flexibility lets different teams use their preferred tools while maintaining a single source of truth.

Design Insight: Preventing the Data Swamp

The difference between a data lake and a data swamp comes down to governance maturity. A data swamp lacks metadata, has unclear ownership, contains duplicate and stale datasets, and offers no quality guarantees. To prevent this, you need automated data quality checks at ingestion and regular validation pipelines that catch schema drift and anomalies. Clear naming conventions, mandatory documentation, and automated lineage tracking make datasets discoverable and trustworthy. Equally important is retention policies, archival processes, and periodic audits to remove datasets that no longer serve business needs. When you build governance into the architecture from day one rather than bolting it on later, you stay on the data lake side of the line.

Watch the Full Design Process

I walked through this entire data lake architecture in real-time, showing how each component connects and the reasoning behind key decisions. You can watch the full design demonstration across multiple platforms:

Try It Yourself

Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document.

Day 88: Log Aggregation - AI System Design in Seconds

Matt Frank — Sat, 04 Jul 2026 20:00:13 +0000

In today's microservices world, logs are everywhere. Your hundreds of services are constantly generating messages, and without a centralized system to collect and analyze them, debugging production issues becomes a nightmare. A well-designed log aggregation platform is your safety net, enabling quick incident response, performance monitoring, and compliance auditing across your entire infrastructure.

Architecture Overview

A robust log aggregation system needs three core responsibilities: collection, processing, and querying. On the collection side, lightweight agents run on each service or node, capturing logs and forwarding them to a central pipeline. These agents are designed to be stateless and low-overhead, so they don't impact your application's performance. They typically batch logs together and use connection pooling to minimize network overhead.

The processing layer is where the magic happens. Logs stream into a message broker like Kafka or RabbitMQ, which acts as a buffer between producers and consumers. This decoupling is critical because it allows you to ingest logs at whatever rate services produce them, without losing messages. From the message broker, stream processors normalize log formats, extract structured fields, and enrich logs with metadata like service names and deployment versions. This transformation layer ensures consistency across your entire system.

Finally, the storage and query layer handles the indexed data. Most systems use Elasticsearch or similar document stores that excel at full-text search and time-series analysis. The indexing strategy matters tremendously here, typically organizing logs by date and service to optimize both storage and query performance. Alongside your primary store, you might maintain a colder storage tier like S3 for long-term retention and compliance requirements, keeping hot storage lean and responsive.

The Flow

Logs travel through a predictable journey: agents collect them, the message broker queues them reliably, processors transform and enrich them, and finally they land in your search index. This architecture ensures no message is lost during normal operations while keeping query latencies low for your operations team.

Handling Traffic Spikes: Buffering is Your Friend

So what happens when log volume suddenly increases tenfold? This is where the message broker becomes invaluable. Instead of pushing logs directly to your index, the broker acts as a shock absorber. When a traffic spike hits, logs pile up in the queue while your processing cluster methodically works through the backlog. The key is to size your processing capacity to handle the sustained load once the spike subsides, not just the peak.

To truly handle 10x volume without dropping messages, you need multiple strategies working together. First, implement auto-scaling on your processor instances so they spin up quickly as queue depth grows. Second, use separate processing pipelines for high-priority logs versus debug logs, ensuring critical data gets through even when the system is overwhelmed. Third, add circuit breakers that gracefully degrade indexing for non-critical fields during overload, keeping the core log message and timestamp always searchable. Finally, disk-backed queues ensure that even if your broker restarts, queued messages survive.

Watch the Full Design Process

See how this architecture comes together in real-time:

Try It Yourself

Ready to design your own log aggregation system? Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document. Whether you're handling thousands or millions of logs per second, you'll have the foundation to build confidently.

This is Day 88 of our 365-day system design challenge. What architecture challenge will you tackle next?

Day 89: A/B Testing Platform - AI System Design in Seconds

Matt Frank — Sat, 04 Jul 2026 13:03:10 +0000

A/B Testing Platform Architecture: Ensuring Consistency Across User Sessions

Running experiments at scale means making split-second decisions for millions of users, but there's a fundamental challenge lurking beneath the surface. How do you ensure that when Sarah returns to your app tomorrow, she sees the exact same variant she saw today, even when your infrastructure has scaled to handle thousands of concurrent experiments? This architectural problem sits at the heart of every modern A/B testing platform, and solving it elegantly determines whether your experiments produce reliable data or become a statistical nightmare.

Architecture Overview

An A/B testing platform needs to orchestrate four interconnected systems working in harmony. The Experiment Management Service handles the lifecycle of your experiments, from creation through analysis. It stores experiment definitions, control groups, variant configurations, and statistical thresholds. Meanwhile, the User Assignment Engine is where the magic happens, responsible for deterministically assigning users to variants based on a combination of factors like user ID, experiment ID, and a consistent hashing algorithm.

The Metric Tracking System captures every interaction users generate, whether that's clicks, conversions, page views, or custom events. These raw events flow into a time-series database or data warehouse where they can be aggregated and analyzed. Finally, the Statistical Analysis Service processes accumulated metrics to calculate conversion rates, confidence intervals, and p-values, determining when results reach statistical significance.

What ties these components together is a well-designed assignment strategy. Rather than randomly picking variants each time a user visits, the system uses a deterministic approach. By hashing the user's ID combined with the experiment's ID, you always get the same output. This hash determines which variant bucket the user falls into (variant A gets hashes 0-49, variant B gets hashes 50-99, for example). This approach is computationally cheap and eliminates the need to store every single assignment in a database.

The Assignment Flow

When a user loads your application, the client or server calls the Assignment Engine with the user ID and experiment ID. The engine performs the hash calculation, checks if the user is even eligible for this experiment (respecting targeting rules and exclusion criteria), and returns the assigned variant. This happens in milliseconds, making it suitable for real-time request handling.

Design Insight: Session Consistency Through Deterministic Hashing

The answer to ensuring users see the same variant across sessions lies in removing randomness from the equation. Instead of storing assignment decisions in a separate lookup table, deterministic hashing makes the assignment a pure function of user ID and experiment ID. Every time Sarah's browser calls the assignment endpoint, the same user ID plus the same experiment ID produces the same hash bucket, guaranteeing the same variant assignment.

This approach solves several problems simultaneously. It scales without requiring a massive assignment database, it handles new users without any lookup latency, and it naturally handles the "same variant across devices" problem if you're assigning based on a universal user ID rather than device ID. The trade-off is that you can't change user assignments retroactively without risking data corruption, so your experiment design must be finalized before running traffic through it.

Watch the Full Design Process

Want to see how this architecture comes together in real-time? We've documented the complete design process across multiple platforms:

Try It Yourself

Building a system design from scratch can feel overwhelming, but InfraSketch makes it intuitive. Head over and describe your A/B testing platform (or any system) in plain English. In seconds, you'll have a professional architecture diagram complete with a design document. Whether you're preparing for an interview, designing your company's next platform, or just exploring system architecture, this tool turns description into diagrams instantly.

This is Day 89 of our 365-day system design challenge. Start designing today.

Day 87: Recommendation Engine - AI System Design in Seconds

Matt Frank — Fri, 03 Jul 2026 20:00:14 +0000

Recommendation Engine: Architecting Intelligence at Scale

Building a recommendation engine that feels like magic requires orchestrating multiple layers of intelligence, data processing, and real-time decision-making. The systems powering Netflix, Spotify, and YouTube aren't just storing user preferences, they're solving a fundamentally complex problem: how to predict what someone will love before they know it themselves. This is the challenge of modern streaming platforms, and getting the architecture right can mean the difference between engagement and abandonment.

Architecture Overview

A production recommendation engine typically consists of four interconnected layers working in concert. The Data Ingestion Layer captures raw signals continuously: user views, pause points, ratings, search queries, and click-through behavior. This data flows into a Feature Engineering Pipeline that transforms raw events into meaningful signals like viewing duration ratios, genre affinity scores, and temporal patterns. These features feed into multiple recommendation models operating in parallel, including collaborative filtering (learning from similar users), content-based filtering (matching item attributes), and hybrid approaches that blend both strategies.

The Real-Time Serving Layer is where architecture complexity becomes critical. When a user opens the app, you can't afford to recompute recommendations from scratch. Instead, you maintain pre-computed candidate sets, ranking models, and personalization rules that can serve results in milliseconds. This layer typically connects to a fast cache layer (Redis or similar), a lightweight ranking service, and a feature store that provides instant access to precomputed user and item embeddings.

The final piece is the Feedback Loop and Iteration System. Every recommendation served generates implicit feedback: was it clicked, watched, or ignored? These signals flow back into the training pipeline, allowing models to improve continuously. This closed-loop design is what makes modern recommendation systems adaptive, constantly evolving toward better predictions.

Design Decision: Hybrid Filtering Strategy

Pure approaches rarely work in production. Collaborative filtering excels at discovering serendipitous content by learning from "people like you," but it can miss obvious matches. Content-based filtering ensures relevance by matching item attributes but risks creating filter bubbles. The best architectures blend both, running them in parallel and using ranking models to intelligently combine their outputs based on context.

Design Insight: Solving the Cold Start Problem

The cold start problem haunts recommendation engines: new users have no history, so collaborative filtering has nothing to learn from. New items have no ratings, so they're invisible to the algorithm. Production systems handle this through a multi-pronged approach. For new users, content-based filtering combined with curated onboarding experiences (asking about favorite genres, for example) provides initial recommendations while collaborative filtering "wakes up" after a few interactions. For new items, content metadata, editorial signals, and A/B testing help surfaces promising content to small cohorts of engaged users. Some teams implement "exploration arms" in their ranking function, deliberately showing new content to diverse user segments to generate training data faster. The key insight: cold start isn't a problem to solve once, it's an ongoing challenge requiring explicit strategies layered throughout your architecture.

Watch the Full Design Process

See how this architecture comes together in real-time, from blank canvas to complete system design:

Try It Yourself

Want to design your own recommendation system? Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document. This is Day 87 of the 365-day system design challenge, and InfraSketch makes it easy to go from concept to architecture without the whiteboard friction.

Day 88: Log Aggregation - AI System Design in Seconds

Matt Frank — Fri, 03 Jul 2026 13:04:09 +0000

In today's microservices world, logs are everywhere. Your services generate them constantly, but without a centralized system to collect, index, and search them, you're flying blind when debugging production issues. A well-designed log aggregation system transforms scattered log files into searchable, actionable insights across hundreds of services.

Architecture Overview

A robust log aggregation platform consists of three distinct layers working in harmony. The collection layer sits closest to your services, gathering logs from multiple sources through lightweight agents or direct API calls. These collectors buffer incoming logs and forward them to a central ingestion point, ensuring no data is lost during transit. The processing layer receives this firehose of data, applies transformations, filters, and enriches logs with metadata like service names and timestamps. Finally, the storage and search layer indexes logs in a distributed system optimized for fast retrieval and analytics.

The key components that make this work are message queues (like Kafka or RabbitMQ) that act as shock absorbers between collection and processing, distributed storage systems (like Elasticsearch or specialized time-series databases) that handle petabytes of log data, and a query interface that lets engineers search across millions of log entries in seconds. Think of the message queue as your safety net, it prevents backpressure from overwhelming your collectors when the processing layer gets busy.

Design decisions here are critical. You'll want to partition logs by service or time to distribute the load, implement retention policies to manage storage costs, and add monitoring on the aggregation system itself, so you know if logs are being dropped. The whole system needs to be redundant and fault-tolerant because losing logs during an outage defeats the purpose of having them.

Handling a 10x Spike in Log Volume

This is where architecture really matters. When a service goes viral or a bug causes a logging loop, your log volume can spike dramatically without warning. The answer isn't to add infinite processing power, it's to build strategic buffering and prioritization into your design. Message queues act as the first line of defense, absorbing sudden spikes and letting you process logs at a steady rate behind the scenes. If you're still dropping messages, implement tiered storage: critical logs (errors, exceptions) get indexed immediately, while lower-priority logs (debug statements) are batched and indexed asynchronously or sampled to reduce volume.

Auto-scaling is your friend here. Your processing layer should automatically spin up more workers when queue depth increases, distributing the load across more machines. You can also implement backpressure, gracefully slowing down log ingestion if your system can't keep up, rather than crashing and losing everything. The key insight is that losing some debug logs during a crisis is better than losing all logs because your system collapsed.

Watch the Full Design Process

See how this architecture comes together in real-time with AI-powered diagram generation. Watch the full design process and architecture walkthrough across your favorite platforms:

Try It Yourself

Ready to design your own log aggregation system? Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document. No drawing skills required, no wrestling with diagramming tools. Whether you're planning a new platform or optimizing an existing one, you can iterate on designs instantly and explore trade-offs like we did with the 10x spike scenario.

This is Day 88 of the 365-day system design challenge. Tomorrow we'll tackle another critical architecture problem. What system would you like to see next?