Over the past few months, AI agents have gone from being little experiments to becoming actual coworkers. They're writing code, browsing the web, answering emails, generating reports, and orchestrating workflows that would normally require entire teams. The more I thought about that future, the more one question kept bothering me.
What happens when there aren't ten AI agents? What happens when there are fifty thousand?
Everyone loves talking about what AI agents can do, but very few conversations revolve around what happens behind the scenes once they begin operating at scale. Every one of those agents continuously changes state. They start tasks, finish them, fail unexpectedly, retry operations, wait for resources, and move through dozens of transitions during their lifetime. Every transition is an event that needs to be captured, observed, and sometimes persisted forever.
That sounds simple until thousands of agents decide to do it at the exact same moment.
At that point, your application isn't just handling API requests anymore. It's trying to process an overwhelming stream of concurrent events without losing information, without destroying database performance, and without making users stare at a dashboard that updates several seconds behind reality.
That became the problem I wanted to solve.
I had recently started reading Designing Data-Intensive Applications, a book that has quickly become one of my favourite engineering books. Rather than reading it like a textbook, I wanted to approach it differently. Every chapter became a challenge to build something inspired by the ideas I had just learned.
Instead of simply reading about distributed systems, I wanted to experience them.
So I asked Claude to generate a realistic system design interview question.
It came back with this:
How do you ingest, strictly order, and visualize concurrent state transitions from 50,000 autonomous AI agents executing distributed workflows in real time, ensuring zero dropped events and sub-second dashboard updates without overwhelming your persistent audit database?
I stared at the question for a while.
Then I grabbed a pen and paper.
Before writing a single line of code, I sketched the architecture.
That habit has become one of the biggest improvements in my engineering journey. I rarely jump straight into an editor anymore because architecture forces you to think about constraints before implementation. It exposes bottlenecks early, encourages better questions, and often removes complexity before it ever reaches the codebase.
As I continued sketching, one realization became obvious.
The dashboard and the database had completely different priorities.
A live dashboard cares about speed. It wants updates almost immediately because users expect to watch agents move through their workflows in real time. A database, however, cares about durability and efficiency. It doesn't benefit from writing every single event individually if those writes can instead be grouped together without losing correctness.
Trying to satisfy both with the same processing pipeline felt like a mistake.
So I separated them.
Beacon sits between AI agents and every downstream system.
Incoming events are accepted immediately by lightweight gateway services before being published into Kafka. Rather than sending every event directly to a database, Kafka becomes the central event log that absorbs bursts of traffic while preserving ordering for each individual agent.
That ordering is surprisingly simple.
Every event is produced using the agent's identifier as the Kafka message key. Kafka guarantees ordering inside each partition, so every event belonging to the same agent always arrives in the order it was produced. No expensive sorting logic is required later because the ordering guarantee already exists inside the messaging system itself.
Sometimes the simplest decisions end up removing entire categories of complexity.
Once events enter Kafka, the pipeline splits into two completely independent consumer groups.
The first consumer exists solely for the live dashboard. Its responsibility is simple: consume new events and immediately broadcast them to connected browsers using WebSockets. Users watching the dashboard see state transitions appear almost instantly, regardless of what is happening elsewhere in the system.
The second consumer has a completely different job.
Instead of prioritizing speed, it prioritizes efficiency. Events are accumulated in memory before being written to persistent storage in batches. This dramatically reduces the number of database transactions while maintaining a complete audit history. Offsets are only committed after a successful batch write, ensuring that failures never silently discard data.
Neither consumer blocks the other.
The dashboard remains responsive even if persistence slows down, and the database continues writing efficiently even during bursts of activity.
That separation became the core architectural idea behind Beacon.
While building the project, I intentionally chose technologies that emphasized the architectural ideas rather than production complexity.
Redpanda provides a lightweight Kafka-compatible broker that is incredibly easy to run locally. SQLite keeps persistence simple while allowing the storage layer to be swapped for PostgreSQL later without changing the overall design. FastAPI made building independent services straightforward, while React handled the live dashboard that visualizes everything happening inside the system.
The result is a project that demonstrates distributed event processing without requiring an entire Kubernetes cluster to understand.
Beacon also reminded me that system design is far more enjoyable when ideas become software.
Reading about partitions, event streams, consumer groups, batching strategies, and ordering guarantees is valuable, but seeing those concepts come alive inside a running system changes how you understand them. Suddenly they stop being interview buzzwords and become practical engineering tools for solving real problems.
That has become my favourite way to learn.
Read something.
Think about it.
Build it.
Break it.
Understand it.
Beacon is completely open source because I wanted it to become more than just another portfolio project. Whether someone wants to learn distributed systems, monitor their own AI agents, experiment with Kafka, or contribute improvements, I hope the project becomes a useful starting point.
If you've ever wanted to understand how large-scale event pipelines work without reading thousands of lines of production code, I hope Beacon gives you that opportunity.
I'll continue building projects inspired by the ideas I learn, documenting the process, and sharing both the successes and the mistakes along the way.
Because at the end of the day, that's how I learn best.
⭐ GitHub: Beacon
📺 Full build video: I Built a System That Can Handle 50,000 AI Agents (Open Source)
Architecture Diagram:
Top comments (0)