For a long time, most of the things I built looked similar. A user opens an application, clicks a button, a request is sent to the backend, the backend processes it, stores some data, and returns a response. Whether it was a booking system, a social platform, or another web application, the pattern was always familiar. I was focused on building products that users could interact with directly.
When I decided to start learning system design, I promised myself I wouldn't do it the way I usually learn new topics. Instead of spending weeks watching videos or reading articles before writing any code, I wanted to learn by building something that forced me to think differently. That decision eventually became Antlers.
At first, I thought I was building a real-time heatmap. Looking back now, I realize that the heatmap was probably the least interesting part of the project. What I was really building was the infrastructure required to continuously process thousands of events, transform them into something meaningful, and deliver those insights to users with almost no delay.
The original problem was inspired by something I imagined happening inside a company like Glovo or Uber Eats. Suppose the marketing or operations team wants to display a live heatmap showing where orders are currently being placed across a city. From a customer's perspective, this creates excitement and social proof by showing that the platform is active. From an operations perspective, it provides immediate visibility into where demand is increasing.
The obvious implementation would be to save every driver location or order event to the database and have the frontend periodically request the latest data. That approach works when there are only a handful of users. However, once you begin thinking about thousands of drivers sending location updates every few seconds, it becomes clear that constantly writing to the database and repeatedly querying it is an expensive way to answer a simple question: "Where is activity happening right now?"
That realization completely changed the way I approached the project.
Instead of treating the database as the center of the system, I started thinking about the movement of data itself. A driver's location is not just something that should be stored forever. It is first an event. Events arrive, they are processed, transformed into useful information, and only then do you decide what is worth storing permanently. This small change in perspective led me toward an event-driven architecture.
Rather than processing everything inside the API, incoming location updates are immediately sent to Kafka, allowing the API to remain responsive regardless of how much traffic it receives. Dedicated consumers then process those events and aggregate them into geohash buckets inside Redis. Instead of broadcasting thousands of individual GPS coordinates to connected clients, the backend only sends aggregated density information through WebSockets, allowing the frontend to reconstruct the heatmap efficiently.
One of the most interesting lessons from this architecture was understanding that not every piece of information deserves to live in a relational database. Driver locations are constantly changing. By the time one coordinate has been written to disk, the driver may already be somewhere else. Redis turned out to be a much better fit because it allows the system to maintain an in-memory representation of what the city looks like right now, rather than what it looked like several seconds ago.
As the project evolved, I started asking myself another question. If the system already knows where drivers are and how demand changes over time, why should it only describe the present? Why couldn't it make reasonable predictions about the immediate future?
My first thought was to integrate an external AI API, but the more I considered that approach, the less it made sense. Continuously sending snapshots of operational data to an external model would introduce additional latency, recurring costs, and API rate limits. More importantly, the prediction problem itself wasn't particularly complicated. I wasn't trying to generate natural language or summarize documents. I simply wanted to estimate how demand was changing over time.
Instead, I began experimenting with a lightweight regression model trained on periodic snapshots taken directly from Redis. Every few minutes, the system records the current state of activity across the city, allowing the model to gradually learn how demand evolves throughout the day. The longer the system runs, the more historical context it accumulates, all without relying on any external AI provider.
The biggest takeaway from building Antlers wasn't learning Kafka, Redis, or WebSockets individually. It was realizing that system design is much less about choosing technologies and much more about deciding where responsibility should live. Once each component has a single responsibility, ingesting events, buffering traffic, aggregating state, broadcasting updates, or making predictions, the overall system becomes much easier to reason about and extend.
Top comments (0)