DEV Community: Anshu Kumar

System Design 01 : Before You Draw a Single Box in System Design

Anshu Kumar — Wed, 28 Oct 2026 12:57:39 +0000

Imagine you're in a system design interview, and a senior engineer asks you to design a notification system that can deliver millions of notifications.

You open Excalidraw and start drawing. An API server, Kafka, Redis, a bunch of workers, and a database. You've seen similar architectures before, so you know roughly where everything goes.

Then the senior engineer asks:

"How many of these notifications need to be delivered immediately, and how many can wait?"

You pause.

Because the answer could change how you design the entire system.

And that's just one of the questions you haven't asked yet.

The hardest part of system design isn't drawing the architecture. It's figuring out what architecture the problem actually needs.

So, before we draw another box, let's start with the questions that should come first.

One notification system, several different problems

Let's start with something familiar.

Imagine you're building a social application. Someone likes a post, another user sends a message, and someone else logs in from a new device.

The application wants to notify the relevant users about these events.

At first, this sounds like a straightforward feature: receive an event, find the recipient, and send a notification.

But consider what happens if we treat all three events the same way.

A message notification might need to arrive within seconds. A like notification can probably tolerate a short delay. A security alert about an unfamiliar login might need to be delivered as quickly as possible.

And a weekly email digest? That can be processed in batches whenever the system has capacity.

We have one product feature, but several different expectations.

This is where the design starts.

Not with a database choice. Not with Kafka. Not with a diagram copied from another system.

With the behavior the system is expected to provide.

Let's say we ask the product team about these use cases and establish the following initial requirements:

These are illustrative assumptions, not universal requirements. A real system might have different latency targets, channels, or priorities.

But writing them down gives us something useful: a description of the problem that we can challenge and refine.

For example, what does "near real-time" actually mean? Two seconds? Five seconds? And does the target refer to accepting a notification, beginning delivery, or the recipient actually receiving it?

Those distinctions will matter later.

For now, we know that our system must support different notification types, multiple channels, and different latency expectations.

That's already more useful than simply saying "millions of notifications."

What does "millions" actually tell us?

Let's say the requirement is ten million notifications per day.

Is that enough to decide how many servers we need?

Not yet. But we can start with a rough estimate.

Our average arrival rate is approximately 116 notifications per second.

That calculation is useful, but it doesn't tell us how the traffic is distributed.

Suppose most of those notifications arrive during a two-hour window. Or a scheduled campaign suddenly creates a large burst of work. Or an external provider goes down, and the system needs to process a backlog after it recovers.

In each case, the system might need to handle much more work at particular moments than the daily average suggests.

Let's record what we know and what we don't:

Notice that we haven't invented a peak number just to make the design look complete.

We have an estimate, an assumption about delivery behavior, and an unanswered question about peak traffic.

We can ask for more information if it's available. Otherwise, we can state an assumption, proceed with the design, and revisit it if the assumption turns out to matter.

This is an important habit: distinguish facts from assumptions instead of silently treating both as requirements.

Now let's see what happens when we try to build the simplest possible system.

Version 1: Just send the notification

Our first design could be very simple.

The application calls our Notification API. The API validates the request and calls the appropriate external provider.

An external provider is simply another service that performs the actual delivery. For email, that might be an email delivery service; for push notifications, it might be a platform's push notification service.

Our application doesn't need to implement the entire delivery infrastructure itself.

For a small application with modest traffic, this design might be perfectly reasonable.

But let's introduce a constraint.

Suppose the email provider takes several seconds to respond. Our Notification API is waiting for that response, which means the application calling our API may also have to wait.

Now imagine the provider becomes temporarily unavailable.

Requests can pile up, connections remain occupied, and a problem with an external dependency can affect the application itself.

The issue isn't that direct delivery is always wrong. It's that direct delivery couples the application's request to the provider's response time and availability.

Can we separate those two things?

Version 2: Accept the work now, deliver it later

Imagine going to a restaurant during a busy evening.

The staff takes your order, records it, and gives it to the kitchen. You don't need to stand in the kitchen and wait for the food to be prepared before the restaurant acknowledges your order.

The order has been accepted. Preparation happens separately.

A notification system can use a similar idea.

Instead of sending the notification directly from the API, we can place the work in a queue.

A queue is a structure that holds work until a consumer is ready to process it. A worker can take a queued notification and attempt delivery independently of the original API request.

A durable queue goes a step further: it retains accepted messages in persistent storage so that a process restart or machine failure doesn't simply erase pending work. The exact guarantees depend on the queue technology and its configuration.

Our architecture now looks like this:

The Notification API validates the request and places a message in the queue. A delivery worker reads the message and contacts the provider.

The API no longer needs to wait for the provider to finish sending the notification.

We've separated two operations:

Accepting responsibility for processing the notification.
Attempting to deliver it to the recipient.

This is asynchronous processing. The system accepts work now and completes it separately, rather than making the caller wait for the entire operation.

But this introduces an important question: what exactly does the API promise when it returns success?

If the API returns success after the notification has been durably accepted, it can tell the caller that the system has accepted the work.

It cannot honestly claim that the recipient has received the notification.

Those are different guarantees.

Even this simplified flow forces us to define what each state means.

For example, a provider accepting an email doesn't necessarily mean the email reached the recipient's inbox. If the product needs that distinction, our state model must represent it.

And notice how we arrived at the queue. We didn't add it because a large system is supposed to have one. We added it because we wanted to isolate API request handling from slow or unavailable delivery providers.

That's a design decision we can explain.

Version 3: A queue doesn't magically solve every problem

Our queue protects the API from waiting for the provider. It can also hold pending work when delivery slows down.

But what happens when the system receives notifications faster than its workers can process them?

The queue grows.

A queue is useful for absorbing bursts, but it doesn't create unlimited processing capacity. If incoming work continuously exceeds processing capacity, the backlog will keep increasing.

We need to understand both sides of the system.

If the arrival rate temporarily exceeds the processing rate, the queue can absorb the difference.

If the arrival rate stays higher for long enough, the system eventually needs more capacity, less incoming work, or a different processing strategy.

We should monitor the queue's size and age, not just whether the API is returning successful responses.

Why the age of queued messages?

Because a queue containing a million messages isn't necessarily unhealthy if they are allowed to wait for hours. A much smaller queue might be a serious problem if security alerts are sitting in it for thirty seconds.

The important measurement depends on the requirement.

For our system, useful measurements might include:

How many notifications are waiting.
How long the oldest notification has been waiting.
How long delivery attempts take.
How many attempts fail.
Whether high-priority notifications meet their latency targets.

This is where requirements become operational measurements.

"Security alerts should arrive quickly" isn't enough on its own. We need a measurable target to determine whether the system is meeting that expectation.

Version 4: What if an important notification is stuck behind millions of emails?

Let's return to the notification types we identified earlier.

Imagine a marketing campaign schedules ten million emails. At the same time, a user triggers a security alert.

If both kinds of work share the same queue and processing capacity, the security alert could wait behind bulk traffic.

The queue is working exactly as designed. The problem is that the design doesn't necessarily protect our high-priority workload.

We now have a few options.

We could let the workers prioritize messages within a shared queue. We could create separate queues for different classes of work. Or we could reserve some worker capacity for high-priority notifications.

Each option has trade-offs.

A shared queue can be simpler to operate, but priority scheduling depends on the capabilities of the queue and consumer design.

Separate queues offer more isolation, but require additional configuration and operational attention. They also don't automatically guarantee fast delivery: if the high-priority workers are overloaded or the provider is slow, latency can still increase.

Suppose we decide that the isolation is worth the added complexity.

Our design could evolve into this:

This is one possible design, not a rule that every notification system needs two queues.

The decision depends on the latency requirements, workload, queue technology, and cost of operating separate processing paths.

More importantly, we've now seen the design change for a reason: our original requirements told us that not all notifications could wait equally long.

That is how architecture should evolve.

Now let's talk about failure

Suppose a worker sends a notification to an external provider.

The provider receives the request, but its response never reaches the worker. The worker times out.

Did the provider send the notification?

We don't know.

The worker has an ambiguous outcome: the request may have failed before delivery, or the provider may have accepted it even though the response was lost.

If the worker retries, the recipient might receive the same notification twice. If the worker doesn't retry, a notification that was never accepted might never be sent.

This is why "just retry on failure" isn't a complete reliability strategy.

At a high level, we need to think about:

Which failures are temporary and worth retrying.
How many attempts are allowed.
How long to wait between attempts.
What happens when all attempts fail.
How we track delivery attempts and outcomes.
Whether the provider supports idempotency mechanisms that can reduce duplicate effects.

A retry policy defines when and how failed work is attempted again. Some systems also move repeatedly failing messages into a dead-letter queue, a separate place for messages that need further investigation or special handling.

We don't need to implement all of that in this first design exercise. The point is to recognize that a new requirement, recovering from temporary failures introduces new responsibilities.

And even a durable queue doesn't solve all of them. It can preserve pending work, but it cannot guarantee that an external provider delivers a notification exactly once.

We'll explore retries, idempotency, and reliable state changes in more depth later in the series.

For now, let's capture the design decisions we've made and the reasons behind them.

This is much more useful than a list of technologies.

It tells us what we chose, why we chose it, and what information could still change the design.

So, do we need Kafka and Redis?

We haven't actually established that yet.

We've established that we need a way to accept notification requests, process them asynchronously, and preserve pending work according to our reliability requirements.

A durable queue is one possible component that provides those capabilities.

Kafka is a distributed event-streaming platform that can retain records and allow consumers to process them independently. It may be a good fit when we need high throughput, replay, or multiple independent consumers, but it also brings operational and design considerations.

A managed queue service might satisfy our requirements with less operational effort.

Redis can be useful for caching, rate limiting, and certain queue-like workloads, depending on the data structures and persistence guarantees involved. But we shouldn't automatically assume that a Redis-backed queue provides the same durability and delivery guarantees as every dedicated queue or event-streaming system.

We would compare the actual guarantees, workload, operational complexity, and cost before choosing.

The same applies to our database. We might need to persist notification state, user preferences, and delivery attempts. But the appropriate data model depends on which queries we need to support, how much data we retain, and what consistency guarantees the product requires.

The technology should follow from those decisions.

If we choose Kafka simply because the prompt says "millions," we've skipped the most important part of the exercise.

The architecture is still a hypothesis

Let's step back and look at what we have:

This isn't a complete production architecture. It doesn't yet specify the schema, exact API contract, worker concurrency, retention policy, provider failover strategy, or every failure-handling detail.

And that's okay.

We now have a baseline design that follows from the requirements we've discussed. We can identify the remaining questions and investigate the ones that could materially change our choices.

For example, if users must query the status of every notification, we'll need to define how delivery state is stored and retrieved.

If the system must meet strict latency targets, we'll need to calculate peak capacity and examine how queueing affects end-to-end latency.

If notifications must respect changing user preferences, we'll need to decide when those preferences are checked and what happens to work already waiting in a queue.

Each new constraint can lead to another revision.

The goal isn't to draw the most complicated architecture possible. It's to reach a design that satisfies the known requirements, then understand where it might fail as those requirements change.

The habit worth building

When you face an unfamiliar system, don't start by asking which database, queue, or cache you should use.

Start by understanding what the system must do.

Find out who uses it, what operations it supports, what the important latency and reliability expectations are, and how much work it needs to handle.

Write down the requirements and assumptions so you can distinguish what is known from what is merely plausible.

Then build the simplest design that satisfies them.

When a constraint exposes a limitation, evolve the design. Explain why the change is necessary, what it improves, and what new complexity it introduces.

You don't need to answer every possible question before drawing anything. You do need to know which unanswered questions could invalidate your current design.

And don't treat a technology name as an explanation. Saying "we'll use Kafka" doesn't explain how the system handles a provider outage, how long work is retained, or what the API promises when it returns success.

Those are the decisions that give an architecture its meaning.

A good system design is not a collection of familiar boxes. It's a set of decisions that you can trace back to requirements, constraints, and explicit trade-offs.

Once you learn to think that way, the diagram becomes a way to communicate your reasoning not a substitute for it.

In the next article, we'll explore what happens after a system accepts a request: how to design APIs, model data, and represent state transitions so that the system's behavior remains predictable as it grows.

I built a company brain in 3 days. Notes on memory, evidence, and what broke.

Anshu Kumar — Sun, 18 Oct 2026 17:05:23 +0000

Why am I even thinking about company memory?

Whenever we talk about AI brain for companies, the first thing that usually comes to mind is a chatbot over some documents. You upload your docs, you ask questions, you get answers.

But I think the actual problem is more interesting than that.

Imagine I am an engineer at a company and I ask:
"Why did INC-231 happen?" (INC-231 is just an incident ID, don't worry about it)

What exactly should happen?

The system needs to understand that I am not asking for a document. I am asking for a causal chain (causal as in cause-and-effect, not a typo for casual). It needs to:

Find the incident in the company's history.
Trace what caused it: which PR, which decision, which commit.
Show me the path, not just the answer.
Cite every source along the way.
Tell me who was involved and who knows the most about it.

And now take this one step further.

What if I then ask:
"What breaks if we migrate the ledger database?"

The system needs to follow service dependencies outward, rank the impact, and then, before doing anything about it, stop and ask a human for permission.

That's where the idea of Recall comes in.

First, let's understand the problem

Company knowledge doesn't live in one place. It lives in:

Incidents (what broke)
PRs and commits (what changed)
ADRs (why decisions were made)
Chat messages (what the team actually said)
People (who knows what)

A normal RAG system chunks all of this into vectors and hopes semantic similarity finds the right pieces. But the questions I care about aren't similarity questions. They're connection questions:

"Why did this happen?" is a path question.
"What breaks if I touch this?" is a traversal question.
"Who knows this system?" is a ranking question.

These are graph questions. So I built a graph.

The graph model

The core idea is simple. Everything a company knows becomes nodes and edges:

People, teams, services, incidents, PRs, commits, decisions, chat messages. All connected. All queryable with Cypher.

But here's the thing I want to be clear about. The demo company, Meridian Pay, is fictional. Every person, incident, and message is seeded test data. What's real is the FalkorDB instance, the queries, and the agent behavior. I say this upfront because the whole point of the project is honesty about evidence.

What "shows its work" actually means

When you ask "why did INC-231 happen?", the agent doesn't just generate an answer. It runs a causal traversal:

Every claim in the answer links back to a node. If the model can't cite it, it doesn't say it. That's the rule.

I think this matters more than it sounds. Most AI answers are confident prose with no provenance. You can't check them. You can't trace them. You just have to trust the model. Recall inverts that: the answer is only as good as its citations, and the citations are right there.

Memory has scope

This is something I thought hard about. In a company, memory isn't global.

If Priya tells the system "the deploy window is 2-4 PM," that's Priya's memory. Anshu shouldn't see it as his own. So each person gets their own isolated memory graph, running as a separate graph in the same FalkorDB instance.

And memories have a lifecycle. When Priya later says "actually, make it 2-4 AM," the old fact isn't deleted. It's superseded:

The past is versioned, never rewritten. And if someone wants a fact gone entirely, there's a forget operation that issues an erasure receipt. You can prove what was forgotten and when.

The agent acts, but only with permission

Here's the flow:

The pause is the feature. The agent proposes a database migration, shows you the blast radius, and then stops. Nothing happens until you click approve. And the approval itself becomes a node in the graph, so the audit trail survives even if the backend restarts mid-flow.

I verified that last part by killing the backend mid-approval. The executor reads the approved proposal from FalkorDB, not from process memory. It picked up where it left off.

The kill-switch

This is the part I'm most serious about. The hackathon rules say FalkorDB must be central, not decorative. "If the product still works once FalkorDB is taken out, the graph is decoration."

So I built the test into the demo. Disconnect FalkorDB and the entire product falls back to honest "I don't know" answers. No graph, no answers. No silent degradation, no pretending.

Most demos tell you the database matters. Mine shows you what happens without it.

The numbers, including the misses

I ran 30 questions across five categories. Graph vs flat RAG. Same corpus, same model, same scoring.

Graph: 24.667. Flat RAG: 19.251.

But here's what I want to be honest about. The graph lost 5.333 points. Two hard zeros where the scorer demanded exact phrasing and the model paraphrased. Plus partial-credit losses scattered across categories. The README documents every one of them.

I could have tweaked the questions or dropped the zeros. I didn't. If I'm building a system whose entire pitch is "show your work," the benchmark has to show its work too.

I re-ran the eval to check reproducibility. Graph won again, 25.0 to 19.251. Same margin. The numbers hold.

What I learned

The hardest bug wasn't in the graph. It was in the test suite.

An e2e check asserted the literal string "Ledger freeze ends Friday" in the model's answer. It broke four different ways across four runs: lowercase, markdown bold, an invisible unicode narrow no-break space, and a paraphrase ("ends on Friday"). The data was correct every time. The assertion was wrong.

I rewrote it to check the structured evidence block instead of the prose. The lesson is simple:

Never assert on LLM prose. Assert on structured data.

The second lesson: scope your memory before you search it. Don't dump the entire memory store into the model and hope it figures out what's relevant. Resolve the user, resolve the scope, retrieve only what applies. Then rank.

The architecture in one sentence

If I had to explain Recall in one sentence:

Recall keeps a company's causal history in a graph and reasons over it, so every answer traces its path, every memory respects its owner, and every action waits for a human.

Final principles

If I had to reduce the whole project to a few principles:

Every claim needs a citation. If the model can't cite it, it doesn't say it.
Memory is scoped. A fact belongs to a person, not to the system.
The past is versioned, never rewritten. Supersede, don't delete. And when you do delete, issue a receipt.
The agent proposes. The human disposes. The pause before execution is the most important part of the loop.
The graph is the foundation, not a feature. Disconnect it and the product should honestly stop, not silently degrade.
Show your misses. A benchmark that hides its zeros is marketing. A benchmark that documents them is evidence.
Never assert on LLM prose. Assert on structured data.

The main idea

The way I currently think about it is:

Company knowledge is a graph. Answers are paths through it. Memory is scoped and versioned. Actions require permission. And everything, including the failures, should be visible.

That visibility is what makes it trustworthy.

By Anshu

Built for the WeMakeDevs x FalkorDB Graph Hacks hackathon. I used OpenCode as a coding assistant with Muse guiding the work. The architecture, the graph schema, and every verification step are my own. Meridian Pay is fictional; the graph, the queries, and the kill-switch are real.

Repo: https://github.com/101xAnshu/Recall
Video: https://youtu.be/F9H5nxtCiow