Most developers learn system design in the wrong way, especially with the help of AI.
They usually study concepts such as load balancing, the CAP theorem, Kafka and Saga.
Something AI can explain.
But knowing the components is not the same as designing a system.
Every real system is the result of a handful of decisions and trade-offs, and each one costs you something.
For example:
Pick strong consistency, and you pay in latency.
Pick a queue, and you pay in debugging across multiple services.
Pick microservices, and you pay in premium operations.
Over the past years, I have built and scaled .NET systems that made every one of these choices, sometimes badly.
This is the short version of what I wish someone had handed me at the start.
In this post, we will explore the 7 decisions every system has to make:
How fresh does this data have to be?
Where should this data live?
Do you buy a bigger machine or more machines?
What do you cache, and where?
Should this be a call or an event?
What happens when more work arrives than you can handle?
How many things do you deploy?
Let's dive in.
How Fresh Does This Data Have To Be?
Most teams answer this once, for the whole system, and then live with the answer for years.
That is the mistake. Freshness is a per-data-type decision.
A payment balance and a product review count do not need the same guarantee.
Treating them the same means you either overpay for the reviews or underprotect the money.
These are the models you are actually choosing between:
| Model | What you are promised | Typical use |
|---|---|---|
| Strong (linearizable) | Every read sees the latest committed write, from any node. | Bank balances, stock counts, leader election. |
| Read-after-write | A user always sees their own most recent write. | Your own post appearing in your own feed. |
| Monotonic reads | Once you see a value, you never see an older one again. | News feeds, where going backward looks broken. |
| Causal | If A caused B, every reader sees A before B. | Comment threads, chat messages. |
| Eventual | All replicas converge given enough time and no new writes. | Analytics counters, review totals. |
Now the part most articles get wrong.
The CAP theorem says a distributed system can only guarantee two of Consistency, Availability, and Partition tolerance.
That "pick two of three" framing is the part Eric Brewer himself walked back in 2012.
Partition tolerance is not optional because networks fail, and a system that does not tolerate them just falls over.
So the real choice is binary, and only active during a partition.
Either you keep accepting writes and risk divergence (AP), or you refuse writes to protect correctness (CP).
CA databases do not exist in any meaningful sense.
Network partition happens
Can the system
safely guess?
Pick CP
Refuse the request
Pick AP
Answer anyway
Bank ledger
Booking seats
Leader election
Stock counts
Social feed
Shopping cart
DNS
CDN cache
A more honest everyday framing is PACELC:
Partition: choose Availability or Consistency.
Else: choose Latency or Consistency.
When the network is healthy, and it usually is, you trade consistency against latency on every write.
Synchronous quorum costs roughly 1-5 ms per write inside a region and 50-200 ms across regions, and a US East to Europe round trip is 80-100 ms before your code does anything.
CAP only shows up when something is on fire. PACELC is the tax you pay every day.
When you do have to pick a side for a specific piece of data, answer these four questions:
What is the cost of stale data? An old balance leads to a bad decision. An old like count leads to nothing.
What is the cost of downtime? If 30 seconds of unavailability costs thousands in lost sales, lean toward AP.
Can you reconcile later? Merging two shopping carts is easy. Merging two conflicting bank transfers is not.
How long do partitions last? Short ones favour CP. Long multi-region ones favor AP.
Most production systems use both CP for payments, orders, and auth tokens and AP for the catalog, recommendations, and activity tracking.
Once you know how fresh each piece of data must be, the list of databases that can hold it gets much shorter.
👉 Read the full article on my newsletter: https://antondevtips.com/blog/getting-started-with-system-design
Top comments (0)