Most system design resources focus on interview preparation — how to design Twitter in 45 minutes, how to draw boxes and arrows convincingly. This article is different. These are the concepts I wish I had internalized earlier because they changed how I approach every technical decision, including the ones I make building TokenPulse.
The fundamental trade-off: consistency vs availability
The CAP theorem states that a distributed system can guarantee at most two of three properties: Consistency, Availability, and Partition tolerance. Since network partitions are unavoidable in real systems, the practical trade-off is between consistency and availability.
Consistency means every read receives the most recent write. If user A writes data and user B immediately reads it, B sees A's write.
Availability means every request receives a response — not necessarily the most recent data, but a response.
Most web applications choose availability over strict consistency, accepting eventual consistency — the guarantee that data will become consistent eventually, even if reads may return slightly stale data during the propagation window.
The practical example: if you update your profile picture on a social platform, some users might see the old picture for a few seconds or minutes while the change propagates. This is acceptable. If your bank account balance is wrong for even one second, that is not acceptable.
Knowing which side of this trade-off your application falls on should drive every database and caching decision you make.
Horizontal vs vertical scaling — and when each applies
Vertical scaling (scaling up) means adding more resources to a single machine — more CPU, more RAM, faster storage. It is simple, requires no architectural changes, and works until you hit the hardware limit or the cost becomes prohibitive.
Horizontal scaling (scaling out) means adding more machines and distributing load across them. It requires your application to be stateless — requests from the same user can be handled by any server.
The question developers often ask is "which should I use?" The correct answer is: vertical first, horizontal when necessary.
Vertical scaling is dramatically simpler. You resize an instance and you are done. Horizontal scaling requires load balancers, session management, distributed caching, and careful thinking about where state lives.
The mistake is assuming you need horizontal scaling before you have validated the need. A well-optimized single server can handle enormous traffic. Stack Overflow ran on five servers for years at significant scale. Optimize your code and your queries before you add machines.
When you do need horizontal scaling, the key constraint is statelessness. Your application servers must not store session state locally. Sessions go in Redis. Files go in object storage. Database writes go to a shared database. Each server is identical and interchangeable.
Caching — the most important performance tool
Caching is storing the result of an expensive computation so you can serve it quickly on subsequent requests. It is the single highest-leverage performance improvement in most systems.
There are four places you can cache:
Client-side (browser cache): HTTP cache headers (Cache-Control, ETag, Last-Modified) tell browsers how long to cache static assets. A properly configured CDN and cache policy means your CSS and JavaScript files are served from the user's local cache — zero network round-trips.
CDN cache: Content delivery networks cache responses at edge nodes geographically close to users. Dynamic content that does not change per-user — marketing pages, blog posts, public API responses — should be cached at the CDN.
Application cache (Redis/Memcached): Server-side in-memory caching for database query results, computed aggregations, and anything expensive to regenerate. Redis is the standard choice — it is fast, supports data structures, and handles expiration automatically.
Database query cache: Most databases cache query results internally. This is largely automatic, but it means repeated identical queries are fast and schema changes or cache invalidation can cause temporary performance drops.
The hardest problem in caching is cache invalidation — knowing when to expire stale data. Three strategies:
TTL (time-to-live): Data expires after a fixed duration. Simple, but data may be stale within the TTL window.
Write-through: When data is written to the database, the cache is updated simultaneously. Consistent, but every write hits both database and cache.
Cache-aside: Application checks cache first. On a miss, reads from database and populates the cache. On a write, invalidates the cache entry. The most common pattern for read-heavy workloads.
Database indexing — what it is and why it matters
Without an index, a database must scan every row in a table to find matching records. This is a full table scan, and its cost grows linearly with the number of rows.
An index is a separate data structure (typically a B-tree) that allows the database to find matching rows in O(log n) time instead of O(n) time. For a table with one million rows, this is the difference between scanning 1,000,000 rows and scanning roughly 20.
The columns to index:
- Primary keys — indexed automatically
- Foreign keys — columns used in JOIN conditions
- Columns in WHERE clauses — especially high-cardinality columns (many distinct values)
- Columns in ORDER BY — if you frequently sort by a column The trade-off: indexes make reads fast and writes slow. Every INSERT, UPDATE, or DELETE must also update every relevant index. For read-heavy applications (most web applications), this trade-off is almost always worth it. For write-heavy applications (logging, analytics ingestion), index carefully.
Composite indexes — indexes on multiple columns — matter for query performance. An index on (user_id, created_at) supports queries that filter by user_id and sort by created_at, but only if the columns appear in that order in the WHERE clause.
Message queues — decoupling for resilience
A message queue is a buffer between a producer (something that generates work) and a consumer (something that processes work). The producer puts a message in the queue and moves on. The consumer reads from the queue and processes at its own pace.
Why this matters:
Handling traffic spikes. If your API receives 10,000 requests in one second, a queue absorbs the burst and processes it steadily rather than overwhelming your database.
Decoupling services. If the email service is down, you do not want to fail the user's registration request. Put the "send welcome email" task in a queue. When the email service recovers, it processes the queue.
Retry logic. If a queue consumer fails to process a message, the message stays in the queue (or moves to a dead letter queue) and is retried. Without a queue, failed tasks are lost.
The practical example from TokenPulse: when a user joins the Pro waitlist, the API endpoint puts two tasks in a queue — "send notification email" and "write to Google Sheets." If Google Sheets is temporarily unavailable, the task is retried automatically. The user's HTTP response does not wait for either task to complete.
Load balancing strategies
A load balancer distributes incoming requests across multiple servers. The distribution strategy matters:
Round robin: Requests go to servers in order — 1, 2, 3, 1, 2, 3. Simple, works when servers are identical and requests take similar time.
Least connections: Requests go to the server with the fewest active connections. Better when requests have variable processing time.
IP hash: The same client IP always routes to the same server. Useful when you cannot make your application fully stateless and need session affinity.
Weighted: Some servers receive more traffic than others — useful when servers have different capacities.
For most Next.js applications on Vercel, load balancing is handled automatically. Understanding the strategies matters when you operate your own infrastructure or need to reason about why requests are distributed the way they are.
The read replica pattern
Write operations go to a primary database. Read operations go to one or more replica databases that are kept in sync with the primary.
This pattern solves a specific problem: most web applications have dramatically more reads than writes. A read-heavy query that scans a large table on the primary database competes with write operations for I/O. Moving reads to a replica eliminates this contention.
The trade-off is replication lag — replicas may be slightly behind the primary. For most reads (show me recent posts, display the product catalog), this is acceptable. For reads that immediately follow a write (show me the thing I just created), you need to either read from the primary or implement a mechanism to wait for replication.
How to think about trade-offs like a senior engineer
Every system design decision is a trade-off. Senior engineers do not make universally correct decisions — they make decisions appropriate to their specific constraints.
The questions to ask for any architectural decision:
What is the actual scale? Do not design for 10 million users if you have 100. Design for 10x your current scale and revisit when you get there.
What is the cost of being wrong? If a caching strategy is wrong, you serve stale data. If a database schema is wrong, you have a migration. Different stakes require different confidence levels before deciding.
What can you change later? Some decisions are easily reversible — switching a caching library, changing an index. Others are difficult — changing your data model, switching databases. Spend more time on irreversible decisions.
What do you actually know? Optimize for known bottlenecks, not hypothetical ones. Use monitoring and profiling to find real problems before solving imagined ones.
If you are building AI-powered applications and want visibility into your token usage costs and rate limits, TokenPulse is a free Chrome extension that tracks usage across Claude, ChatGPT, Gemini, DeepSeek and Grok — no API key required.
Top comments (0)