DEV Community: tejaswipandava

Designing a URL Shortener (like bit.ly)

tejaswipandava — Tue, 20 Oct 2026 07:25:00 +0000

Framework: Problem → Constraint → Decision → Alternative → Trade-off → Failure Mode

This is the "hello world" of system design interviews — but it's deceptively deep. The interesting parts aren't the obvious pieces (a redirect service), they're the second-order decisions: how you generate the short code, how you make reads fast at absurd scale, and how you avoid the two classic failure modes (hot keys and analytics write amplification).

1. Problem

Build a service that takes a long URL and returns a short one (e.g. https://bit.ly/3xK9pQ), such that visiting the short URL redirects the user to the original long URL.

Functional requirements:

Shorten a long URL → return a unique short code.
Redirect: given a short code, look up and 302/301-redirect to the original URL.
(Optional) Custom aliases, expiration dates, click analytics.

Non-functional requirements / scale assumptions:

Read-heavy: redirects vastly outnumber creations. A realistic ratio is 100:1 to 1000:1 reads to writes (people click links far more often than they shorten new ones).
Back-of-envelope: 100M new URLs/day → ~1,150 writes/sec average. At a 100:1 read:write ratio, that's ~115K reads/sec average — spiky well above that for viral links.
Redirect latency must be very low (~tens of ms) — a URL shortener sitting in the critical path of every click on a shared link is effectively invisible infrastructure; users notice it only when it's slow.
High availability for redirects (a link going viral is the worst possible time for the redirect service to be down) — reads should stay up even if the write path degrades.
Short codes should be unguessable-ish (not sequential enough to enumerate every URL in the system) but don't need cryptographic security for the base case.

2. Constraint

The constraints that shape every decision below:

The short code must be short (6-8 chars) but the keyspace must be enormous — 100M URLs/day × several years of retention means billions of entries. Base62 (a-z, A-Z, 0-9) over 7 characters gives 62^7 ≈ 3.5 trillion combinations — comfortably enough, with room for a non-trivial "already taken" collision rate.
Redirects must be fast and horizontally scalable — this is a lookup-by-key problem at massive read volume, which points immediately toward a key-value store + aggressive caching, not a relational join-heavy schema.
No two writers can generate the same code at the same time — code generation must be coordination-free (or cheaply coordinated) because writes happen from many stateless app servers concurrently.
Click analytics (if required) must not slow down or endanger the redirect path — the redirect is the SLA-critical operation; analytics is best-effort by comparison.

3. Decision

3.1 Short code generation: Counter + Base62 encoding, sharded

The core decision: assign every new URL a unique, monotonically-increasing integer ID (from a distributed counter / range-allocated ID block), then encode that integer in Base62 to produce the short code.

A central ID-generation service (or a Snowflake-style ID scheme, or pre-allocated ID ranges handed out to each app server so they can mint IDs locally without a round trip per request) produces the integer.
Base62-encode the integer → e.g. 125 → "cb". This is a pure, stateless, deterministic function — no coordination needed at encode time, only at ID-allocation time.
Store the mapping short_code → long_url in a key-value store (DynamoDB, Cassandra, or even a sharded relational table keyed by short_code).

Why this over random-string-and-check? Generating a random 7-char string and checking for collision before insert requires a read-then-write per creation — under high write volume this is wasted round trips and a race window between the check and the insert unless you use a conditional write. A pre-allocated, monotonic counter sidesteps collisions entirely: every ID is unique by construction, no check required.

3.2 Storage: Key-value store, sharded by short_code hash

Sharding by a hash of short_code (not by creation time / sequential ID) spreads both reads and writes evenly across shards. This matters because...

3.3 Redirect path: Cache-first (Redis) in front of the KV store

Given the 100:1+ read:write ratio, an LRU cache in front of the database captures the vast majority of redirect traffic — especially because real-world link popularity follows a Zipfian/power-law distribution (a small number of links get a huge fraction of all clicks). Cache hit → redirect in single-digit ms without touching the database at all.

3.4 Analytics: Asynchronous, off the critical path

Click events (timestamp, referrer, geo, device) are fired onto a message queue (Kafka/Kinesis) from the redirect handler after the redirect response has already been sent — never inline/blocking. A separate consumer aggregates these into an analytics store (e.g., a columnar/OLAP store or pre-aggregated counters in Redis) on its own schedule.

4. Alternative (and why each was rejected as the primary approach)

Alternative	Why considered	Why rejected as primary
Random string + collision check	Simpler mental model, no central ID allocator needed	Requires a read-check-write per creation; collision probability rises as the keyspace fills, causing more retries over time (non-constant write latency)
Hash the long URL (e.g., MD5/SHA, truncated)	Deterministic — same URL always maps to the same short code, which is a nice property (dedup for free)	Truncated hashes collide far more often than a monotonic counter at the same length (birthday paradox), and you still need a collision-check-and-retry loop; also two different long URLs can legitimately want different short codes (can't always dedupe)
Fully random UUID, base62-encoded	No coordination at all, embarrassingly parallel	UUIDs are 128 bits — encoding one in Base62 still produces a code far longer than 7 chars unless truncated, which reintroduces collisions; defeats the "short" requirement
Sequential integer ID, exposed directly (no Base62)	Trivial to implement	Directly reveals total URL count and makes the entire link space enumerable/guessable — every 4th short URL exists and is trivially walkable by an attacker/scraper
Synchronous analytics write on every redirect	Simple, strongly consistent click counts	Couples the SLA-critical redirect path to the availability/latency of the analytics store; a slow analytics DB now slows down every single click in the product
Single relational DB, no cache	Simplest possible architecture, easy joins for custom features	Cannot survive a viral link — a single link suddenly taking 50K reads/sec will saturate a single DB's connection pool and starve every other link's redirects too (a self-inflicted Hot Partition, see below)

5. Trade-off

Every decision above is a trade-off, not a free win:

Counter + Base62 vs. random string: gains collision-free generation and predictable write latency, but loses the "any node can mint an ID with zero coordination" property — you now depend on an ID-allocation service (or pre-fetched ID ranges) being available. This is mitigated by handing out ID ranges (e.g., blocks of 10,000 IDs) to each app server in advance, so 99.99% of ID mints are purely local and the allocator is only hit occasionally.
Cache-first redirects: gains massive read throughput and low latency, but loses strong consistency — if a URL's target is updated (rare, but supported by some shorteners) or a link is deleted/deactivated, cached readers may serve the stale mapping until TTL expiry or explicit cache invalidation. Acceptable trade-off because URL→target mappings are near-immutable in practice.
Async analytics: gains a redirect path that never blocks on analytics infra, but loses real-time click counts — the count shown on a dashboard lags real clicks by however long the queue+consumer take to catch up (typically seconds, not milliseconds). Acceptable because nobody needs a live click counter with sub-second accuracy.
Sharding by short_code hash: gains even load distribution, but loses the ability to cheaply range-scan "all URLs created in the last hour" (a query a naive sequential-ID layout would support trivially) — if that query is needed, it has to be served from a separate index/analytics store, not the primary KV store.

6. Failure Mode

This is where the URL shortener problem gets interesting — the "trivial" system has real failure modes that mirror classic distributed-systems patterns:

6.1 Hot Partition / Hot Key — the viral link problem

A single short URL going viral (posted on the front page of a major site, shared in a mass text) concentrates all of its read traffic onto one shard/partition and, if uncached, one DB row. This is the textbook Hot Partition failure mode: one key gets so much traffic that its home shard degrades for every other key sharing that shard too — a single viral link can take down redirects for unrelated URLs that happen to co-locate on the same partition.

Mitigation: cache-first design (Section 3.3) absorbs the vast majority of hot-key traffic before it ever reaches the sharded store. For truly extreme spikes, a second layer — a CDN-edge cache or even a static-redirect rule pushed to the CDN — takes it further upstream than application-layer Redis.

6.2 ID-allocator becoming a single point of failure

If ID generation is centralized (one counter service) and it goes down, all writes (new shortens) fail even though reads (redirects) are completely unaffected by this outage. This is a good interview point: correctly identify that this failure is write-path-only and does not threaten the SLA-critical redirect path, then propose the pre-allocated ID-range mitigation from Section 5 so that even a brief allocator outage doesn't stall writes (app servers keep minting from their local range until it's exhausted).

6.3 Cache stampede on a cold, suddenly-popular key

A link goes viral at the exact moment its cache entry expires (or it was never cached because it's brand new but shared immediately) — thousands of concurrent redirect requests all miss the cache simultaneously and all hammer the database for the same key at once. This is the same cache stampede pattern documented in article: Caching Strategies: 8 Patterns Every Engineer Should Know, and the same fix applies: request coalescing (in-flight de-duplication) so only one request per key actually reaches the database while the rest wait on that result.

6.4 Analytics write amplification

If click-event volume isn't decoupled from the redirect path (Decision 3.4), a viral link doesn't just amplify read load — it amplifies write load on whatever store holds analytics, at the same multiplier. Without the async-queue buffer, this write spike can back up and, in the worst case, apply backpressure all the way to the redirect handler itself, turning an analytics slowdown into a user-facing outage — the exact Cascading Failure pattern from 10 Must-Know System Design Failure Modes.

6.5 Short-code exhaustion / collision creep (long-tail failure)

Not an acute outage, but a slow-burn failure mode: if the ID-range or code length was under-provisioned relative to actual growth, the system approaches keyspace exhaustion years into production, forcing a length migration (7→8 chars) that must be backward-compatible with every already-issued short code still in circulation. Worth calling out in an interview as the kind of failure mode that doesn't page anyone at 3am but will show up in a capacity-planning review — the fix is over-provisioning the code length up front (each added Base62 character multiplies the keyspace by 62×, so the cost of provisioning generously is nearly free).

Key Takeaways

The "trivial" URL shortener interview question is really a read-scaling + hot-key problem wearing a simple disguise — the moment you introduce "what if a link goes viral," every classic caching/sharding failure mode becomes relevant.
Counter + Base62 beats random-string-and-check specifically because it makes collisions structurally impossible rather than statistically rare — a good general principle: prefer designs that eliminate a failure mode by construction over designs that merely reduce its probability.
Decoupling the SLA-critical path (redirect) from the best-effort path (analytics) via an async queue is the same pattern that shows up in payment webhooks, notification delivery, and demand-forecasting pipelines elsewhere in this vault — it's a general recipe, not a URL-shortener-specific trick.

Stop Rambling in System Design Interviews — 3 Frameworks That Fix It

tejaswipandava — Mon, 19 Oct 2026 03:06:00 +0000

The RADIO Framework: How to Structure Any System Design Interview Answer

Writing good code alone doesn't clear Senior SDE / IC-level system design rounds anymore. What separates strong candidates from everyone else isn't raw knowledge of architectures — it's the ability to think out loud in a structured way under pressure.

A few hard truths about system design rounds: you can't walk in and improvise, hoping structure will emerge on its own. You can't rely on your resume or past projects to "speak for itself" in the room — the interviewer only sees what you say in the next 45 minutes. And you can't skip structure and expect the interviewer to connect the dots for you; it's your job to make the narrative obvious.

Below are three complementary frameworks I use to stay organized in these interviews: RADIO as the top-level skeleton, a layered checklist for the gritty details inside each RADIO step, and a 4-step scoping approach for nailing the critical opening minutes.

RADIO: A 5-Step Framework for System Design Answers

RADIO is a simple mental checklist to organize any system design answer, so you stay clear and methodical instead of rambling, without ever sounding robotic or scripted.

R — Requirements

Start by scoping the problem before designing anything. What must the system do — the functional requirements? What must it not break — the non-functional requirements like availability, latency, and consistency? And what are the rough scale estimates — users, QPS, data volume — that set the boundaries for everything that follows?

A — Architecture

Zoom out to the big picture before diving into details. Identify the major components and how they connect. Decide the client/server communication style — REST, gRPC, WebSockets, message queues. Most importantly, call out key architectural decisions and why you made them — this is where you demonstrate judgment, not just knowledge.

D — Data Model

Think about how data is shaped, stored, and accessed. What entities or tables exist, and how are they related? What needs to be indexed for fast lookups? What needs to be cached, and what remains the source of truth?

I — Interfaces (APIs)

Define the contracts between components. Keep the API surface minimal and purposeful — every endpoint should map back to a real requirement from step one. Specify request/response shapes only for the core operations, and skip anything that isn't essential to the problem at hand.

O — Optimizations

Don't save this for the last two minutes of the interview — weave it in as you go. Proactively call out what could go wrong: hotspots, single points of failure, slow queries. Discuss how you'd handle scale, load spikes, and latency using techniques like caching, sharding, async processing, and rate limiting.

Why This Works

RADIO isn't a trick or a shortcut — it's a way to stay organized and confident under pressure. It gives you a repeatable structure so you always know "what to say next," which frees up mental bandwidth to actually reason about the problem instead of worrying about how to present it. Once internalized, it shifts you from reactively answering questions to proactively leading the conversation — which is exactly the signal interviewers look for at the Senior SDE / IC4+ level.

Framework #2: The Layered System Design Checklist

Most engineers don't fail system design interviews because they lack knowledge — they fail because they panic midway and lose structure. A strong framework beats random brilliance. Think of this as a complementary, more granular checklist to RADIO above: it's what to actually say inside the "Architecture" and "Optimizations" steps.

1. Start With the Client Journey

Who are the users? What requests are coming in, and at what scale? Trace a single request end-to-end before designing any components — this grounds every later decision in a real flow instead of a set of abstract boxes.

2. Define the Entry Layer

Load balancers, API gateways, authentication, rate limiting, caching. Most candidates jump directly to databases — strong candidates first control traffic, deciding how requests are authenticated, throttled, and routed before deciding how data is stored.

3. Separate Control Plane vs. Data Plane

Metadata and coordination services (the control plane) should not behave the same way as bulk storage and transfer services (the data plane) — they have different consistency, latency, and scaling needs. Calling out this distinction explicitly is a strong signal of architectural maturity.

4. Think Asynchronously

Use queues everywhere it makes sense: notifications, feed generation, video processing, search indexing. If everything is synchronous, the system collapses under scale — you need to decouple producer load from consumer processing rate.

5. Design for Scale From Day One

Partitioning, replication, caching, CDNs, sharding. Interviewers care about why you chose a technique, not just that you can name it — always follow up each term with the specific bottleneck it solves.

6. Observability Matters

Distributed tracing, logging, monitoring, metrics — most candidates forget this entirely. Production systems fail silently without visibility, so call out what you'd monitor and alert on, not just what you'd build.

7. Finally, Discuss Trade-offs

Consistency vs. availability, latency vs. cost, SQL vs. NoSQL, push vs. pull architecture. This is where the real interview begins. The best architects aren't the ones who know the most tools — they're the ones who think in layers, constraints, bottlenecks, and trade-offs.

How the Two Frameworks Fit Together

RADIO Step	Layered Checklist Items
Requirements	Client journey (users, requests, scale)
Architecture	Entry layer, control vs. data plane, async/queues
Data Model	Partitioning, replication, sharding
Interfaces	(API contracts — same as RADIO)
Optimizations	Caching/CDNs, observability, trade-off discussion

Use RADIO as the high-level narrative order you speak in, and this layered checklist as the detailed prompts to make sure you don't forget a dimension — entry layer, control/data plane split, async, observability, trade-offs — while inside each RADIO step.

Framework #3: The 4-Step Scoping-First Approach

Most engineers fail system design interviews not because they lack knowledge, but because they lack structure. You don't need to memorize a hundred architectures — you need a structured, repeatable approach. This framework focuses specifically on how to open a system design interview correctly. It maps most closely onto RADIO's R (Requirements) and the start of A (Architecture), with much more concrete detail on how to scope and estimate.

Step 1 — Clarify the Problem Before Writing Anything

System design interviews aren't about throwing out buzzwords — they're about trade-offs. Most candidates assume things and jump straight into solutions, so make sure you actually know what you're solving first. Nail down the scoping: are we designing just one feature, or the entire system — "just the video upload pipeline" versus "all of YouTube"? Clarify the constraints upfront: how many users, what's the read/write ratio, what are the latency requirements? And agree on the success criteria: what matters most here — scalability, cost, or low latency? These often trade off against each other, so naming the priority early shapes every later decision.

Step 2 — Define Functional and Non-Functional Requirements Clearly

Nail down the functional requirements — what features does the system actually need to support? Then the non-functional requirements — what are the performance expectations around availability, latency, consistency, and durability? Most importantly, identify the biggest technical challenge up front, since this single question should guide the rest of the discussion.

For example, if you're designing YouTube, is the focus on video uploads, recommendations, or live streaming? Each of those has a completely different set of constraints — uploads mean large binary transfer plus transcoding, recommendations mean an ML/data pipeline, live streaming means ultra-low-latency real-time delivery — so the interviewer's emphasis should steer where you spend your time.

Step 3 — Estimate the Scale and Plan Capacity Like an Engineer

Most candidates throw out random numbers. Instead, do quick back-of-the-envelope calculations to validate assumptions. Derive users and requests per second from the stated daily/monthly active users and typical usage patterns, not guesses. Work out storage needs with concrete math — for example, "10MB per user × 100M users = 1PB" — not hand-waving. And figure out throughput: can a single database handle this load, or does it clearly need sharding or read replicas from the outset? The estimate should directly justify later design decisions, e.g., "at this write volume, a single primary DB would bottleneck, so we shard by user ID."

Step 4 — Break the System Into Core Components (High-Level Design)

Define the major building blocks: API gateway, load balancer, service layers, databases. Don't overcomplicate things — simple and scalable always wins over an impressively complex diagram that doesn't map to the actual requirements gathered in steps one through three. Clearly define the interactions and data flow between services.

For a messaging app, this might break down into a User Service for auth and profiles, a Messaging Service for storing and retrieving chats, a Notification Service for real-time updates, and Media Storage for images and videos. Each component has different constraints — messaging needs low-latency writes, media storage needs cheap bulk blob storage, notifications need a push/fan-out mechanism — so build around what's most important for this system's stated priorities, not a generic template.

How This Maps Onto RADIO

RADIO Step	This Framework's Steps
Requirements	Step 1 (Scoping) + Step 2 (Functional/Non-functional)
Architecture	Step 3 (Capacity estimation justifies architecture) + Step 4 (Core components)
Data Model / Interfaces / Optimizations	Covered by RADIO + the Layered Checklist as the design deepens

Takeaway: use this framework specifically to nail the opening ten to fifteen minutes of any system design interview — scope tightly, quantify the scale with real numbers, then break into components — before moving into the deeper RADIO and layered-checklist steps for data modeling, APIs, and optimizations.

Putting It All Together

None of these three frameworks replace actual system design knowledge — you still need to understand caching, sharding, queues, and consistency models. What they give you is a scaffold to hang that knowledge on, so under interview pressure you're navigating a familiar checklist instead of improvising from scratch. Start every interview with the scoping-first approach, narrate your answer using RADIO, and lean on the layered checklist whenever you need to go one level deeper on architecture or optimizations.

If you've got your own mental framework for system design interviews, I'd love to hear it in the comments.

Designing APIs for Massive Datasets (Cursor Pagination, Sparse Fields, Caching)

tejaswipandava — Sun, 18 Oct 2026 04:28:00 +0000

The scenario: Your API returns a list of 2 million products. A mobile app needs to display them. How do you design this so the app doesn't crash and the server doesn't melt?

The naive answer — return everything in one response — fails immediately at scale: 2 million records at even a modest ~400 bytes each is ~800MB of JSON, which will crash a memory-constrained mobile client and saturate a slow connection long before it renders anything. The real answer is a stack of five complementary techniques, each solving a different bottleneck.

1. Cursor-Based Pagination (not offset-based)

The problem with offset pagination

SELECT * FROM products ORDER BY id LIMIT 20 OFFSET 999980;

This is what ?page=50000&size=20 compiles down to. The database still has to scan and discard the first 999,980 matching rows before it can return the 20 you actually asked for — even though an index exists on id/ORDER BY column, OFFSET forces the DB to walk that many index entries and skip them. The deeper the page, the slower the query — page 1 is instant, page 50,000 can take seconds, and the cost grows roughly linearly with offset depth.

The fix: cursor (a.k.a. keyset) pagination

SELECT * FROM products WHERE id > :last_seen_id ORDER BY id LIMIT 20;

Instead of asking for "page 50,000," the client says "give me 20 records after this specific ID (or timestamp, or composite key)." The database uses the index to seek directly to :last_seen_id and reads forward — O(log n) seek + O(page size) read, regardless of how deep into the dataset you are. Page 1 and page 100,000 cost the same.

API shape:

GET /products?limit=20&cursor=eyJpZCI6MTAwMjB9   // base64-encoded cursor, e.g. {"id": 10020}

Response:
{
  "items": [...],
  "next_cursor": "eyJpZCI6MTAwNDB9"
}

Trade-offs worth stating explicitly:

❌ Cursor pagination can't jump to an arbitrary page number ("go to page 500") — only forward/backward from a known cursor. If a UI truly needs numbered page jumps (rare for infinite scroll, common for admin tables), offset pagination (or a hybrid) may still be needed for that specific view.
✅ Cursors should encode a unique, stable sort key (or tuple, e.g., (created_at, id) to break ties) — using a non-unique column alone can cause skipped or duplicated rows if two rows share the same value.
✅ Works cleanly with infinite-scroll UIs (the dominant mobile pattern), which never need to jump to an arbitrary page anyway.

2. Sparse Field Selection (Response Shaping)

A product with 30 fields (full description, specs, reviews, inventory count, etc.) might be ~4KB. The list screen only renders name, price, and thumbnail URL — maybe 200 bytes. Returning full objects for a list view is pure waste: 20x more data transferred than what's rendered.

Return a lean DTO/projection for list endpoints — not the same serialization used for the detail endpoint.
Detail fields (description, specs, reviews) load on-demand when the user taps into a specific item (GET /products/{id} — a separate, richer response).
Optionally support a fields= query param (GraphQL-style field selection, or a simple comma-separated whitelist) so different client screens can request exactly what they need without needing a new endpoint per view.

Rule of thumb: design list/detail as two different response shapes from day one — don't let the detail DTO become the "default" that list endpoints lazily reuse.

3. Database-Level Filtering (push filters down, never up)

If a user searches "bluetooth speaker" against 2 million products:

❌ Worst: fetch all 2M rows, filter in the mobile app.
❌ Bad: fetch all 2M rows, filter in the application/server layer.
✅ Correct: push the filter into the database query (WHERE name ILIKE '%bluetooth%' backed by a proper index, or a dedicated search engine like Elasticsearch for full-text search at this scale) so the database does the heavy lifting using indexes, and the server only ever receives and forwards the ~340 matching rows.

Why this matters beyond just this example: any filtering logic that runs after data leaves the database has already paid the full cost of reading/transferring the unfiltered dataset — the filtering step itself becomes irrelevant to performance at that point. The entire goal is to narrow the result set as early as possible in the pipeline.

4. Aggressive Caching of Early Pages

Real usage data almost always follows a power law: ~80% of users never scroll past page 3. This makes the first few pages of the most common queries (default sort, popular filters/search terms) extremely cache-friendly.

Cache the first N pages (e.g., first 5) of common queries in Redis, keyed by (query_params, cursor).
Most user sessions are served entirely from cache and never touch the database.
The database only handles cache misses and deep-pagination requests — a small, predictable fraction of total traffic.
(See Caching Strategies.md for cache-aside/TTL/invalidation patterns to apply here — this is a textbook cache-aside candidate with a short TTL, since product catalogs change but not on every request.)

5. Response Compression

A trivial but high-leverage win: enable gzip (or brotli) compression on API responses.

A ~50KB JSON payload commonly compresses down to ~8KB over the wire — JSON's repetitive key names and text-heavy structure compress extremely well.
On slow/high-latency mobile networks (3G, poor signal), this is the difference between a multi-second load and a near-instant one — compression reduces transfer time, which dominates total latency on constrained networks far more than server-side processing time does.
Usually a single configuration line (e.g., gzip on; in Nginx, or a compression middleware in the app framework) — no application code changes required.

The Full Architecture, Stacked

Client (mobile app)
   │  GET /products?limit=20&cursor=... &q=bluetooth+speaker
   ▼
API Gateway / Load Balancer  ── gzip compression on response ──►
   ▼
App Server
   │  1. Parses cursor + filter params
   │  2. Checks Redis cache for (query, cursor) — HIT for ~80% of requests (early pages)
   ▼
   on cache MISS:
   │  3. Pushes filter + keyset condition to DB:
   │       SELECT id, name, price, thumbnail_url        ← sparse fields
   │       FROM products
   │       WHERE name @@ 'bluetooth speaker'             ← DB-level filter (indexed / search engine)
   │         AND id > :cursor
   │       ORDER BY id LIMIT 20                          ← cursor pagination, index seek
   │  4. Writes result back into Redis (short TTL)
   ▼
Response: { items: [ {id, name, price, thumbnail_url}, ... ], next_cursor: "..." }

Each layer removes an order of magnitude of unnecessary work from the layer above it — by the time data reaches the client, it's a tiny, precisely-shaped, compressed payload instead of the naive 800MB firehose.

Quick Comparison Table

Technique	Solves	Cost if skipped
Cursor pagination	Deep-page query slowness	O(offset) scan cost that grows with dataset size
Sparse fields	Oversized payloads per item	~20x more bytes transferred than needed
DB-level filtering	Wasted compute/transfer on irrelevant rows	Full table scan/transfer before any filtering happens
Caching early pages	Redundant DB load for the most common requests	Every request (even identical ones) hits the DB
Compression	Wire transfer time on slow networks	5-6x larger payload size over the wire

Meta-Lesson

None of these five techniques are individually complex — each is a well-known, "simple" optimization. The real insight isn't knowing any single technique; it's recognizing that they compose, and each targets a different bottleneck in the pipeline (query cost, payload size, filtering cost, repeated-request cost, transfer cost). A shallow answer stops at "add pagination." A deeper answer explains why offset pagination specifically breaks down at scale, and layers the remaining four techniques on top with a clear rationale for each.

The 1-2-3 Method: Preparing for SDE-2/3 Roles Alongside a 9-5

tejaswipandava — Sun, 18 Oct 2026 04:18:00 +0000

Core Insight

You don't need extra time — you need a better system. Long hours, deadlines, and meetings don't have to block prep if the structure is right.

1️⃣ One Pattern at a Time

Instead of scattered, random practice, focus on 1 DSA pattern per week.
Pair it with 1 System Design case study every weekend.
Depth > breadth: mastering a pattern (e.g., sliding window, two pointers, topological sort) generalizes to dozens of problems, instead of grinding random questions.

Weekly cadence example:
| Day | Focus |
|-----|-------|
| Mon-Fri | 1 DSA pattern, ~2 hrs/day (theory + 3-5 problems) |
| Sat | System design case study (read/design + notes) |
| Sun | Mock interview + review weak spots |

2️⃣ Two Hours of Consistency

2 focused hours daily beat 6 distracted hours crammed into a Sunday.
Small, steady progress compounds — same principle as spaced repetition and incremental system building.
Protect this block like a meeting: no phone, single task, timer-boxed.

Tips to protect the 2 hours:

Block it on your calendar like a work meeting.
Do it early morning or right after work before energy drops.
Track streaks (not just problems solved) to stay motivated.

3️⃣ Three Mock Interviews for Better Communication

Sit in mock interviews every weekend — coding + system design + behavioral.
Advanced SDE interviews (SDE-2/3) test trade-offs and clarity of thought, not just correctness.
Behavioral rounds matter as much as coding speed — practice articulating decisions out loud.

What to rotate across mocks:

DSA mock — focus on communicating approach before coding.
System design mock — focus on trade-off discussions (consistency vs availability, SQL vs NoSQL, etc.).
Behavioral mock — STAR-format stories, leadership/conflict scenarios.

Why This Works

Removes the "no time" excuse by fitting prep into a realistic daily/weekly rhythm instead of requiring large uninterrupted blocks.
Builds compounding skill (patterns) + compounding habit (consistency) + compounding confidence (mock reps).
Balances technical depth (DSA/System Design) with communication (mocks), which is what actually gets evaluated at SDE-2/3 bar-raiser rounds.

Personal Action Plan (fill in)

[ ] Pick this week's DSA pattern: ___________
[ ] Pick this weekend's system design case: ___________
[ ] Schedule daily 2-hour block: ___________ (time of day)
[ ] Schedule 3 mocks for the weekend: DSA / System Design / Behavioral

Caching Strategies: 8 Patterns Every Engineer Should Know

tejaswipandava — Sat, 17 Oct 2026 04:27:00 +0000

The fastest way to produce a shallow system design: saying "I'll add Redis for caching" and stopping there. Naming "Redis" without explaining what you're caching and why signals you're naming a tool, not solving a problem.

The real question isn't "would you use Redis?" — it's "which caching pattern fits this access pattern?" Choosing the wrong strategy leads to stale data, cache stampedes, consistency issues, and poor performance, regardless of which underlying technology you picked.

The Right Way to Think About Caching

Never say "for caching." Say something with an actual access pattern attached:

"User session data, accessed 10K/sec, TTL 30 minutes, cache-aside pattern."
"Hot product listings — 20% of items get 80% of reads, LRU eviction."
"Computed aggregates, refreshed every 5 minutes, write-behind to reduce DB load."

Each of these names: what's being cached, how hot it is, how long it should live, and which read/write pattern fits it. That's the difference between reciting a buzzword and demonstrating real design judgment.

The rule: access pattern first, technology second — always.

The 8 Patterns

1️⃣ Cache-Aside (Lazy Loading)

The application (not the cache) is responsible for the loading logic — the app makes two separate calls, one to the cache and (on a miss) a second one to the DB, and the app itself writes the result back into the cache.

App checks cache first
    ├─ HIT  → return directly
    └─ MISS → App queries DB directly → App writes result into cache → return

// Cache-aside, Java — the branching ("check, then fetch, then populate") lives in the app
User user = cache.getIfPresent(userId);
if (user == null) {                    // MISS — app itself talks to the DB
    user = userRepository.findById(userId);
    cache.put(userId, user);            // app itself populates the cache
}
return user;

Best for: read-heavy workloads, frequently accessed data, and applications that want explicit control over caching logic (what gets cached, when, and how it's keyed).

Trade-off: stale data is possible between writes — if the DB row changes but the cached copy isn't invalidated, reads serve outdated data until TTL expiry or explicit invalidation. This app-level branching logic also has to be duplicated at every call site that reads this data, unless it's wrapped in a shared helper.

2️⃣ Read-Through

Similar outcome to cache-aside, but the loading logic lives inside the cache layer itself rather than the application. Critically, the app makes only one outbound call, ever — it never bounces back to the DB itself. On a miss, the cache's own configured loader (registered once, at setup) fetches from the DB, stores the result, and returns it, all inside that single call.

App → Cache (ONE call, period)
         Cache internally decides:
           ├─ HIT  → return from cache
           └─ MISS → cache's own loader fetches from DB → cache stores it → cache returns it
App never talks to the DB directly, in either the hit or miss case.

// Read-through, Java (Caffeine) — the DB-fetch logic is registered ONCE, not per call-site
LoadingCache<String, User> cache = Caffeine.newBuilder()
    .maximumSize(10_000)
    .build(key -> userRepository.findById(key)); // "loader" — lives inside the cache, not the app

// Every call site, hit or miss, looks identical — no branching logic in app code at all:
User user = cache.get(userId);

Benefits: simpler application code (no manual "check cache, then DB, then populate" boilerplate repeated everywhere), centralized caching logic, easier maintenance since the fetch-on-miss behavior lives in one place (the loader) instead of being duplicated across every call site.

Where does Bloom filter / Cache Penetration protection go here? Since the app only ever makes one call, a Bloom filter check has to sit inside that single call's miss path, so it can block the loader from ever reaching the DB for an invalid key — either as a guard right before cache.get() in app code, or (cleaner) pushed one layer deeper, directly inside the loader function itself:

LoadingCache<String, User> cache = Caffeine.newBuilder()
    .build(key -> {
        if (!bloomFilter.mightContain(key)) return null; // definitely invalid — skip DB entirely
        return userRepository.findById(key);
    });

Cache-Aside vs. Read-Through, in one line: in cache-aside the app owns the "check → miss → fetch → populate" sequence (two calls out); in read-through the cache owns that entire sequence behind a single call, with the DB-fetch logic registered once as a loader instead of repeated per call-site.

3️⃣ Write-Through

Every write goes to the cache first, then synchronously to the database, before the write is considered complete.

Benefits: strong cache consistency — the cache always contains the latest data, since a write never returns success without also having landed in the DB.

Trade-off: higher write latency, since every write pays the cost of two synchronous operations instead of one.

4️⃣ Write-Back (Write-Behind)

Write to the cache immediately and return; the database update happens asynchronously afterward.

Benefits: extremely fast writes (the caller doesn't wait on the DB), reduced database load (writes can be batched/coalesced before flushing).

Trade-off: potential data loss if the process crashes after the cache write but before the async DB flush completes.

5️⃣ Write-Around

Writes bypass the cache entirely and go directly to the database. The cache is only populated later, when that data happens to be read (effectively falling back to cache-aside on the read path).

Best for: write-heavy systems, and data that's written often but rarely read — e.g., logs, audit trails, bulk-ingested records. Helps avoid cache pollution: without write-around, every write would occupy cache space for data that may never actually be read again, evicting genuinely hot data to make room for it.

6️⃣ Refresh-Ahead

Popular ("hot") data is proactively refreshed before it expires, rather than waiting for a request to miss and trigger a reload.

Benefits: lower cache miss rate, consistently low latency (users never hit the "cold" path for data the system already predicted would still be needed), better overall experience.

Commonly used for: product catalogs, trending content, dashboard data — anything with predictable, sustained popularity where "refresh proactively" is cheaper than "let it expire and pay for a stampede of misses" (see pattern #7).

7️⃣ Cache Stampede Protection (Thundering Herd)

What happens when a hot cache key expires and thousands of requests hit the database simultaneously? That's a cache stampede (also called "Cache Breakdown" — see "The 3 Cache Killers" section below for how this compares to Cache Penetration and Cache Avalanche).

Key expires → 10,000 requests miss at the same moment → 10,000 identical queries hit the DB

Common solutions:

Distributed locks — only one process is allowed to recompute/refill the key; others wait.
Request coalescing (single-flight) — concurrent requests for the same missing key share one in-flight recomputation instead of each triggering their own.
Staggered expiration times (jitter) — add randomness to TTLs so a large batch of keys set at the same time doesn't all expire in the same instant.

8️⃣ Distributed Cache

Multiple application instances share a centralized cache rather than each maintaining its own local, inconsistent in-process cache.

Benefits: horizontal scalability (add more app instances without duplicating cache state per instance), shared cached data (one instance's cache-fill benefits every other instance), consistent performance across nodes (no "cold" instance behind a load balancer that hasn't warmed its own local cache yet).

Examples: Redis, Memcached, Hazelcast.

The 3 Cache Killers — How Redis Crashes Your Database

Caching looks simple on paper: check cache → hit, return; miss → fetch from DB, populate cache. This works fine until traffic hits scale. At scale, databases rarely die from bugs — they die from cache failures. These are the three named failure modes worth knowing cold:

1. Cache Penetration

Problem: Users repeatedly request data that doesn't exist in either the cache or the DB (e.g., userId = -1, or a scraper probing random IDs). Since the data doesn't exist, it can never be cached — so every request is a guaranteed cache miss and a guaranteed DB miss, forever. If this is automated/attacked at volume, it's effectively an unthrottled DB query flood with no cache layer able to absorb any of it.

Fixes:

Bloom filter in front of the cache/DB — a probabilistic structure that can definitively say "this key does not exist" before ever touching the DB, blocking invalid keys cheaply.
Cache the empty/negative result for a short TTL (e.g., null sentinel with 30-60s TTL) — so repeated requests for the same nonexistent key hit the cache (returning "not found") instead of re-querying the DB every time.

2. Cache Breakdown (Hot Key Expiry) — same as "Cache Stampede," pattern #7 above

Problem: A single extremely popular key (a viral profile, a trending product) is read by millions of requests. The instant that one key expires, all of those requests miss simultaneously and hit the DB together — a single hot key becomes a synchronized flood.

Fixes: (see pattern #7 for full detail) mutex/distributed lock so only one request recomputes the value while others wait, or logical/soft expiration — serve the stale value immediately while refreshing it in the background, so no caller ever waits on a synchronous DB hit.

3. Cache Avalanche

Problem: Many different keys expire at (approximately) the same moment — e.g., they were all set with the same TTL at the same time, or the cache server/cluster restarts and loses everything at once. Unlike breakdown (one hot key), this is a mass, simultaneous expiry across the whole keyset, flooding the DB with a broad wave of misses all at once.

Fixes:

TTL jitter — randomize expiry times (e.g., baseTTL + random(0, 60s)) so a batch of keys set together doesn't all die in the same instant.
For the "cache cluster restart" variant: warm the cache before routing traffic to it, and/or run the cache in a replicated/HA setup so a single node restart doesn't wipe the whole dataset.

Quick Comparison

Killer	Trigger	Scope	Primary Fix
Penetration	Requests for data that never existed	Any single (invalid) key, repeatedly	Bloom filter + cache negative results
Breakdown / Stampede	One hot key expires	One key, many concurrent requests	Mutex lock or logical/soft expiration
Avalanche	Many keys expire together (or cache restarts)	Whole keyset at once	TTL jitter + cache warming/HA

The meta-lesson: good caching design isn't about the happy path — it's about explicitly naming what breaks at scale and how you'd catch it before it takes down the database, not after.

Eviction Policies

Independent of which pattern above you use, once the cache is full, something has to decide what gets removed to make room:

LRU (Least Recently Used) — the default, works for most cases. Evicts whatever hasn't been accessed in the longest time.
LFU (Least Frequently Used) — better when access patterns are stable over time (the same items stay popular consistently). Evicts based on access count, not recency.

Example: for a product catalog, LFU tends to be the better fit. Popular products get accessed consistently over long periods — a bestseller shouldn't get evicted just because there was a brief lull in requests for it, which is exactly the failure mode LRU is prone to for this access pattern.

Key Questions a Caching Design Should Answer

Cache-Aside vs. Read-Through — when would you choose each? Cache-aside when you want explicit application-level control over what/how to cache (e.g., custom key composition, partial object caching); read-through when you want simpler, centralized application code and are fine with the cache layer's built-in loader behavior.
How do you handle stale data? Short, workload-appropriate TTLs; explicit invalidation on write (delete-or-update the cache entry when the source-of-truth changes); versioned cache keys; accept and bound staleness explicitly where it's tolerable (e.g., "eventually consistent within 30 seconds" for a dashboard) rather than pretending the cache is always perfectly fresh.
What is a cache stampede? See pattern #7 above — a hot key's expiry causing a burst of simultaneous cache misses that all hit the database at once.
How would you design cache invalidation? Options: TTL-based (simplest, bounded staleness), event-driven invalidation (the write path explicitly deletes/updates the affected cache key(s), e.g., via a CDC stream or direct call), or versioned/keyed-by-content approaches (the cache key itself encodes a version, so an old version simply stops being requested rather than needing active eviction). "There are only two hard things in computer science: cache invalidation and naming things" — worth acknowledging this is a genuinely hard problem, not a solved one.
How do you maintain cache consistency? Pick the pattern that matches your consistency needs: write-through for strong consistency (slower writes), write-behind/cache-aside for eventual consistency (faster, but a staleness window exists) — and be explicit about which one you're trading for which.
Redis vs. Memcached? Redis supports richer data structures (lists, sets, sorted sets, hashes, streams), persistence (AOF/RDB), replication, and pub/sub — useful when the cache needs to do more than store flat key-value pairs. Memcached is simpler, purely in-memory, multi-threaded per instance, and can have a lower per-operation overhead for pure key-value caching at very high throughput when none of Redis's extra features are needed.
How would you cache data for millions of users? Use a distributed cache (pattern #8) with consistent hashing to shard keys across cache nodes, size the cluster based on hot working-set size (not total data size), apply an eviction policy suited to the access pattern, and add stampede protection for any keys shared across large numbers of users (e.g., a global config or a trending list) since those are exactly the keys most likely to cause a thundering herd.

Real-World Example: Talkdesk's Dual-Store Architecture

At Talkdesk, a dual-store architecture was used: DynamoDB for key-value storage, Elasticsearch for full-text search, and Redis as the hot-data cache layer sitting in front of both. The point: the access pattern drove every technology choice — DynamoDB for point-lookup-style key-value access, Elasticsearch for flexible/full-text query needs, and Redis specifically for the subset of data that's read disproportionately often and can tolerate the cache's consistency/eviction trade-offs.

The Rule

Access pattern first. Technology second. Always.

Naming "Redis" is not a design decision — it's a detail that falls out of already having answered: what is being read/written, how often, how stale can it be, and what happens when it's evicted or expires? Answer those first, and the caching strategy (which of the 8 patterns above, which eviction policy, TTL length, stampede protection) follows naturally and demonstrates real design judgment.

The N+1 Query Problem — "It works locally, but is extremely slow in production"

tejaswipandava — Fri, 16 Oct 2026 04:19:00 +0000

The scenario: Your API loads 100 users and their orders. It works locally — but becomes extremely slow in production. What could be causing it?

One of the most common (and sneaky) causes: the N+1 Query Problem. 🐢

What Is It?

Suppose the application needs to display 100 users along with their orders. You'd expect the database to fetch everything together in one efficient round-trip. But here's what often actually happens:

1️⃣ The application sends one query to fetch all 100 users.
2️⃣ It then sends one additional query per user to fetch that user's orders.

Result:

✅ 1 query to fetch the users
❌ 100 additional queries — one per user — to fetch their orders

That's 101 database queries for a single API request.

This is the textbook N+1 Query Problem:

1 query fetches the parent records, then N additional queries fetch the related records
individually — one per parent row.

With 100 users, this might go unnoticed in a local test with a fast local DB and no network latency. With 10,000 users, the same code path triggers 10,001 queries — and it becomes catastrophic in production, especially over a real network to a real database.

Why It Hurts

Every additional query introduces:

Network round-trip time (multiplied by N)
Database connection pool usage (queries competing for limited connections)
Query-processing/parsing overhead per call
More load on the database server
Significantly higher API response time — often the actual root cause of "works locally, slow in prod," since local dev DBs have near-zero latency per query while production round-trips add up fast.

Why Does It Happen?

Most commonly with an ORM using lazy loading. The application loads the users first (one query). Later, when code accesses user.getOrders() inside a loop, the ORM silently and transparently issues another database query per user at that exact point — invisible in the code (it just looks like a normal getter call), which is exactly why it's easy to miss in code review and easy to reproduce only under realistic data volumes.

How to Fix It

🔗 1. Fetch related data together (join fetch)

Use a join or fetch strategy that retrieves the parent (users) and the related child data (orders) together, in a single query, instead of triggering a separate query per parent row.

📦 2. Use batch fetching

Instead of loading orders one user at a time, retrieve the orders for multiple users in one batch query (e.g., WHERE user_id IN (1, 2, 3, ..., 100)). This can turn 100 queries into a small handful — often reducing hundreds of queries down to just a couple.

🎯 3. Fetch only the required fields (projections)

If the API only needs a user's name, order number, and status, don't load the complete User,Order, and every related entity graph. Project only the fields actually needed for the response — less data over the wire, less mapping overhead, and often avoids triggering lazy-loaded associations entirely. — less data over the wire, less mapping overhead, and often avoids triggering lazy-loaded associations entirely.

📄 4. Add pagination

Never load thousands of users (and their orders) in a single request. Retrieve a bounded page and only the data needed for that page — this caps the worst-case query count and payload size regardless of how large the underlying table grows.

How Do You Detect the N+1 Problem?

Enable database query logging (e.g., Hibernate's show-sql / format_sql, or the database's own slow-query log) and actually read what gets generated for a given endpoint.
Monitor the number of queries per API request — a sudden spike proportional to the result set size (e.g., queries scale linearly with the number of users returned) is the classic signature.
Use an APM tool (e.g., New Relic, Datadog, Elastic APM) — most surface a per-request query count and flag "N+1 detected" patterns automatically.
Look for repeated queries with different IDs in the log — the same SELECT ... FROM orders WHERE user_id = ? template repeated 100 times with different parameter values is the smoking gun.
Test using realistic data volumes — this bug is invisible with 3 test users locally and becomes an incident with 10,000 users in production; load/integration tests need production-like data sizes to catch it before release.

Spring Boot / JPA: Which Fix Should You Prefer?

A common follow-up question: "Which approach do you prefer in Spring Boot — fetch joins, entity graphs, batch fetching, or DTO projections?"

There isn't a single universally-correct answer — it depends on the access pattern — but here's how to reason about the trade-offs:

Approach	How it works	Best when	Watch out for
Fetch Join (`JOIN FETCH` in JPQL)	Single SQL query joins parent + child in one round-trip	You always need the child collection alongside the parent for this query	Fetching multiple collections with `JOIN FETCH` in one query causes a Cartesian product explosion (rows multiply); JPA restricts fetching more than one `List`-type collection this way
Entity Graph (`@EntityGraph`)	Declaratively specifies which associations to eagerly fetch, per query/method, without changing the entity's default fetch type globally	You want eager fetching for specific use cases but keep lazy loading as the sane default elsewhere	Still one query per graph, but multiple collections in the same graph can hit the same Cartesian-product issue as fetch joins
Batch Fetching (`@BatchSize`, or `hibernate.default_batch_fetch_size`)	Lazy loading still triggers on access, but Hibernate batches the next N pending lazy loads into one `IN (...)` query instead of one-by-one	You have unavoidable lazy access patterns (e.g., generic repository code used in many contexts) and want a broad, low-effort mitigation	Doesn't eliminate extra queries entirely, just batches them — still more than a single join, though far better than true N+1
DTO Projections (JPQL constructor expressions, Spring Data interface projections, or a mapper)	Query selects only the exact fields needed directly into a DTO — no entity graph traversal at all	Read-only / API-response use cases where you don't need full entity objects or their lazy associations	Loses JPA entity features (dirty checking, lazy nav) — but for a read API, you don't want those anyway

Default preference: start with DTO projections for read-heavy API endpoints — they sidestep the N+1 problem entirely because there's no entity graph to lazily traverse in the first place, and they also naturally satisfy "fetch only the required fields" (fix #3) at the same time. Reserve fetch joins / entity graphs for cases where you genuinely need full entity objects (e.g., you're about to modify and save them), and use batch fetching as a safety net for lazy-loaded associations you can't easily eliminate across a large, shared codebase.

🔗 Related

Designing APIs for Massive Datasets — takes pagination (fix #4) and field projection (fix #3) further for very large datasets: cursor-based pagination, sparse fields, DB-level filtering, caching, and compression.
Caching Strategies — caching read-heavy query results to avoid re-paying the query cost at all.

Optimistic Locking vs. Pessimistic Locking — Handling Concurrent Updates in SQL Databases

tejaswipandava — Thu, 15 Oct 2026 04:15:00 +0000

What happens when two users try to update the same database row at exactly the same time? Without proper concurrency control, one update can silently overwrite the other, leading to inconsistent or incorrect data (the classic lost update problem). SQL databases solve this with two main concurrency control strategies: Optimistic Locking and Pessimistic Locking.

✅ Optimistic Locking (Version-Based)

Instead of locking the row up front, each transaction reads the current version number along with the data. When updating, the database verifies the version hasn't changed since it was read:

UPDATE accounts
SET balance = ?, version = version + 1
WHERE id = ? AND version = 5;

If another transaction already modified the row (and bumped the version), this UPDATE affects zero rows — the version check fails immediately, and the application detects the conflict via the affected-row count and can retry.

Key characteristics:

✔ No row-level lock held while "thinking" — the row stays fully available to others.
✔ Better throughput for read-heavy systems, since nothing blocks concurrent readers/writers.
✔ Conflicts are detected at UPDATE time, not up front.
✔ The application is responsible for retrying on failure (re-read the row, reapply the change, try again).

Mental model: "Try first. If someone already changed the data, fail fast and retry."

🔒 Pessimistic Locking (Row-Level Lock)

Before modifying data, the transaction explicitly locks the row so no one else can touch it:

SELECT * FROM accounts
WHERE id = 101
FOR UPDATE;

While one transaction holds the lock:

Other transactions requesting the same row wait in a queue rather than failing.
They execute only after the lock is released (commit/rollback).
No version conflict ever occurs, because updates to that row are forced to happen sequentially.

Key characteristics:

✔ Row locked immediately — guarantees exclusive access for the duration of the transaction.
✔ Other requests wait instead of failing outright.
✔ Strong consistency, even under heavy contention on the same rows.
✔ Lower concurrency (more waiting/blocking) but highly predictable execution order.

Mental model: "Wait your turn before making changes."

📌 When Should You Use Each?

Choose Optimistic Locking when:

Conflicts are rare.
The application is read-heavy.
High throughput matters more than avoiding occasional retries.
Retries are acceptable at the application layer.

Examples: e-commerce product catalogs, user profiles, content management systems.

Choose Pessimistic Locking when:

Data integrity is critical and lost updates are unacceptable.
Many users frequently update the same records (high contention).
Predictable, sequential execution is more valuable than raw throughput.

Examples: banking transactions, digital wallets, inventory management, ticket booking systems.

💡 Key Takeaway

There isn't a universally "better" approach. The right choice depends on your application's contention level, consistency requirements, throughput goals, and user experience. Understanding this trade-off is essential for designing scalable, reliable, high-performance backend systems — it's a decision made per use case (or even per table/operation), not a single global choice for the whole system.

In practice, each strategy carries a distinct failure mode worth watching for in production:
retry storms under optimistic locking when contention is higher than expected, and lock contention or deadlocks under pessimistic locking when transactions hold locks longer than intended.

Bloom Filters — The Probabilistic "Definitely Not" Gate

tejaswipandava — Wed, 14 Oct 2026 04:11:46 +0000

A Bloom filter answers exactly one question, cheaply, at massive scale: "could this key possibly exist?" It can say "definitely no" with 100% certainty, or "maybe yes — go check for real." It never says "yes" with certainty, and that asymmetry is the entire point of the data structure.

It is not a cache. It doesn't store keys, values, or "the list of things that failed before." It stores nothing but bits — and it's that detail that makes it so compact it can represent membership across tens of millions of keys in a few megabytes of memory.

The Core Mechanism

Setup: a fixed-size bit array (say, 1,000,000 bits, all initialized to 0) plus a small number of independent hash functions (commonly 3–7).

Insert a key (done once, for every valid key that exists — e.g., every real userId in the DB): run the key through each hash function, each one produces an index into the bit array, and set those specific bits to 1.

insert("user_42"):
    hash1("user_42") = 3   → bitArray[3]   = 1
    hash2("user_42") = 57  → bitArray[57]  = 1
    hash3("user_42") = 910 → bitArray[910] = 1

Do this for every valid key up front. After millions of inserts, the bit array is a dense mix of 1s and 0s — no individual key is retrievable from it. It's a pure Yes/No oracle, not a lookup table.

Check a key (at request time): run it through the same hash functions, and check whether all of the resulting bit positions are currently 1.

check("user_999999"):              // never actually inserted
    hash1("user_999999") = 3    → bitArray[3]   == 1 ✓
    hash2("user_999999") = 488  → bitArray[488] == 0 ✗   → STOP. Definitely NOT present.

If any bit is 0, the key is guaranteed absent — reject immediately, never touch the cache or DB. If all bits are 1, the key is "probably present" — but it might be a coincidence of other keys' bits overlapping (a false positive). Proceed to the real cache/DB check to confirm.

False Positives vs. False Negatives

These two failure modes are not symmetric — one is an accepted, tunable trade-off; the other should never structurally happen, and if it does, it's a sign something's broken operationally.

False positives — expected, and the whole trade-off of using this structure

Caused by hash collisions: unrelated keys happen to set an overlapping combination of bits, so a never-inserted key can coincidentally have all its bit positions already 1 from other keys. This gets more likely as the filter fills up (higher load factor). It's mitigated, not eliminated, by sizing the bit array bigger and/or using more hash functions — a direct memory-vs-accuracy trade-off, not a bug.

False negatives — should be structurally impossible, with exactly one real-world exception

Insertion only ever sets bits — it never clears them. So if a key was properly inserted, its bits are guaranteed to stay 1 forever... unless something else clears one of them. The only mechanisms that break this guarantee:

Cause	Why it happens
Naive deletion	Standard Bloom filters don't support deletion — clearing a bit to "remove" key A can silently break key B if B shared that bit position.
Counting Bloom Filter overflow	The deletion-capable variant uses small counters instead of single bits (increment on insert, decrement on delete). If a counter is sized too small and overflows/wraps, it can corrupt back to looking empty.
Staleness / not re-inserted on write	The filter was built from a DB snapshot, and a new valid key was created afterward but never inserted into the filter. Structurally the filter is fine — this is an operational gap, not a flaw in the structure itself.

Quick comparison

	False Positive	False Negative
Possible by design?	Yes — accepted trade-off	No — only via deletion bugs, overflow, or staleness
Consequence	One extra (cheap) real check against cache/DB	A real, existing key gets wrongly rejected
Severity	Low — self-correcting, costs one wasted lookup	High — silently drops valid users/data

Sizing It Correctly

The false-positive rate isn't a fixed property of a Bloom filter — it's something you design for upfront based on three numbers: the bit array size m, the number of hash functions k, and the number of items you expect to insert n. Given a target false-positive rate p, the standard formulas are:

m = -(n * ln(p)) / (ln(2))^2 to get the number of bits you need, and k = (m / n) * ln(2) to get the optimal number of hash functions for that bit array.

As a concrete example, to hold 10 million keys at a 1% false-positive rate, you land on roughly 9.6 million bits (about 1.2 MB) and 7 hash functions — which is the entire appeal of this structure: representing membership across 10 million keys in little more than a megabyte, instead of the hundreds of megabytes a real set of those keys would cost.

Where It Lives (Placement)

A Bloom filter is an in-memory data structure, not a network service — it lives wherever the code that needs to make the "should I even bother checking the cache/DB?" decision is running.

Location	When to use	Trade-off
App server memory (most common)	Default choice — checking a local bit array is nanoseconds	Each instance's filter must be independently built/kept in sync
Cache server (e.g., RedisBloom module, `BF.ADD` / `BF.EXISTS`)	Many app instances, want one shared/consistent filter instead of N copies	Adds a network hop to check — still far cheaper than a DB hit
API Gateway / Load Balancer	Rare — only to block abusive/scraping traffic before it reaches any app server at all	Unusual to couple business-logic membership checks into an infra layer

Cache-Aside vs. Read-Through

Where the check physically goes in your code depends on which caching pattern you're using. In Cache-Aside, the app owns the "check cache → miss → query DB" sequence directly, so the Bloom filter check goes right before the app's own DB call:

if (!bloomFilter.mightContain(userId)) return null; // skip cache AND DB entirely
User user = cache.getIfPresent(userId);
if (user == null) {
    user = userRepository.findById(userId);
    cache.put(userId, user);
}

In Read-Through, the app only ever makes one call to the cache; the DB-fetch logic is hidden inside the cache's registered loader. So the Bloom filter check has to move inside that loader, since that's the only place a DB call actually happens:

LoadingCache<String, User> cache = Caffeine.newBuilder()
    .build(key -> {
        if (!bloomFilter.mightContain(key)) return null; // loader skips the DB call
        return userRepository.findById(key);
    });

The Cold-Start Problem: What Happens on App Restart?

A freshly created Bloom filter starts with every bit at 0. If an app instance restarts and starts serving traffic before its filter is repopulated, mightContain(anyKey) returns false for every key — including real, valid ones. This isn't a tunable false-positive trade-off anymore; it's the filter wrongly rejecting all legitimate traffic, which is worse than not having a Bloom filter at all.

This has to be designed for explicitly:

1. Warm up before accepting traffic

Rebuild the filter from a full scan of valid keys in the DB before the instance passes its readiness check / joins the load balancer rotation:

@PostConstruct
public void warmUpBloomFilter() {
    bloomFilter = BloomFilter.create(...);
    userRepository.streamAllIds().forEach(bloomFilter::put);
    // only now mark this instance ready / healthy
}

2. Persist the filter so you don't rebuild from scratch every time

Serialize the bit array periodically to durable storage (disk, S3, or a shared store), and reload that snapshot on startup instead of re-scanning the whole DB every restart:

bloomFilter.writeTo(outputStream);                              // periodic snapshot
BloomFilter<String> restored = BloomFilter.readFrom(in, funnel); // on startup

3. Keep it in sync on writes — this is the part that's easy to miss

Every time a new valid key is written to the DB, the filter has to be updated in the same write path, or that new key will be falsely rejected until the next full rebuild:

public User createUser(User user) {
    User saved = userRepository.save(user);
    bloomFilter.put(saved.getId()); // keep filter in sync, same code path
    return saved;
}

If each app instance holds its own independent filter (the common case), a write on Instance A doesn't automatically propagate to Instances B, C, D. There are three common ways to handle that:

Approach	Mechanism	Trade-off
Shared filter (RedisBloom)	All instances read/write the same filter over the network	Network hop per check; sync problem disappears entirely
Pub/Sub broadcast	Instance A publishes "key X added" to Kafka/Redis pub-sub; every instance updates its own local filter	Local in-memory speed retained; small propagation-delay window
Periodic full rebuild	Every instance re-scans the DB on a schedule (e.g., every 10 min)	Simplest; accepts a bounded staleness window by design

The deeper rule: never let the filter become correctness-critical

The deeper rule underneath all three options is the same: because a cold or desynced filter risks false negatives, treat it as a pure optimization, never as the sole gate standing between a request and a real answer. Track a filterReady flag set only once warm-up completes — before that, fall back to hitting the real cache/DB for every request, accepting the temporary loss of the optimization rather than risking wrongly rejecting valid data.

FAQ

How does a Bloom filter differ from a cache? A cache stores actual values and can answer "yes, here's the data." A Bloom filter stores no data at all — only bits — and can only ever answer "definitely no" or "maybe." It's a filter in front of the real store, not a replacement for one.

Can you delete from a Bloom filter? Not from a standard one — clearing a bit can corrupt other keys sharing it. Use a Counting Bloom Filter (small counters instead of single bits) if deletion is required, and size the counters generously to avoid overflow.

What happens if the filter fills up? The false-positive rate rises as the load factor increases. Size the bit array ahead of time based on expected key count and target false-positive rate, using the sizing formulas above.

What if the app restarts? See the Cold-Start section above — warm up before serving, persist snapshots, and keep multi-instance filters in sync via a shared store or broadcast.

Backpressure — "Your system isn't crashing from too much traffic. It's crashing from too little control."

tejaswipandava — Mon, 12 Oct 2026 13:42:00 +0000

Most production outages aren't caused by a lack of raw compute — they're caused by the absence of a feedback mechanism between a fast producer and a slower consumer. That mechanism is called backpressure, and it's one of the most underrated concepts in distributed systems design.

🤔 The Problem

Imagine:

Incoming traffic: 1000 requests/sec
Your system can actually process: 200 requests/sec

Without any control mechanism, here's the familiar death spiral:

Traffic spikes.
Latency increases (requests queue up waiting to be processed).
Requests pile up in memory/queues.
Clients time out and retry, which adds more load on top of the backlog — amplifying it.
Thread pools / memory are exhausted.
The system crashes — and takes healthy downstream services down with it.

The root cause isn't the 1000 RPS itself. It's that nobody told the producer to slow down.

💡 What Backpressure Actually Is

Backpressure is a feedback signal from a consumer to a producer, communicating its current capacity. When the consumer is overwhelmed, it tells the producer to slow down — or stop — before the consumer collapses.

Backpressure is not about handling peak traffic. It's about surviving overload without collapsing.

With backpressure applied:

Excess load is rejected early, at the edge, instead of silently queuing.
Latency stays predictable for accepted requests.
Downstream services stay protected from cascading failure.
The system degrades gracefully instead of catastrophically.

Why it must be applied

Avoid overloading downstream services — upstream producers shouldn't be able to push consumers past their limits.
Resource efficiency — don't burn memory/CPU buffering data you'll never actually get to process; spend those resources on work you can finish.

Real-world examples

Network communication: a client sends 1000 RPS, the backend can only handle 700 RPS — requests queue up and eventually exhaust memory.
File processing: reading from disk is usually faster than writing (or than a downstream consumer processing records) — loading a large file faster than it can be consumed causes RAM pressure.

Pain points

Propagation — efficiently propagating a backpressure signal across microservice boundaries (not just within one process) is genuinely hard.
Latency cost — backpressure mechanisms (buffering, signaling, waiting) add latency of their own. Budget for it; it's the price of stability.

🌍 Where Backpressure Shows Up

Context	How backpressure is signaled
Reactive Streams (Project Reactor, RxJava)	Subscriber calls `request(n)` — it pulls exactly `n` items; the publisher cannot push more than that
Kafka	Consumers `poll()` at their own pace — Kafka never pushes to consumers; consumer lag is the backpressure signal
RabbitMQ	Consumer sets a prefetch count — broker only pushes that many unacknowledged messages at a time
TCP	Receiver advertises a receive window; sender throttles once the window fills
HTTP/2	Stream-level and connection-level flow-control windows, same idea as TCP but per-stream
Microservices (sync calls)	Bounded thread pools / connection pools + timeouts — once exhausted, new calls fail fast
File I/O	Disk reads are usually faster than writes/consumption — buffered readers must pause when the write/consume side falls behind

💡 The Two Models: Push vs Pull

This is the single most important mental model for backpressure:

	Push-based	Pull-based
Who drives the pace	Producer	Consumer
Risk	Producer can overwhelm a slow consumer	Consumer naturally throttles itself
Example	Naive message broadcast, unbounded event emission	Kafka `poll()`, Reactive Streams `request(n)`
Backpressure needed explicitly?	Yes — must be bolted on	Built-in by design

Most modern backpressure-aware systems (Reactive Streams, Kafka) default to pull-based consumption specifically because it makes backpressure the natural behavior rather than an afterthought — processing speed defines system throughput, not emission rate.

A small code example (Project Reactor)

Flux.range(1, 1_000_000)
    .onBackpressureBuffer(1000,                     // bounded buffer, not unbounded
        dropped -> log.warn("dropping: {}", dropped),
        BufferOverflowStrategy.DROP_OLDEST)
    .publishOn(Schedulers.boundedElastic())
    .subscribe(this::process);

The subscriber implicitly signals demand via request(n); if process() can't keep up, the buffer absorbs a bounded burst and then drops rather than growing memory unboundedly.

🛠️ Implementing Backpressure — Layer by Layer

Walking the same example (1000 RPS in, 200 RPS capacity) up the stack:

1. API Layer — Limit In-Flight Requests

Why first: the API is the entry point; if it doesn't apply backpressure, nothing downstream can be saved.
Behavior: allow a fixed number of concurrent requests; reject the rest immediately with 429 Too Many Requests / 503 Service Unavailable.
Result: pressure is pushed back upstream to the client; internal components stay protected.
Principle: a fast failure is better than a slow success. A client that gets an instant 429 can retry with backoff; a client stuck waiting 30s for a timeout ties up resources on both ends.

2. Bounded Queues

Problem: unbounded queues don't prevent overload — they just delay the failure while quietly exhausting memory.
Fix: bounded queues with explicit capacity limits. When full:
- stop accepting new tasks,
- reject or block the producer intentionally,
- this caps memory usage and makes overload visible instead of hidden.

3. Pull-Based Consumption

Push-based producers blast messages continuously, regardless of consumer readiness, and consumers get overwhelmed.
Pull-based consumers request work only when ready — the consumer controls the pace.

4. Respect Downstream/DB Limits as a Signal

Databases naturally enforce backpressure via:

connection pool limits,
row/table locks,
query execution timeouts.

Don't fight these limits — fail fast when connections are exhausted, and propagate the failure upstream instead of retrying blindly against an already-saturated DB.

5. Load Shedding (Last Resort)

When overloaded severely, the system should prioritize core functionality:

reject low-priority requests first,
disable non-critical features,
serve cached/degraded responses instead of computing fresh ones (e.g., a monitoring system switching to sampling instead of capturing every event under load).

🔎 How Do You Know You Need Backpressure?

Watch for these signals before they turn into an outage:

Queue depth climbing steadily instead of oscillating around a baseline.
p99 latency rising while p50 stays flat (a classic sign of queuing, not raw slowness).
Consumer lag growing in Kafka/queue-based systems.
Thread pool / connection pool utilization approaching 100% sustained, not just spiking.

⚠️ Backpressure vs. Look-Alikes

These get confused constantly — know the distinction:

Mechanism	What it does	Why it's not backpressure
Retries	Client resends a failed request	Actually amplifies load if the root cause was overload — the opposite of relief
Autoscaling	Adds more capacity reactively	Reacts too slowly (minutes) for a traffic spike that happens in seconds; doesn't stop the immediate overload
Rate limiting	Caps requests per client/key ahead of time	A static, predefined cap — not a dynamic feedback signal based on real-time consumer capacity
Circuit breaker	Stops calling a failing dependency entirely	Binary open/closed based on failure rate — not a graduated capacity signal
Backpressure	Dynamic, continuous signal from consumer capacity back to producer	The only one of these that's a true closed feedback loop

🎯 Key Takeaway

A common misconception:

❌ "Backpressure means rate limiting requests."

A more precise way to think about it:

✅ "Backpressure is a feedback loop where a consumer signals its real-time capacity back to the producer, so the producer can slow down, buffer within bounds, or shed load — before the consumer is overwhelmed. Rate limiting is a static, predefined cap; backpressure is dynamic and capacity-aware, and is typically implemented via bounded queues, pull-based consumption (Kafka, Reactive Streams), or explicit rejection at the entry point."

The distinction matters because it's a control-loop concept, not just a synonym for throttling.

10 Must-Know System Design Failure Modes

tejaswipandava — Sun, 11 Oct 2026 09:46:00 +0000

It's easy to explain how a system works when everything goes right. The harder — and more valuable — question is: what happens when it does not?

Distributed systems don't fail in infinite ways — they fail in a small number of recognizable patterns, and the same ones keep appearing in postmortems at every company. Once you can name a pattern, you can design against it before it ever becomes an incident. None of these need exotic infrastructure to hit — most show up the first time a system carries real traffic.

1. Single Point of Failure

Any component on the request path with no redundancy — one load balancer, one primary database, one message broker, one auth service. When it dies, everything behind it becomes unreachable, even though those instances are perfectly healthy.

Clients ──► [Load Balancer] ──► App Server 1, 2, 3 (all healthy)
                  ✗ dies
Clients ──► ✗ ──► unreachable, despite 3 healthy app servers

Fix: Redundancy + automatic failover. Run at least two instances of everything on the critical path, put health checks in front of them, and spread instances across availability zones so one datacenter problem can't remove an entire tier. Failover must be tested — a standby that's never taken traffic is a guess, not a plan.

A useful habit when reviewing any architecture: trace each request path and ask which single box, if deleted, stops all traffic. That's the box to fix first.

2. Cascading Failure

One slow dependency rarely stays contained. A database slows from 20ms to 5s → the service calling it holds its (fixed-size) thread pool open waiting → it stops answering its own callers → they block too. The slowness travels backwards up the call chain until the whole system looks down, even though only one component actually degraded.

Client → Service A → Service B → Service C → DB (slow: 20ms → 5s)
                                     ↑ threads held open
                          ↑ B's threads exhausted, stops responding
             ↑ A's threads exhausted, stops responding

Fix: Containment.

Aggressive timeouts — a caller gives up rather than waiting forever.
Circuit breakers — stop sending traffic to a dependency that's already failing.
Bulkheads — give each dependency its own connection pool, so one sick downstream can't consume every thread the service owns.

3. Retry Storm

Retries are usually good — they stop being good when everyone retries at the same moment. A service comes back after a blip; every client that failed during the outage retries at once, often multiple times each. The recovering service now sees several times its normal load, falls over again, generating a fresh round of retries. The retry logic becomes the outage.

3 clients × 3 retries each = 9x load on a service that just barely recovered → falls over again

Fix:

Exponential backoff with jitter — spreads attempts across a window instead of synchronizing on the same second.
Retry budget — caps retries as a percentage of total traffic.
Circuit breaker — clients stop retrying a service that's clearly down.

4. Cache Stampede

Caches hide how much load your database really receives. When one popular key expires, every request that would have been a cache hit becomes a miss at the same instant. All of them run the same expensive query against the database simultaneously — which suddenly takes the full read volume it's been shielded from for months.

Cache key expires → 1,000 concurrent requests all miss → all 1,000 hit the DB with the same query

Fix:

Request coalescing (single-flight) — the first miss recomputes the value; the rest wait on that in-flight result instead of each triggering their own DB query.
Refresh hot keys shortly before expiry — keeps them warm, avoiding the miss entirely.
Add jitter to TTLs — stops thousands of keys from expiring together in the first place.

5. Hot Partition

Sharding assumes traffic spreads evenly across shards. Real traffic doesn't. Pick a low-cardinality shard key (country, tenant, day) and one shard receives most of the writes while others idle. That shard saturates first — and adding more shards doesn't help, because the skew follows the key, not the machine count.

Shard key = country → Shard 1 (US): 92% of writes | Shard 2, 3: idle

Fix:

Choose a high-cardinality, evenly-distributed key — e.g., a hash of the user ID.
For genuinely "celebrity" entities that are hot no matter what: salt the key into sub-partitions, or give those entities dedicated capacity.
Track per-shard metrics, not cluster averages — an average hides the one node that's on fire.

6. Replication Lag

Read replicas make reads cheap, but they're always slightly behind the primary. Under load, that gap grows from milliseconds to seconds. A user updates their profile, the write commits on the primary, the next read lands on a lagging replica — the change appears to have vanished. Nothing is technically broken, but the bug report is still real.

Client writes → Primary (committed) → Replica (3s behind) → Client reads → gets STALE value

Fix: Be explicit about which reads can tolerate staleness.

Read-your-writes routing — route to the primary for a short window after a write by that same user.
Track replication position — only serve a read from a replica that has caught up past the write's position.
Analytics/feed reads can safely stay on replicas — they don't need freshness.

7. Duplicate Processing

Any retried request may execute twice. The first attempt reaches the server and succeeds, the response is lost to a timeout, and the client retries the same operation. For a read, harmless. For a payment, an email send, or an inventory decrement, the user is charged twice and the ledger is wrong.

Client → charge $100 → Server processes, charges $100 → response lost (timeout)
Client retries → charge $100 → Server charges $100 AGAIN → $200 total charged

Fix: Idempotency. Client attaches a unique idempotency key to each logical operation; server records that key with the result. When a request arrives carrying a key it's already processed, the server skips the work and returns the original response — same request, same outcome, any number of attempts.

8. Queue Backlog

Queues absorb bursts, which is exactly why they hide problems. If producers publish faster than consumers process — even by a small margin — the backlog grows continuously. Every health check stays green while the age of the oldest message climbs from seconds to hours, and by the time anyone notices, the data being processed is badly out of date.

Producers: 10,000 msgs/sec  |  Consumers: 2,000 msgs/sec  →  queue grows by 8,000/sec, forever

Fix: Make the imbalance visible and bounded.

Alert on consumer lag and oldest-message age, not just queue length alone.
Autoscale consumers on lag.
Bound the queue so it applies backpressure, and shed low-value work under sustained overload instead of accepting everything and falling further behind.

9. Poison Message

One malformed message can stop an entire queue. The consumer picks it up, throws, and never acknowledges it. The broker redelivers it, the consumer throws again, and the loop repeats forever. Throughput drops to zero while every valid message behind the bad one waits — and the crash loop can take the whole consumer group down with it.

[bad msg] [msg2] [msg3] ... → consumer crashes on bad msg → redelivered → crashes again → forever
                                                                              ↑ msg2, msg3... never processed

Fix:

Cap delivery attempts, then move the message to a Dead Letter Queue so the consumer can continue past it.
Alert on anything landing in the DLQ.
Keep enough context on the message to fix the underlying bug and replay it later.

10. Split Brain

During a network partition, two nodes can both conclude they are the primary. Each accepts writes from the clients that can still reach it, and the two copies of the data diverge. When the partition heals, there's no correct way to merge them — both sides accepted writes assuming they were authoritative.

Network partition:
  Node A (thinks it's primary) ← writes from Client 1
  Node B (thinks it's primary) ← writes from Client 2
  → data diverges, no clean merge once partition heals

Fix: Make leadership require a majority.

Quorum-based writes — only the side holding more than half the nodes can commit.
Consensus-based leader election (e.g., Raft) — prevents two leaders being elected in the same term.
Fencing tokens — let storage reject writes from a leader whose lease has already expired.

Patterns Worth Pairing Together

A few of these failure modes naturally show up as a package when you propose a design:

Add a cache → think about stampede protection up front.
Add sharding → name your shard key and why it won't go hot.
Add retries → pair them with backoff and jitter.
Add replicas → decide which reads can tolerate lag.

Anyone can draw boxes on a diagram — the useful skill is explaining how those boxes behave on the worst day of the quarter.

Quick Reference Table

#	Failure Mode	One-line cause	One-line fix
1	Single Point of Failure	No redundancy on a critical-path component	Redundancy + tested automatic failover
2	Cascading Failure	Slow dependency exhausts caller's thread pool, propagates upstream	Timeouts, circuit breakers, bulkheads
3	Retry Storm	Synchronized retries multiply load on a just-recovered service	Exponential backoff + jitter, retry budget, circuit breaker
4	Cache Stampede	Popular key expires, all misses hit DB simultaneously	Request coalescing, early refresh, jittered TTLs
5	Hot Partition	Low-cardinality shard key skews load to one shard	High-cardinality key, salting, per-shard metrics
6	Replication Lag	Replica behind primary, stale read after a write	Read-your-writes routing, replication-position tracking
7	Duplicate Processing	Retried request re-executes a non-idempotent side effect	Idempotency keys
8	Queue Backlog	Producer rate > consumer rate, backlog grows silently	Lag-based alerting/autoscaling, bounded queue, load shedding
9	Poison Message	Malformed message crash-loops the consumer forever	Capped attempts → Dead Letter Queue
10	Split Brain	Partition causes two nodes to both act as primary	Quorum writes, consensus leader election, fencing tokens

Summary

Distributed systems fail in a recurring, nameable set of ways: missing redundancy, failures propagating through tight coupling (timeouts/thread pools), synchronized client behavior (retries, cache expiry), uneven data/traffic distribution, staleness from replication, non-idempotent side effects, and disagreements about leadership during partitions. Each one has a well-understood fix — redundancy, timeouts/circuit breakers/bulkheads, backoff with jitter, request coalescing, better shard keys, staleness-aware read routing, idempotency keys, lag-based backpressure, dead letter queues, and quorum-based consensus. Designing a system well means assuming each of these will eventually happen, and deciding in advance how the system should behave when it does.

API Gateway vs Load Balancer vs Reverse Proxy

tejaswipandava — Sat, 10 Oct 2026 04:34:00 +0000

In system design and distributed architectures, these three components are often confused, but they serve distinct roles — and in practice, a single production system usually uses all three together, layered on top of each other.

1️⃣ API Gateway

Acts as a single entry point for client-facing APIs — crucial in a microservices architecture where dozens of services would otherwise each need their own public endpoint.
Handles authentication, authorization, and rate limiting to secure and monitor API traffic.
Implements request routing to the correct microservice, and can aggregate/compose multiple backend responses into a single response (API composition / backend-for-frontend).
Other common responsibilities: request/response transformation, API versioning, request validation, quota management, analytics/logging.
Examples: AWS API Gateway, Kong, Apigee, Zuul.

2️⃣ Load Balancer

Distributes incoming traffic evenly across multiple backend servers/instances for high availability and horizontal scalability.
Prevents server overload and provides fault tolerance by detecting failed instances (health checks) and routing away from them.
Operates at either:
- Layer 4 (Transport — TCP/UDP): routes based on IP/port, very fast, protocol-agnostic, no visibility into HTTP content.
- Layer 7 (Application — HTTP/HTTPS): can route based on URL path, headers, cookies; enables smarter routing (e.g., /api/v1/* → service A) but adds a bit more overhead.
Examples: AWS ELB/ALB/NLB, NGINX, HAProxy.

3️⃣ Reverse Proxy

Sits in front of one or more backend servers and forwards client requests to them — clients never talk to the backend servers directly.
Adds security by hiding backend server details/IP addresses (backend is not directly internet-addressable), and can filter malicious traffic.
Commonly performs SSL/TLS termination (decrypts HTTPS at the proxy so backend servers don't bear that CPU cost) and caching of static/responses to improve performance.
Examples: Nginx, Apache HTTP Server (mod_proxy), Envoy, Traefik.

🎯 Key Distinction — Purpose, Not Just Technology

The confusing part: the same software (e.g., NGINX) can act as all three, depending on configuration. The distinction is about role/intent, not the specific product:

Aspect	API Gateway	Load Balancer	Reverse Proxy
Primary purpose	API management (auth, rate limit, routing, composition)	Distribute traffic across many servers	Forward/hide backend, terminate SSL, cache
Layer	Usually L7, API-aware	L4 or L7	Usually L7
Client-facing?	Yes, single entry point for APIs	Sometimes (public-facing) or internal	Yes, typically
Knows about business logic/APIs?	Yes (routes per-endpoint, auth per-API)	No, just traffic distribution	Not necessarily
Adds auth/rate limiting?	Yes, core feature	No (not its job)	Not typically (can via plugins)
Example	AWS API Gateway, Kong	AWS ELB, NGINX (as LB)	NGINX (as proxy), Envoy

Mental model:

Reverse Proxy = "I sit in front of your server(s) and hide them; I may also cache/terminate TLS."
Load Balancer = "I sit in front of many servers and decide which one gets each request."
API Gateway = "I sit in front of your APIs, and I understand and manage API-level concerns (auth, quotas, routing per endpoint, aggregation)."

A Load Balancer and Reverse Proxy are often the same deployed component performing overlapping duties (many reverse proxies also load balance). An API Gateway is typically a more specialized, higher-level layer sitting in front of a load balancer, not a replacement for one.

🏗️ How They Typically Fit Together in a Real Architecture

Client
  → CDN (optional, static/cacheable content)
    → API Gateway (auth, rate limiting, routing, composition)
      → Load Balancer (distributes across service instances)
        → Reverse Proxy (per-service, TLS termination/caching) [often merged into the LB layer]
          → Backend service instances

In many real deployments, the reverse proxy and load balancer are literally the same NGINX/Envoy instance — "reverse proxy" describes what it's doing (hiding/forwarding to backends), while "load balancer" describes how it decides where to forward (distribution algorithm).

🎯 Q&A

Can one tool act as all three (API Gateway, LB, and Reverse Proxy)?
Yes — NGINX and Envoy are commonly configured to do all three simultaneously: terminate TLS (reverse proxy), distribute traffic across upstream instances (load balancer), and with plugins/extensions (or paired with Kong), handle auth/rate-limiting/routing (API gateway). The distinction is about the role being performed, not a strict 1:1 mapping to a specific product.

Why would you need both an API Gateway and a Load Balancer?
The API Gateway handles cross-cutting API concerns (auth, quotas, per-endpoint routing, composition) once, at the edge — it typically then forwards the request to a load balancer, which handles the mechanics of picking which specific healthy instance of that service should serve the request. They operate at different levels of abstraction: API Gateway = "which service, with what policy," Load Balancer = "which instance of that service, right now." This also answers where rate limiting belongs — at the API Gateway, since it's an API-level policy tied to caller identity, not something a Load Balancer (which is only aware of traffic distribution and health) is designed to handle.

L4 vs L7 load balancing — when would you choose each?
L4 (TCP/UDP) is faster and simpler — good when you just need raw connection distribution and don't care about HTTP semantics (e.g., a database proxy, or extremely high-throughput traffic where every microsecond counts). L7 (HTTP-aware) is needed when routing decisions depend on the request content — path-based routing (/api/* vs /static/*), header/cookie-based session affinity, or A/B testing/canary routing by request attributes.

How does a reverse proxy improve security?
It hides backend server IPs/topology from the public internet (backends aren't directly addressable), centralizes where security policies (WAF rules, IP allowlisting, TLS termination/certificate management) are enforced, and can absorb/filter malicious traffic (e.g., basic DDoS mitigation, malformed request rejection) before it ever reaches application servers.

What's the difference between a forward proxy and a reverse proxy?
A forward proxy sits in front of clients and makes requests on their behalf to the internet (e.g., corporate proxy hiding internal users from external sites — the server doesn't know who the real client is). A reverse proxy sits in front of servers and handles requests on their behalf from clients (the client doesn't know which backend server actually served the request).

Does adding a Reverse Proxy/Load Balancer/API Gateway add latency?
Technically each hop adds some latency (extra network hop + processing), but it's usually negligible (sub-millisecond to a few ms) compared to the benefits: security, TLS offload, caching, smarter routing, and horizontal scalability — which reduce overall latency far more than the hop itself costs, especially under load.

Summary

An API Gateway manages API-level concerns like auth, rate limiting, and routing across microservices. A Load Balancer distributes traffic across multiple instances of a service for availability and scale. A Reverse Proxy sits in front of backend servers to hide them, terminate TLS, and cache responses. In practice they're often layered together, and the same software (e.g., NGINX) can perform all three roles depending on configuration.

Latency vs Throughput vs Bandwidth

tejaswipandava — Fri, 09 Oct 2026 03:33:00 +0000

Many engineers use these terms interchangeably, but they measure three different things. Understanding the distinction helps you design better systems and reason clearly about networking and distributed systems.

1️⃣ Latency

Definition: The time it takes for data to travel from sender to receiver — "how long does a single request take?"

Examples:

API response time
Database query time
Network round-trip time (RTT)

If a request takes 40 ms to reach its destination, the latency is 40 ms.

Units: ms, μs
Lower is better.

2️⃣ Throughput

Definition: The amount of data/work successfully delivered per unit time — "how much work gets done every second?"

Examples:

Requests per second (RPS)
Messages processed per second
MB transferred per second

A system processing 10,000 requests per second has high throughput.

Units: RPS, Mbps, Gbps, Transactions/sec
Higher is better.

3️⃣ Bandwidth

Definition: The maximum capacity of a network link — "the width of the highway."

Examples:

100 Mbps internet connection
1 Gbps network interface
10 Gbps data center link

Bandwidth = theoretical maximum. Throughput = what you actually achieve.

Units: Mbps, Gbps
Higher is better.

🎯 The Highway Analogy

Concept	Highway Equivalent
Latency	How long it takes one car to travel from A to B
Throughput	How many cars reach the destination per hour
Bandwidth	Number of lanes available on the highway

A highway can have:

✅ High bandwidth, ❌ High latency (many lanes, but the road is very long / has tolls)
✅ Low latency, ❌ Low throughput (short road, but only 1 lane — a car gets there fast, but few cars fit)

These metrics are related but not identical — you can have low latency with low throughput, or the reverse, and designing for one doesn't automatically improve the other.

❓ Q&A

How are latency and throughput related, and can a system have one without the other?

Yes. Latency is the time for a single unit of work to complete; throughput is the rate at which many units of work complete over time. A system can have low latency but low throughput (a single request is fast, but the system can't handle many concurrently), or higher latency but high throughput (each request is slower, but thousands are processed in parallel via pipelining/batching).

Can a network have high bandwidth but low throughput?

Yes. Bandwidth is the theoretical max capacity; throughput is what's actually achieved. Common causes: protocol overhead, packet loss/retransmission, congestion, small TCP window sizes, server-side processing bottlenecks, inefficient application code, or too few concurrent connections to saturate the link.

For example, a 100 Mbps link achieving only 60 Mbps throughput is usually explained by real-world losses — TCP/IP header overhead, retransmissions from packet loss, congestion control backing off, the latency/RTT limiting how much data can be "in flight" (bandwidth-delay product), competing traffic sharing the link, or hardware/software processing limits.

What factors contribute to network latency, and how do you reduce it?

Latency is made up of several components:

Propagation delay (physical distance, speed of light in fiber)
Transmission delay (time to push bits onto the wire, depends on bandwidth)
Processing delay (routers, servers, load balancers, serialization/deserialization)
Queuing delay (congestion, buffering at routers/servers)
DNS resolution, TLS handshake, connection setup (TCP 3-way handshake)
Application-level delays (DB queries, GC pauses, lock contention)

Common ways to reduce it:

Cache close to the user (CDN, in-memory, edge caching)
Reduce network hops / use geo-distributed regions closer to users
Use connection pooling & keep-alive to avoid repeated handshakes
Use async/non-blocking I/O to avoid queuing delays
Batch or pipeline requests where possible
Optimize serialization (protobuf/avro vs JSON)
Use faster transport protocols (HTTP/2, QUIC/HTTP/3, gRPC)
Reduce database query time (indexes, denormalization, read replicas)
Load balance to avoid hotspots/queuing on a single node
Precompute / prefetch data

How does caching impact latency?

Caching reduces latency by serving data from a faster, closer store (memory, CDN edge, local cache) instead of round-tripping to a database or origin server. It removes the need for expensive computation/disk/network I/O on repeated requests. It primarily helps latency, and indirectly throughput too, since the backend capacity freed up can serve more requests.

How does horizontal scaling impact throughput?

Adding more instances/nodes behind a load balancer lets a system handle more concurrent requests, directly increasing throughput (more RPS). It typically does not improve — and may slightly worsen — the latency of an individual request (still bounded by network RTT + per-request processing time), unless it also reduces contention/queueing delay.

Why doesn't increasing bandwidth always reduce latency?

Bandwidth affects how much data can be sent per second, not how fast a single bit/packet travels. Latency is dominated by propagation delay (distance/speed of light) and processing/queuing delays — a wider highway (more lanes) doesn't make a single car arrive faster if the road length and speed limit stay the same. Bandwidth helps throughput; it only marginally helps latency by reducing transmission delay for large payloads.

Does latency or throughput/bandwidth matter more for a given use case?

It depends on the workload:

Online gaming: latency matters most. Games need near real-time responsiveness (low RTT) for actions to feel instant; packet sizes are small, so throughput is rarely the bottleneck. High latency causes lag even with plenty of bandwidth.
Video streaming: bandwidth and sustained throughput matter most. Streaming needs enough sustained data rate to deliver frames without buffering; a few hundred ms of extra latency (buffering) is imperceptible, but insufficient bandwidth causes stuttering and quality drops. (Exception: live, interactive video — e.g., video calls — behaves more like gaming and is latency-sensitive.)

💡 Summary

Latency measures how long a request takes, throughput measures how much work is completed per unit time, and bandwidth measures the maximum capacity of the communication channel. Treating these as the same thing is one of the most common mistakes in reasoning about system performance.