DEV Community: Arnav Sharma

How to Secure OAuth 2.0 in Production

Arnav Sharma — Fri, 14 Aug 2026 10:17:11 +0000

You got OAuth working. Users click "Login with Google," a token comes back, your API accepts it. Shipped.

And then someone steals that token from the URL fragment. Or intercepts the authorization code on a mobile device. Or crafts a redirect URI that funnels credentials to their domain. These aren't theoretical attacks. They're documented in RFC 9700 with section numbers and mitigation steps, and they work against implementations that looked fine on the happy path.

If you've read my intro to OAuth and OpenID Connect, you know the basic flow. This post picks up where that one left off. Here's what you need to actually harden it.

The flows you need to stop using

Two OAuth grants are officially dead. Not deprecated-but-still-kinda-fine. Dead.

Implicit flow (response_type=token) puts access tokens directly in the URL fragment. That means they show up in browser history, leak through Referer headers to any third-party resource on your callback page, and get exposed by open redirectors. You can't sender-constrain them. You can't rotate them. RFC 9700 Section 2.1.2 says SHOULD NOT. OAuth 2.1 removes it entirely.

Resource Owner Password Credentials (ROPC) hands the user's actual password to your client application. This breaks the entire point of delegated authorization. It's also incompatible with MFA, WebAuthn, passkeys, or any modern authentication mechanism. RFC 9700 Section 2.4 says MUST NOT. Gone in 2.1.

So what do you use? Authorization Code flow. For everything. Public clients, confidential clients, SPAs, mobile apps. One flow to rule them all, protected by PKCE.

🔑 PKCE is non-negotiable

PKCE (Proof Key for Code Exchange, RFC 7636) solves three problems at once: code interception, code injection, and CSRF.

Here's the attack without it. Your app redirects the user to the authorization server. The AS issues a code and redirects back. But on mobile, multiple apps can register the same custom URI scheme. A malicious app intercepts the redirect, grabs the code, and exchanges it for tokens. Game over.

With PKCE, your app generates a random code_verifier before starting the flow, computes a code_challenge from it (SHA-256 hash, base64url-encoded), and sends only the challenge to the AS. When exchanging the code for tokens, you prove possession of the original verifier. The attacker has the code but not the verifier. Useless.

import crypto from 'node:crypto';

function generatePKCE() {
  const verifier = crypto.randomBytes(32).toString('base64url');
  const challenge = crypto
    .createHash('sha256')
    .update(verifier)
    .digest('base64url');
  return { code_verifier: verifier, code_challenge: challenge, method: 'S256' };
}

Always use S256. The plain method sends the verifier as the challenge itself, which means anyone who can read the authorization request already has it. Defeats the purpose.

RFC 9700 Section 2.1.1 makes PKCE mandatory for public clients. But honestly, use it for confidential clients too. OAuth 2.1 will require it for all client types. There's no downside.

🎯 Redirect URI: exact match or get wrecked

Your authorization server must validate redirect URIs with exact string matching. Not pattern matching. Not wildcard subdomains. Exact.

Why? Because https://*.myapp.com/callback also matches https://evil.myapp.com/callback. And subdomain takeovers are common. An attacker claims an abandoned subdomain, registers it as a redirect URI, and now authorization codes flow directly to them.

It gets worse with naive pattern matching. A poorly implemented check for https://myapp.com might also accept https://attacker.com/.myapp.com. Combined with open redirectors, this lets attackers steal tokens without needing a subdomain takeover at all.

The only exception RFC 9700 allows: localhost with variable ports for native desktop apps during development (RFC 8252 Section 7.3). Everything else? Exact strings.

Where to store tokens in the browser

This is probably the most debated topic in OAuth security. Every option has problems.

localStorage: Accessible to any JavaScript on your page. One XSS vulnerability and every token is gone. Not great.

Memory only: Safe from XSS-based theft, but tokens vanish on page refresh. Terrible UX.

httpOnly secure cookies: JavaScript can't read them, so XSS can't steal them. But now you need CSRF protection on every request. And you're back to dealing with cookie semantics, SameSite attributes, and cross-origin headaches.

The BFF pattern (Backend For Frontend): This is what RFC 9700 recommends for browser-based apps. Your frontend never touches tokens at all. A server-side component handles the OAuth flow, stores tokens in a server session, and issues a plain httpOnly session cookie to the browser. The browser sends the cookie, the BFF attaches the access token before forwarding to your API.

Tokens never exist in JavaScript. XSS can't steal what isn't there. It adds a component to your architecture, but it's the only approach where a single XSS doesn't mean full token compromise.

🧠 Refresh token rotation and reuse detection

Access tokens should live for 5-15 minutes. Short enough that a stolen one has limited blast radius. But you need refresh tokens for session continuity.

For public clients, RFC 9700 Section 4.14.2 requires either sender-constraining (DPoP or mTLS) or refresh token rotation. Rotation means every time a client uses a refresh token, the AS issues a new one and invalidates the old one. Each refresh token is single-use.

The important part: reuse detection. If someone presents an already-used refresh token, that's a compromise signal. The AS should immediately revoke the entire token family. Not just that one token. Everything associated with that session.

So if an attacker steals a refresh token and races the legitimate client, one of them will present the invalidated token. The AS catches it and nukes both sessions. Aggressive, but correct.

⚡ Validating tokens on your resource server

If you're accepting JWTs as access tokens, validation isn't optional. Every request needs these checks:

import jwt from 'jsonwebtoken';
import jwksClient from 'jwks-rsa';

const client = jwksClient({
  jwksUri: 'https://as.example/.well-known/jwks.json',
});

async function validateAccessToken(token) {
  const { header } = jwt.decode(token, { complete: true });
  const key = await client.getSigningKey(header.kid);
  return jwt.verify(token, key.getPublicKey(), {
    issuer: 'https://as.example',
    audience: 'https://api.example.com',
    algorithms: ['RS256'],
  });
}

Check the signature against the AS's published JWKS. Verify iss matches your expected authorization server. Confirm aud includes your resource server's identifier. Reject if exp is in the past. Fail on any one of these and you reject the token. No partial credit.

And one thing I see constantly: don't use ID Tokens as access tokens. An ID Token's audience is the client application. It's an authentication assertion, not an authorization credential. Sending it as a Bearer token to your API is wrong, even if it "works" because your API doesn't check the audience claim. If you need a refresher on how JWTs work and why claims matter, I wrote about that in my JWT deep dive.

The state parameter still matters

Even with PKCE handling CSRF protection, you should still use state. It's cheap insurance.

import crypto from 'node:crypto';

// Before redirecting to AS
const state = crypto.randomBytes(16).toString('base64url');
session.oauthState = state;

// On callback
if (query.state !== session.oauthState) throw new Error('CSRF detected');
delete session.oauthState; // one-time use

Bind it to the user's session. Make it one-time-use. And if your AS supports the iss response parameter (RFC 9207), validate that too. It prevents mix-up attacks where a malicious authorization server tricks your client into sending codes to the wrong token endpoint.

📌 What to actually do

Stop reading specs for a second. Here's the concrete checklist:

Use Authorization Code + PKCE (S256) for every client type. No exceptions
Register redirect URIs as exact strings. Kill any wildcard patterns
Store tokens server-side (BFF pattern) for browser apps. localStorage is a liability
Set access token lifetime to 5-15 minutes. Use refresh token rotation with reuse detection
Validate JWT access tokens fully: signature, issuer, audience, expiration
Never send ID Tokens to your API as bearer credentials
Include state in every authorization request. One-time, session-bound, unpredictable

RFC 9700 is 80+ pages, but those seven points cover 90% of what will actually get you owned in production. The spec exists because people shipped without them and got burned.

If you haven't read the basics yet, start with my post on how OAuth and OpenID Connect work together.

More from me

I write about backend systems, auth, and developer tooling at arnavsharma.dev.

How Nginx Handles Thousands of Connections With One Thread

Arnav Sharma — Fri, 14 Aug 2026 10:16:03 +0000

You've copied an nginx config from Stack Overflow. It works. You have no idea why.

Maybe you added a location block that broke something else. Maybe you spent an hour debugging a 502 that turned out to be a missing trailing slash on proxy_pass. Maybe you're just stacking directives you found in blog posts, hoping nothing conflicts.

That's fine. Everyone starts there. But the moment you need to debug a production routing issue at midnight, "it works, don't touch it" stops being a strategy. So let's actually understand what this thing does under the hood.

🧠 The process model

When you start nginx, you get one master process and several worker processes. That's it. No thread pools per request, no spawning child processes for each connection.

The master runs as root. It reads your config, binds to ports 80 and 443, and spawns workers. Then it basically sits there managing the lifecycle: starting new workers, gracefully shutting down old ones during config reloads, and restarting them if they crash.

Workers do all the real work. Every connection, every request, every byte of response goes through a worker. By default, worker_processes is set to 1, but basically everyone sets it to auto, which spawns one worker per CPU core.

# /etc/nginx/nginx.conf (top-level context)
worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 4096;
    multi_accept on;
}

Each worker handles connections independently. They don't share memory or coordinate on who takes which request. The OS kernel distributes incoming connections across the listening workers.

And here's where it gets interesting.

⚡ Why the event loop wins

Apache's traditional model (prefork) spawns one process per request. Its worker MPM uses one thread per request. Either way, you're paying for an OS-level context switch and a chunk of memory for every single connection sitting there waiting for data.

Nginx does something different. Each worker runs a single-threaded event loop. One thread, thousands of connections.

The worker registers all its sockets with the kernel using epoll (Linux) or kqueue (macOS/BSD). Then it waits. When any socket has data ready, the kernel tells the worker which ones. The worker processes those events, fires off responses, and goes back to waiting. Never blocks. Never sits idle consuming resources for a connection that isn't doing anything right now.

This is why nginx can handle 10,000+ concurrent connections on hardware where Apache would fall over. It was literally built to solve the C10K problem back in 2002.

But there's a catch. If anything in the event loop blocks, like reading a large file from a slow disk, the entire worker stalls. Every other connection on that worker just waits. That's why nginx added thread pools (aio threads;) to offload blocking disk I/O to a pool of 32 threads by default. The event loop stays responsive.

How config blocks map to requests

The config is hierarchical: http wraps server wraps location. Each level inherits from its parent and can override.

http {
    # applies to all virtual hosts
    gzip on;

    server {
        listen 80;
        server_name api.myapp.com;
        # this server block handles requests to api.myapp.com

        location /users {
            proxy_pass http://127.0.0.1:3000;
        }
        location /static {
            root /var/www;
        }
    }

    server {
        listen 80;
        server_name admin.myapp.com;
        # different hostname, different routing
    }
}

When a request arrives, nginx picks the right server block by matching the Host header against server_name. Priority: exact match first, then longest leading wildcard (*.example.com), then longest trailing wildcard (mail.*), then first matching regex. No match? Falls back to default_server.

Then it finds the right location block. This is where people get tripped up.

🎯 Location matching precedence

Location matching isn't first-match-wins. It has a specific priority order that ignores the sequence in your config file:

= /exact - Exact match. Stops immediately if matched.
^~ /prefix - Longest prefix match with the "stop searching" modifier. Skips regex.
~ /regex or ~* /regex - Regular expressions (case-sensitive and case-insensitive). First regex match in config order wins.
/prefix - Longest prefix match without the modifier. Used only if no regex matched.

So a request to /api/users/123 checks all prefix locations, finds the longest match, then checks regexes. If a regex matches, it wins over the prefix. Unless that prefix had ^~, which blocks regex from overriding it.

Confusing? Yeah. In my experience, most routing bugs come from people assuming locations are evaluated top-to-bottom. They're not.

The six jobs nginx actually does

Nginx wears a lot of hats. But they all boil down to six things:

Static file serving with root or alias, try_files for SPA fallbacks, and sendfile on for zero-copy delivery straight from kernel space. Fast.

Reverse proxying with proxy_pass. Takes a client request, forwards it to your app server, and returns the response. Your app never sees the internet directly. I've written more about what reverse proxies do and how they differ from forward proxies .

Load balancing via upstream blocks. Round-robin by default, or least_conn, ip_hash, and weighted distribution. If you want the full breakdown of how these algorithms compare, I covered that separately.

TLS termination. Nginx handles the SSL/TLS handshake, decrypts traffic, and forwards plain HTTP internally. Your app servers don't need certificates or crypto overhead.

Caching with proxy_cache_path. Stores upstream responses on disk and serves them directly for repeat requests without hitting your backend.

Rate limiting using limit_req_zone and limit_req. Throttle by IP, by endpoint, whatever you need.

upstream backend {
    least_conn;
    server 10.0.0.1:8080 weight=3;
    server 10.0.0.2:8080;
    server 10.0.0.3:8080 backup;
}

server {
    listen 443 ssl;
    server_name app.example.com;
    ssl_certificate /etc/ssl/cert.pem;
    ssl_certificate_key /etc/ssl/key.pem;

    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

That's TLS termination, load balancing, and reverse proxying in 15 lines.

🛠️ Gotchas that bite everyone

These are the ones I see constantly. Bookmark this section.

The trailing slash on proxy_pass. This single character changes everything. proxy_pass http://backend; (no trailing slash) passes the full original URI. proxy_pass http://backend/; (with trailing slash) strips the matched location prefix. So if your location is /api/ and the request is /api/users, the first forwards /api/users, the second forwards /users. Mismatching this causes mystery 404s.

Missing proxy_set_header Host. Without it, your upstream sees the internal hostname (like 127.0.0.1:3000) instead of the original Host header. And without X-Forwarded-For, your app has no idea what the client's actual IP address is. Every proxy config needs these headers.

413 Request Entity Too Large. The default client_max_body_size is 1m. One megabyte. Any file upload over that gets rejected with a 413 before your app even sees it. Set it explicitly: client_max_body_size 50m; or 0 to disable the limit entirely.

502 versus 504. A 502 means nginx connected to your upstream but got garbage back (or the connection was refused). Your app probably crashed. A 504 means nginx waited for a response and gave up after proxy_read_timeout (default 60s). Your app is alive but slow. Different problems, different fixes.

Never skip nginx -t before reload. Always run nginx -t to test config syntax. Then nginx -s reload for a graceful reload where old workers finish their current requests. A full restart drops every active connection. I've seen teams push broken configs to production because they reloaded without testing first. Don't be that team.

📌 Takeaways

Nginx's event-driven workers handle thousands of connections each without spawning threads per request
Config hierarchy is http > server > location, with inheritance flowing down
Location matching has a fixed priority order that ignores config file sequence
Most production nginx issues come from trailing slashes, missing proxy headers, and the 1MB body size default
Always nginx -t before you reload. Always.

If you're running nginx as a load balancer, the post on load balancing algorithms goes deeper on the strategies behind upstream blocks.

More from me

More posts on this kind of thing at arnavsharma.dev.

Microservices vs Monolith: Why You Should Probably Start With One Big App

Arnav Sharma — Fri, 14 Aug 2026 10:15:14 +0000

What if the architecture that "doesn't scale" is the one that gets you to production fastest?

I keep seeing the same pattern. A dev starts a side project, draws 8 services on a whiteboard before writing a single line of code, spends three weekends wiring up service discovery and inter-service auth, and the actual product logic is 200 lines across all of them. The project dies in a Docker Compose file.

Microservices aren't wrong. But they solve problems most early-stage projects don't have. And the cost of splitting too early is way higher than the cost of splitting too late.

🎯 Four options, not two

People talk about this like it's binary. Monolith or microservices. But there are actually four architecture patterns on the spectrum, and two of them don't get enough attention.

The monolith. One deployable unit. All modules share a process, a database, and a release cycle. Not a dirty word. Just means you ship one thing.

The modular monolith. Still one deployment, but with enforced internal boundaries. Modules have explicit public APIs and can't reach into each other's guts. Shopify runs a 2.8 million line Rails app this way. They use a tool called Packwerk to enforce dependency rules between components. One deploy, but internally decoupled. This is honestly where most teams should land.

Microservices. Independently deployable services, each owning its own data, talking over the network. The defining trait isn't size. It's that you can deploy service A without touching service B.

The distributed monolith. The failure mode nobody plans for. You split your code into 15 services, but they still have to be deployed together, tested together, and they fail together. You've got all the operational complexity of a distributed system with none of the independence benefits. The worst of both worlds. And honestly? This is what most failed microservices migrations actually produce.

What you gain when you split

I'll be fair to the microservices side. When the split is done well, you get real benefits:

Dimension	Monolith	Microservices
Deployment	Ship everything together	Deploy one service independently
Team ownership	Coordinate releases across teams	Each team owns full lifecycle
Fault isolation	One bug can bring down the whole app	Failures stay contained per service
Scaling	Scale the entire unit	Scale only the hot path
Tech flexibility	One stack for everything	Each service picks its own tools

These are real. At Amazon's scale, with thousands of teams, independent deployment isn't a nice-to-have. It's survival. But Amazon has the org structure to back it up.

🧠 What gets worse (the hidden costs)

Here's what people skip over when they're excited about service boundaries.

Network calls replace function calls. An in-process method call takes nanoseconds. A network request takes milliseconds plus serialization overhead. Chain five services together and your p99 latency floor is 250ms before your code even does anything. That latency budget adds up fast, especially when you're trying to hit sub-second response times. If you want more on how traffic flows through these layers, the post on API gateways covers the routing side.

No cross-service transactions. Forget ACID. You're in saga territory now, dealing with eventual consistency and compensation logic. For a lot of apps, this complexity just isn't worth it.

CI/CD per service. N services times M environments equals an explosion of build pipelines. Each one needs monitoring, alerting, and someone who knows what to do when it breaks at 3am.

Observability costs more than the services themselves. Distributed tracing, log aggregation, correlation IDs, and service meshes. You need all of it just to answer "why did this request fail?" Something that was a stack trace in a monolith becomes a detective hunt across 6 services.

On-call surface multiplies. Each service needs an owner. With 50 services you need 50 runbooks, and you'd better hope those runbooks are up to date.

Cognitive load. Developers now need to understand network failure modes, retries, idempotency, and circuit breakers. That's a lot of accidental complexity for a team that just wants to ship features.

Conway's law is not optional

Here's a thing that doesn't get said enough. Your architecture will mirror your org structure whether you plan for it or not. Conway's Law from 1967. It's not a suggestion.

Amazon's two-pizza team model works because one team equals one service equals clear ownership. But if you're a 20-person company with 3 teams and you've drawn 50 microservices on a diagram? You don't have the people to support that. You'll end up with roughly as many real services as you have real teams, no matter what the architecture doc says.

So match your architecture to your org. Not your aspirations.

📌 Signals it's time to split

Start with a monolith. But stay alert for these signals:

Deployments require cross-team coordination and are becoming a bottleneck. Teams are waiting on each other to ship.
Teams keep stepping on each other's code. Merge conflicts every sprint, broken features from unrelated changes.
One component needs to scale 100x while the rest stays flat. Your checkout service gets hammered during sales but your admin panel sits idle.
Compliance or security requires data isolation. PCI, PII regulations that demand certain data lives in its own boundary.
Build times are so long they destroy productivity. If your CI takes 45 minutes, people stop running it.

None of these signals mean "rewrite everything into 30 services overnight." They mean: identify one bounded context, extract it using the strangler fig pattern, and see if your life gets better. One at a time.

⚡ The playbook (short version)

If you're going to split, do it in this order:

Start modular. Enforce boundaries inside the monolith first. Shopify's approach. Separate your modules with clear interfaces, even if they still deploy together. This is cheap and reversible.
Identify the bounded context. Which module has a genuinely different scaling profile, team ownership, or data isolation need?
Strangler fig. Route traffic to the new service incrementally. Old code handles what's left. No big-bang rewrite. Martin Fowler described this pattern back in 2004, and it remains the safest way to extract.
Own the operational cost. Before you extract, make sure you have CI/CD, monitoring, tracing, and an on-call rotation for the new service. If you don't have these, you're not ready.

If you're partitioning data across services and using event-driven communication, understanding how Kafka partitions and consumer groups work will save you some grief on the messaging side.

Real-world evidence

I'm not just making this up. The pattern "start monolith, split later" has strong backing:

Martin Fowler wrote in 2015: "Almost all the successful microservice stories have started with a monolith that got too big and was broken up." His explicit advice is don't start a new project with microservices, even if you're sure it'll be big enough to justify them eventually.

Segment had 140+ microservices (one per integration partner) and consolidated them back into a single service. Three engineers were spending most of their time keeping the system alive instead of building features. After consolidating, their shared library improvement rate jumped from 32 to 46 per year and on-call paging for load spikes disappeared entirely.

Amazon's Prime Video team moved a video quality monitoring tool from Step Functions plus Lambda back into a single ECS process and cut that tool's infrastructure cost by over 90%. But here's the thing people get wrong about this one: it was one monitoring tool, not Prime Video's entire streaming platform. Prime Video still runs hundreds of services. The 90% figure applies to this specific pipeline's cost, not Amazon's bill.

DHH has been running Basecamp on a single Rails monolith serving millions of users with a small team since 2016. His argument: microservices add complexity that only pays off at organizational scale most companies never reach. Hard to argue with the results.

The short answer

Start with a monolith. Make it modular. Split when a real signal appears, not when a conference talk makes you feel behind. The distributed monolith is the actual enemy here, and you create it by splitting before you understand your domain boundaries.

Most of us aren't building the next Prime Video. We're building apps that need to ship, iterate, and not collapse under the weight of their own infrastructure.

Where else to find me

I write about system design and architecture decisions at arnavsharma.dev.

How Message Queues Work: The Architecture Your Signup Endpoint Needs

Arnav Sharma — Fri, 14 Aug 2026 10:14:07 +0000

Your signup endpoint takes four seconds. The user clicks "Create Account," stares at a spinner, and you're just sitting there sending a welcome email synchronously. The password got hashed in 200ms. The database write took 50ms. The remaining 3.7 seconds? Your SMTP provider thinking about life.

And the user doesn't care about that email. They just want in.

This is the exact problem message queues solve. You take the slow work, shove it into a queue, and respond to the user immediately. Something else picks it up later. Probably within milliseconds. But the point is: not on the request path.

Same story with image resizing, PDF generation, webhook delivery. Anything that's slow and doesn't need to happen before you respond to the user? Queue it.

🎯 What a queue actually is

A message queue is a buffer sitting between a producer and a consumer, managed by a broker. The producer drops a message in. The consumer pulls it out and processes it. That's it.

The producer doesn't know or care who processes the message. The consumer doesn't know or care who sent it. They're decoupled. If your email service goes down for thirty seconds, messages pile up in the queue and get processed when it recovers. No lost signups. No retries from the client.

This is point-to-point messaging: one message goes to one consumer. If you've got three workers pulling from the same queue, the broker hands each message to exactly one of them. Competing consumers. More workers means faster drain.

Here's what enqueueing looks like with BullMQ (a Redis-backed queue for Node.js):

import { Queue } from "bullmq";

const emailQueue = new Queue("welcome-emails");

// In your signup handler - takes ~1ms instead of 4 seconds
await emailQueue.add("send-welcome", {
  userId: "usr_abc123",
  email: "newuser@example.com",
});

And the worker that picks it up:

import { Worker } from "bullmq";

const worker = new Worker("welcome-emails", async (job) => {
  // This runs outside the request path
  await sendWelcomeEmail(job.data.email);
  console.log(`Welcome email sent to ${job.data.userId}`);
}, { connection: { host: "127.0.0.1", port: 6379 } });

worker.on("failed", (job, err) => {
  console.error(`${job?.id} failed: ${err.message}`);
});

Your signup endpoint now returns in 250ms. The email gets sent a few hundred milliseconds later by a separate process. The user never notices.

⚡ Ack, visibility timeout, and redelivery

So what happens when a worker crashes mid-processing? The message just disappears? No.

Queues don't delete a message when it's picked up. They hide it. In SQS, this is called a visibility timeout: the message becomes invisible to other consumers for N seconds (default 30). If your worker finishes and deletes the message, great. If it dies, the timeout expires and the message reappears for another worker to grab.

RabbitMQ does the same thing differently. Your consumer sends a manual ack when it's done. No ack? The broker requeues.

But here's where it gets tricky. Say processing takes 35 seconds and your visibility timeout is 30. The message reappears while the first worker is still on it. Now two workers are processing the same job. Duplicates.

This is why queues give you at-least-once delivery, not exactly-once. Your consumers need to be idempotent. Processing the same message twice should produce the same result. Check if the email was already sent before sending it again.

🛠️ Dead letter queues and what to alert on

Sometimes a message is just bad. Malformed payload, referencing a deleted user, hitting a bug that'll never self-heal. It fails, requeues, fails again. Forever.

A dead letter queue (DLQ) catches these. You configure a max receive count, say 3. After three failed attempts, the broker moves the message to a separate DLQ instead of requeuing it. Your main queue stays healthy. You check the DLQ during working hours, figure out what went wrong, fix the bug or data issue, and replay the messages. No one got paged at 2am over a malformed payload.

Now, the metric that actually matters: queue depth. Specifically, the number of messages waiting to be processed. In SQS, that's ApproximateNumberOfMessagesVisible.

If depth is growing, your consumers can't keep up. Maybe one crashed. Maybe traffic spiked. Maybe a downstream service is slow. Whatever the cause, a growing queue is the canary. Set an alarm on it. Auto-scale your consumers based on it. A flat or zero depth means things are healthy.

Quick note on brokers: RabbitMQ gives you routing, priorities, and complex topologies. SQS gives you zero ops and infinite scale. BullMQ is great for Node.js apps already running Redis. Kafka is a distributed commit log designed for high-throughput streaming — it's a different animal entirely, and I wrote about how its partitions and consumer groups work separately. Pick based on what you're already running.

More writing

The rest of my writing lives at arnavsharma.dev, if this was useful.

Kafka vs RabbitMQ: Which Message Broker Should You Actually Pick?

Arnav Sharma — Fri, 14 Aug 2026 10:12:53 +0000

Kafka vs RabbitMQ: Which Message Broker Should You Actually Pick?

Why is every "Kafka vs RabbitMQ" article just a feature table and some made-up throughput numbers?

You don't pick a broker from a spreadsheet. You pick it because one of them matches how your data actually flows and the other one fights you the entire time. The difference isn't speed or popularity. It's architecture. And once you see it, the rest of the decision becomes obvious.

🧠 One is a log, the other is a router

Kafka is a distributed append-only commit log. Producers write messages to the end of a partition. That's it. Messages sit there, indexed by offset, until a retention policy (time or size) eventually cleans them up. Nobody deletes them when they're "done."

Consumers pull. They track their own position in the log by committing offsets back to Kafka. The broker doesn't know or care whether you've processed a message. It just stores segments and serves reads.

RabbitMQ is the opposite. It's a smart broker. Messages arrive at an exchange, get routed through bindings to one or more queues, and the broker pushes them to consumers. When a consumer sends an ack, the message is gone. Deleted. The broker owns delivery state.

So Kafka says: "here's the log, read wherever you want." RabbitMQ says: "tell me where things should go and I'll deliver them." Everything else follows from this split.

⚡ RabbitMQ's exchange model

Kafka has topics and partitions. You publish to a topic, and the partition key decides which partition it lands in. Simple. But there's no broker-side routing logic. Consumers get everything on their assigned partitions and filter client-side if needed.

RabbitMQ gives you four exchange types, each with different routing rules:

Exchange type	What it does
Direct	Routes to queues whose binding key exactly matches the message's routing key
Topic	Wildcard matching on routing key patterns (`order.*.created`, `#.error`)
Fanout	Broadcasts to every bound queue. No filtering.
Headers	Matches on message header attributes instead of routing key

This means the broker itself decides who gets what. You can have one publisher sending order events and the broker splits them across an invoice queue, a notification queue, and an analytics queue based on routing keys. No consumer-side logic. No duplicated subscriptions.

If your system needs complex message routing, RabbitMQ handles it natively. With Kafka, you'd either use multiple topics or build filtering into every consumer. Not the end of the world, but it's work you don't have to do with RabbitMQ.

Replay vs delete-on-ack

This is where the architectural choice really bites.

Kafka retains messages. Period. A consumer can rewind to offset zero and reprocess everything from the beginning. Deployed a bug that corrupted downstream data? Reset the consumer group offset and replay. Need a second service to read the same events independently? Just add another consumer group. The data's still there.

I wrote about how consumer groups and partition assignment work in a previous post, so I won't re-explain the mechanics here.

RabbitMQ's classic and quorum queues delete messages on ack. Gone forever. If you need reprocessing, you're out of luck unless you built your own archival system. Dead-letter exchanges catch rejected messages, but that's error handling, not replay.

But here's the thing. RabbitMQ added Streams back in version 3.9. Streams are an append-only replicated log with offset-based consumption. Basically Kafka semantics inside RabbitMQ. They support replay, time-based seeking, and fan-out without re-delivery. So the line is blurring. Still, if replay is your primary use case, Kafka was built for it from day one.

Ordering guarantees

Kafka: Total order within a partition. Messages sharing the same key hash to the same partition, so per-key ordering is strict. Want ordered processing across multiple consumers? That's what consumer groups give you, with each partition going to exactly one consumer.

RabbitMQ: FIFO within a single queue. Clean and simple. But the moment you add competing consumers (multiple consumers on one queue for parallelism), messages get dispatched round-robin and ordering breaks. You either accept that or run single-consumer queues, which limits throughput.

Delivery semantics

Both default to at-least-once delivery. Your consumer might see the same message twice if something crashes mid-processing. But how they handle stronger guarantees differs:

	At-least-once	Effectively-once
Kafka	`acks=all` + retries (default)	Idempotent producer (default since Kafka 3.0) deduplicates retries via sequence numbers. Transactions give atomic read-process-write across partitions.
RabbitMQ	Publisher confirms + consumer ack	No built-in exactly-once. Deduplication is your problem. Quorum queues guarantee replication, but not dedup.

So if you need end-to-end exactly-once (or close to it), Kafka has first-party support. With RabbitMQ you're implementing idempotency yourself. Not impossible, just extra work.

Ops in 2025-2026

A quick update on where both projects stand operationally, because a lot of older comparison posts are outdated:

Kafka 4.0 (March 2025) fully removed ZooKeeper. It's gone. KRaft mode is the only option now. This cuts a huge operational dependency. No more separate ZK cluster to babysit. Easier to deploy, fewer moving parts.

RabbitMQ 4.0 removed classic mirrored queues entirely. Quorum queues (Raft-based replication) are the only replicated queue type going forward. Better consistency, better throughput under replication. If you're reading guides that mention ha-mode policies, they're obsolete.

📌 I'm not quoting throughput numbers

Every comparison post throws around "Kafka does 2 million msgs/sec" or "RabbitMQ maxes out at 50K." I'm not doing that.

Why? Because those numbers mean nothing without context. Message size, persistence settings, replication factor, ack mode, batch size, hardware, network. Change any one variable and the number changes by 10x. A benchmark where Kafka batches 1KB messages with acks=1 and RabbitMQ uses durable quorum queues with per-message confirms isn't a comparison. It's fiction dressed up as data.

Here's what I'll say: Kafka is architecturally optimized for throughput (sequential disk writes, zero-copy with sendfile, batching). RabbitMQ optimizes for per-message routing flexibility and low latency at moderate scale. That's the shape of it. Actual numbers depend entirely on your deployment.

So which one do you pick?

This is the decision guide. Be honest about your actual requirements.

Pick Kafka when you need event streaming, event sourcing, or replay. When you have high throughput requirements (think hundreds of thousands of messages per second). When multiple independent services need to consume the same data stream. Log aggregation, change data capture, CQRS. All Kafka territory.

Pick RabbitMQ when you need complex routing logic at the broker level. When you're doing request-reply patterns or RPC. When you need per-message priority, TTL, or dead-letter handling. When you want polyglot protocol support (AMQP, MQTT, STOMP) without running separate brokers. Moderate scale, task queues, workflow orchestration.

Pick neither when you don't want to operate a message broker at all. If your system has low throughput, simple fan-out, and you're running on AWS anyway, just use SQS. Or SNS+SQS for pub/sub. Zero cluster management, no replication to configure, no disk monitoring. I talked about how API gateways sit in front of these services if you're building event-driven architectures on managed infrastructure.

Honestly, for a lot of teams shipping their first async system, SQS is the right answer. No shame in it. You can always migrate to Kafka or RabbitMQ when you outgrow it. Operating a broker cluster before you need one is just yak shaving.

Where else to find me

My other posts, plus what I'm building right now, are at arnavsharma.dev.

What Is Idempotency? A Practical Guide for API Developers

Arnav Sharma — Fri, 14 Aug 2026 10:11:51 +0000

A user clicks "Pay Now." The request times out. No confirmation screen. So they click again. Totally reasonable thing to do. And now their card has been charged twice.

This isn't some weird edge case. It happens all the time in production. Network blips, client retries, at-least-once message delivery, webhook providers re-firing because they didn't get an ACK fast enough. The same request hits your server more than once, and your code happily processes it each time.

The fix has a name: idempotency.

🛠️ What idempotent actually means

An operation is idempotent if running it multiple times produces the same server-side effect as running it once. That's it. From RFC 9110 §9.2.2.

The response can differ though. A DELETE that returns 200 the first time and 404 the second is still idempotent. The resource is gone either way. Same effect, different status code.

People mix this up with safe. A safe method doesn't change state at all. GET is safe. PUT is not safe (it changes stuff) but it is idempotent (doing the same PUT ten times leaves you in the same state as doing it once).

So: all safe methods are idempotent. Not all idempotent methods are safe.

Here's the HTTP method breakdown per RFC 9110 §9.2.2:

Method	Safe	Idempotent
GET	yes	yes
PUT	no	yes
DELETE	no	yes
POST	no	no

POST is the odd one out. Each call can create a new resource or fire a new side effect. There's nothing in the protocol that prevents it. And that's exactly why payment endpoints (almost always POST) need extra work to become idempotent.

⚡ The idempotency key pattern

Stripe popularized this approach and it's become the industry standard for making POST endpoints safe to retry.

The idea: the client generates a unique key (a UUIDv4 works fine) before sending the request and passes it in an Idempotency-Key header. The server uses that key to detect replays.

On the first request, the server processes normally and stores the key alongside the response. On a retry with the same key and same parameters, it returns the stored response without re-executing anything. No double charge.

But what if someone sends the same key with different parameters? That's a bug on the client side, and the server returns a 409 Conflict. You don't want to silently return a cached response when the request body doesn't match. That would mask real errors.

Stripe expires keys after 24 hours. After that, a new execution happens.

Here's a stripped-down middleware that does this:

async function idempotency(req: Request, res: Response, next: NextFunction) {
  const key = req.headers["idempotency-key"] as string;
  if (!key) return next();

  const fingerprint = hash(req.body);
  const cached = await db.query(
    `SELECT fingerprint, status, body FROM idempotency_keys
     WHERE key = $1 AND created_at > NOW() - INTERVAL '24h'`,
    [key]
  );

  if (cached.rows.length) {
    if (cached.rows[0].fingerprint !== fingerprint) {
      return res.status(409).json({ error: "Key reused with different params" });
    }
    return res.status(cached.rows[0].status).json(JSON.parse(cached.rows[0].body));
  }

  // Wrap res.json to capture the response for storage
  const origJson = res.json.bind(res);
  res.json = (data: any) => {
    db.query(
      `INSERT INTO idempotency_keys (key, fingerprint, status, body)
       VALUES ($1, $2, $3, $4) ON CONFLICT (key) DO NOTHING`,
      [key, fingerprint, res.statusCode, JSON.stringify(data)]
    );
    return origJson(data);
  };
  next();
}

Notice the ON CONFLICT (key) DO NOTHING at the bottom. That's doing the real heavy lifting.

🧠 The database constraint is the actual guard

Here's the mistake I see constantly. People write logic like this:

Check if the key exists (SELECT)
If not, process the request
Insert the key

Looks reasonable. Totally broken in practice.

Two identical requests arrive 5ms apart. Both hit step 1, both see "no key exists," both proceed to step 2. You've just processed the payment twice. This is a classic TOCTOU race — time-of-check to time-of-use.

The fix is to let the database handle atomicity. A UNIQUE constraint on the idempotency key column means only one insert can ever succeed. The second one fails, and your code catches that conflict:

INSERT INTO payments (idempotency_key, amount, customer_id, status)
VALUES ('ord_123_pay', 4999, 'cus_abc', 'completed')
ON CONFLICT (idempotency_key) DO NOTHING;

One statement. Atomic. No race condition. The database's locking mechanism handles concurrent access for you. If you're building anything that processes payments, webhook events, or queue messages, this pattern should be your default.

And yeah, retry strategies (exponential backoff, jitter, circuit breakers) and distributed transactions are their own topics. They work alongside idempotency but don't replace it. Separate posts for those.

If your API sits behind an API gateway, the gateway might handle retries automatically. Another reason your handlers need to be idempotent even when you don't think you're retrying.

More writing

Everything else I've written is over at arnavsharma.dev.

How Database Indexes Work (And Why Yours Might Be Useless)

Arnav Sharma — Thu, 13 Aug 2026 07:59:30 +0000

How database indexes work (and why yours might be useless)

You added an index on customer_id. The query still does a sequential scan. You added another index on status. Same thing. You now have five indexes on a table, writes are slower, and the planner hasn't touched a single one of them.

Why?

Because an index isn't magic. It's a trade. And if you don't understand what you're trading — or when the database decides the trade isn't worth it — you'll keep throwing indexes at problems they can't solve.

🧠 What an index actually stores

Forget the "it makes queries faster" hand-wave. An index is a separate data structure, sorted by your chosen column, where each entry holds two things: the column value (the key) and a row locator that tells the database where the full row lives.

That row locator is where Postgres and MySQL diverge in a way that matters.

In Postgres, tables are heap-organized. Rows sit in an unordered pile. The locator is a ctid, basically a (page number, slot offset) pair. A physical address. The index says "the row with customer_id = 42 is at page 87, slot 3." Direct jump.

InnoDB does it differently. The table itself is a B-tree, organized by primary key. They call this a clustered index. Secondary indexes don't store a physical address. They store the primary key value. So when you look up customer_id = 42 through a secondary index, InnoDB finds the PK in the index leaf, then descends the clustered index B-tree a second time to reach the actual row.

Two lookups instead of one. That second descent is cheap for a single row. It gets expensive in bulk.

🔑 The two-step lookup and when sequential scan wins

Here's the thing people miss. Every index lookup that returns a full row is a two-step process:

Walk the index structure to find the matching entries.
For each entry, follow the locator to fetch the full row from the table.

Step 1 is fast. Sorted structure, logarithmic depth. Step 2 is the problem. Each row fetch is a random I/O to a potentially different page on disk. Fetch 10 rows, you might hit 10 different pages. Fetch 10,000 rows, that's potentially 10,000 random page reads scattered across the table.

A sequential scan, by contrast, reads pages in order. One continuous stream. Sequential I/O is dramatically faster than random I/O, even on SSDs.

So the database planner does math. If your query returns a small fraction of the table, say 50 rows out of a million, the index wins easily. But as that fraction grows, random I/O piles up until a single sequential pass through the whole table is genuinely cheaper.

Where's the crossover? Roughly 5-15% of the table, depending on row width, storage speed, and planner settings like random_page_cost. Not a fixed number. A rule of thumb. But it means an index on a column where most queries match 20% of rows is dead weight.

-- This query returns ~30% of a million-row table.
-- The planner will almost certainly ignore your index on status.
SELECT * FROM orders WHERE status = 'completed';

⚡ Selectivity, cardinality, and the column that never gets indexed

Selectivity is the fraction of rows a predicate matches. Low selectivity (few rows match) means the index pays off. High selectivity (many rows match) means it doesn't.

Cardinality is how many distinct values a column has. A boolean column has cardinality 2. An email column might have cardinality in the millions.

Low cardinality columns are the classic trap. You index is_active (true/false). Half the table is true. The planner won't use that index for WHERE is_active = true because fetching 500,000 rows via random I/O is worse than scanning the whole million-row table sequentially.

Same story with a status column holding three values. Each value matches roughly 33% of rows. The index exists, takes up space, slows every write, and the planner ignores it.

But here's a trick. If you only ever query the rare value:

-- Partial index: only indexes rows where status = 'pending'
-- (maybe 2% of the table)
CREATE INDEX idx_orders_pending ON orders (created_at)
WHERE status = 'pending';

-- The planner will happily use this for:
SELECT * FROM orders
WHERE status = 'pending' AND created_at > NOW() - INTERVAL '7 days';

Smaller index, higher selectivity, actually gets used. And cheaper to maintain because it only tracks the rows that match the predicate.

Things that silently prevent index usage

Even with good selectivity, the planner might still ignore your index. Common blockers:

Function wrapping the column. WHERE LOWER(email) = 'bob@test.com' can't use a plain index on email. The index is sorted by raw values, not lowercased ones. Fix: expression index. CREATE INDEX ON users (LOWER(email)).

Type mismatch. WHERE int_column = '42' forces a cast. The planner can't match the index. Fix: use the right literal type.

Leading wildcard. WHERE name LIKE '%smith' needs to scan every entry because the prefix is unknown. Fix: a trigram index with pg_trgm.

OR across different columns. WHERE a = 1 OR b = 2 can't walk a single index cleanly. Fix: separate indexes on each column (Postgres can bitmap-OR them), or restructure as a UNION.

The write cost nobody talks about

Every index you add is a promise: "I will maintain this sorted structure on every single write."

INSERT a row? The database updates every index on that table. UPDATE an indexed column? Old entry removed, new entry inserted. DELETE a row? Every index gets cleaned.

That's per-write, per-index overhead. A table with 8 indexes means every INSERT does 8 additional B-tree modifications. And if you have expression indexes, the database recomputes the expression on each write too.

Then there's bloat. Postgres uses MVCC, so old row versions stick around until VACUUM cleans them up. But those dead tuples also exist in the index. On a heavily-updated table, indexes can bloat to 2-5x their ideal size. VACUUM reclaims the space inside the index, but the file on disk doesn't shrink. You end up with a 4GB index that's half dead entries, slowing scans through the index structure itself.

So every index is a bet. You're betting that the read performance gain outweighs the write cost and the maintenance burden. For a read-heavy table with highly selective queries, that bet pays off. For a write-heavy table where the indexed column has low cardinality? You're paying the cost with zero benefit.

-- Check index usage: are your indexes actually being used?
SELECT indexrelname, idx_scan, idx_tup_read
FROM pg_stat_user_indexes
WHERE schemaname = 'public'
ORDER BY idx_scan ASC;
-- Indexes with idx_scan = 0 are candidates for removal.

What this post doesn't cover

The internal node layout of B+ trees (fanout, splits, how height stays low) gets its own post. Same for multi-column index ordering strategy and index-only scans where the database never touches the table at all. Those are different problems with different mental models.

If you're building systems that route traffic between services, the trade-off thinking here is similar to decisions you'd make at the API gateway layer. And if you want another example of hidden costs in layered systems, the write amplification story in Docker image layers rhymes with index bloat more than you'd expect.

📌 Key takeaways

An index stores the key plus a row locator. In Postgres that's a ctid (physical address). In InnoDB it's the primary key, requiring a second tree descent.
The second fetch (random I/O per row) is why the planner ignores your index past roughly 5-15% of the table. Not a fixed threshold, but a useful mental benchmark.
Low-cardinality columns (booleans, status fields) produce indexes the planner will never use. Partial indexes targeting the rare value are the fix.
Every index costs you on writes: maintenance overhead, expression recomputation, and MVCC-driven bloat in Postgres.
Functions on columns, type mismatches, leading wildcards, and OR across columns all silently block index usage. Each has a specific fix.

Where else to find me

Plenty more posts at arnavsharma.dev if this helped.

Exactly-Once vs At-Least-Once: What Kafka's Guarantee Actually Covers

Arnav Sharma — Thu, 13 Aug 2026 07:57:57 +0000

Exactly-once delivery is a lie: what your broker actually guarantees

Your message broker's docs say "exactly-once semantics." You read that, nod, and assume your consumer handler runs exactly one time per message. Ship it.

Then you get duplicate charges in production. Or a user's order gets created twice. And you're staring at the logs thinking: I thought this was supposed to be exactly-once?

Here's the thing. The broker isn't lying to you. But it's not promising what you think it's promising. And the gap between what "exactly-once" means on the marketing page and what it means in the protocol spec is where your bugs live.

⚡ The three delivery semantics

Before we untangle the confusion, let's get precise about what the three guarantees actually mean. They look different depending on whether you're the producer or the consumer.

At-most-once. Producer fires a message and doesn't retry on failure. If the network drops it, it's gone. On the consumer side, you commit your offset before processing. If your process crashes after committing but before finishing work, that message is skipped forever. No duplicates, but you lose data.

At-least-once. Producer retries until it gets an acknowledgment. If the ack gets lost but the broker already wrote the message, you get a duplicate in the log. Consumer side: you process the message first, then commit the offset. Crash after processing but before committing? You'll reprocess that message on restart. Duplicates happen. Guaranteed.

Exactly-once. Each message appears in the log once and its effect is applied once. Sounds perfect. But there's a catch.

Why exactly-once delivery is impossible

Think about what happens at the network level. Your producer sends message M to the broker. The broker writes it. The broker sends an ACK back. The ACK gets lost.

Now your producer is stuck. It can't tell the difference between "the broker never received M" and "the broker received M but the ACK didn't make it back." Two completely different situations, identical from the producer's perspective.

So it has to choose. Retry and risk a duplicate? Or don't retry and risk losing the message? There's no third option. This is the Two Generals Problem (1975): no finite protocol can guarantee two parties reach agreement over an unreliable channel. The sender can never know its last message arrived.

Not a Kafka limitation. Not a broker limitation. A mathematical impossibility for any system communicating over a network that can drop packets.

So when someone says "exactly-once delivery," they're either wrong or they're talking about something else.

🎯 Exactly-once processing: the achievable goal

Here's the pivot. You can't guarantee a message is delivered exactly once. But you can guarantee its effect is applied exactly once. Different thing entirely.

The industry sometimes calls this "effectively-once." Your consumer might receive the same message three times. Doesn't matter, as long as the business result only happens once. Two ways to get there:

Idempotent processing. Design your writes so that applying the same operation twice produces the same result. An upsert keyed on a deterministic message ID. A dedup check before inserting. If the same message shows up again, the second write is a no-op.

Atomic offset + effect commit. In a single database transaction, write your business result AND advance your consumer's offset/cursor. On restart, you read the last committed offset from your DB. You might reprocess a message, but the dedup check inside the transaction rejects it.

Here's what that looks like in practice. An at-least-once consumer with a PostgreSQL dedup table:

async function handleMessage(msg: { id: string; payload: unknown }, pool: Pool) {
  const client = await pool.connect();
  try {
    await client.query("BEGIN");
    // Insert message ID — ON CONFLICT means we've seen this before
    const { rowCount } = await client.query(
      `INSERT INTO processed_messages (message_id) VALUES ($1) ON CONFLICT DO NOTHING`,
      [msg.id]
    );
    if (rowCount === 0) {
      await client.query("ROLLBACK");
      return; // duplicate, skip it
    }
    // Business logic: only runs once per message ID
    await client.query(
      `INSERT INTO orders (id, data) VALUES ($1, $2)`,
      [msg.id, JSON.stringify(msg.payload)]
    );
    await client.query("COMMIT");
  } catch (e) {
    await client.query("ROLLBACK");
    throw e;
  } finally {
    client.release();
  }
  // ACK to broker ONLY after commit succeeds
}

The consumer uses at-least-once delivery from the broker. Messages might arrive more than once. But the ON CONFLICT DO NOTHING clause on processed_messages means the business write only happens once. Exactly-once processing on top of at-least-once delivery.

What Kafka's "exactly-once" actually covers

Kafka does offer exactly-once semantics. But with a hard boundary.

The idempotent producer (enable.idempotence=true, default since Kafka 3.0) assigns each producer a PID and attaches monotonic sequence numbers per partition. The broker deduplicates by rejecting messages with a sequence number it's already seen. Scope: one producer session, one partition.

Transactions (transactional.id) go further. They atomically write to multiple partitions and commit consumer offsets in a single operation. Consumers with isolation.level=read_committed only see committed messages. Zombie producers with the same transactional ID get fenced off.

Here's a transactional produce-and-commit using KafkaJS:

const producer = kafka.producer({
  transactionalId: "my-app-topicA-0",
  maxInFlightRequests: 1,
  idempotent: true,
});
await producer.connect();

const txn = await producer.transaction();
try {
  await txn.send({ topic: "output", messages: [{ value: "result" }] });
  await txn.sendOffsets({
    consumerGroupId: "my-group",
    topics: [{ topic: "input", partitions: [{ partition: 0, offset: "42" }] }],
  });
  await txn.commit();
} catch (e) {
  await txn.abort();
  throw e;
}

This gives you exactly-once from input topic to output topic. Read a message, produce a result, commit the offset, all atomically. Powerful stuff.

But here's the boundary everyone misses. This only works when both source and sink are Kafka topics. The moment your consumer writes to an external database, calls an HTTP API, sends an email, anything outside Kafka's transaction fence, you're back to at-least-once. The transaction can't wrap your PostgreSQL insert or your Stripe API call.

So yes, Kafka has exactly-once. For Kafka-to-Kafka pipelines. For everything else, you need the patterns from the previous section.

The practical playbook

You've got a consumer that writes to a database. Messages will arrive more than once. Here's what actually works:

1. Dedup table. Store (message_id, processed_at) in the same transaction as your business write. Reject on conflict. Simplest pattern, works everywhere.

2. Idempotent upserts. If your write is naturally idempotent (setting a user's email to a value, not incrementing a counter), just use ON CONFLICT DO UPDATE or equivalent. No separate dedup table needed.

3. Atomic offset commit. Store the consumer offset in your application database, not in Kafka's __consumer_offsets topic. On restart, seek to the stored offset. Combined with a dedup check, this closes the gap completely.

4. Dedup windows. Keep a TTL-bounded set of seen message IDs in Redis or memory. Cheaper than a DB check, but only works within the window. Messages replayed after the TTL expires will be processed again. Partial fix.

And honestly? Sometimes at-most-once is the right call. Metrics counters where a missing data point is fine but a duplicate inflates your graphs. Fire-and-forget telemetry. Log shipping. Not everything needs exactly-once processing.

If you're running Kafka with consumer groups, the rebalancing protocol already causes redelivery on partition reassignment. Your consumers need to handle duplicates regardless of what guarantee you think you've configured.

📌 Key takeaways

Exactly-once delivery is impossible over an unreliable network. The Two Generals Problem proves it. Your broker can't fix physics.
Exactly-once processing is achievable. Idempotent handlers + dedup tables turn at-least-once delivery into effectively-once effects.
Kafka's exactly-once covers Kafka-to-Kafka only. External writes (databases, APIs, emails) fall outside the transaction boundary.
Design for at-least-once. Assume every message arrives more than once and make your handlers safe for it.
The dedup table pattern works everywhere. Single transaction, message ID check, business write. Hard to mess up.

For how Kafka's partitioning and consumer groups handle parallelism and rebalancing, see Kafka partitions and consumer groups. And if you're routing traffic to your consumers through an API gateway, the retry behavior at the gateway layer adds another source of duplicates to plan for.

More writing

If this was useful, there's more where it came from at arnavsharma.dev.

Event-Driven Architecture Explained: Events, Commands, and the Tradeoffs Nobody Mentions

Arnav Sharma — Thu, 13 Aug 2026 07:56:56 +0000

Event-driven architecture: what happens after "payment succeeded"

Your checkout handler used to do two things. Charge the card, save the order. Ship it.

Then product asked for a confirmation email. Fine, three things. Then analytics wanted a purchase event. Then the warehouse needed a stock reservation. Then the loyalty team added points accrual. Then fraud detection wanted a copy. Then someone said "we should notify the seller too."

Now your handler does seven things after payment succeeds, and if the email service is slow, the customer stares at a spinner while all seven finish in sequence. One fails? The whole request blows up. Not great.

This is the exact pain point that pushes teams toward event-driven architecture. Instead of one handler calling seven services synchronously, you announce "hey, payment succeeded" and let each service react on its own.

Events vs commands

First distinction that matters: an event is a fact about something that already happened. "OrderPlaced." Immutable. Past tense. The producer doesn't know or care who's listening.

A command is an imperative. "PlaceOrder." It's directed at a specific service and expects a result.

The difference sounds academic until you realize it changes coupling entirely. A command ties you to the receiver. An event doesn't. Your checkout service publishes "PaymentSucceeded" and moves on. Whether three services or thirty react to it, the checkout service doesn't change.

And the envelope carrying either one? That's just a message. The broker (Kafka, SQS, EventBridge, whatever) routes messages. It doesn't care about the semantics inside.

Three flavors of event

Martin Fowler identified four patterns that people lump under "event-driven." Three of them are actually about events, and they solve different problems.

Event notification is the thin version. The event carries just an ID and a type: { "type": "OrderPlaced", "orderId": "abc-123" }. Receivers call back to the source if they need details. Low coupling, but chatty.

Event-carried state transfer is the fat version. The event includes the full state delta: order total, line items, shipping address. Receivers cache this locally and never call back. Great for resilience. But now every consumer is coupled to your schema.

Event sourcing goes further. You don't store current state at all. You store every event that ever happened, in order, and rebuild state by replaying them. Think git commits. You can reconstruct any past state, get a full audit trail, and answer "what did this look like last Tuesday?" The cost is complexity and storage.

Which flavor you pick depends on the problem. Most teams start with notification, move to state transfer when the callback traffic gets annoying, and reach for event sourcing only when audit or temporal queries are a hard requirement.

⚡ Choreography vs orchestration

Once events flow, you need a topology. Two options.

Choreography is decentralized. Services react to events and emit their own events. Nobody's in charge. OrderPlaced triggers PaymentService, which emits PaymentConfirmed, which triggers InventoryService, which emits StockReserved. It's like a dance where everyone knows their part.

// Each service just listens and reacts
import { SNSClient, PublishCommand } from "@aws-sdk/client-sns";

const sns = new SNSClient({});

async function onPaymentConfirmed(event: PaymentEvent) {
  const reservation = await reserveStock(event.data.orderId);

  // announce what happened, move on
  await sns.send(new PublishCommand({
    TopicArn: process.env.INVENTORY_TOPIC_ARN,
    Message: JSON.stringify({
      eventId: crypto.randomUUID(),
      type: "StockReserved",
      data: { orderId: event.data.orderId, warehouseId: reservation.warehouseId },
    }),
  }));
}

Simple fanouts? Choreography works beautifully. But the flow is implicit. When something breaks, figuring out what happened means tracing events across five services with correlation IDs. Painful.

Orchestration puts a central coordinator in charge. Step Functions, Temporal, Conductor. The orchestrator calls each step, handles retries, and owns the rollback logic. You get a visible workflow graph and explicit error handling. But now that coordinator is a single point of logic (though not necessarily a single point of failure).

So: choreography for simple fan-outs with few steps. Orchestration for complex multi-step flows where ordering matters or where you need compensation logic. Most real systems use both. The order fan-out is choreographed, but the payment+inventory+shipping sequence might be orchestrated.

🎯 What actually gets harder

I'm not going to pretend this is all upside. Event-driven architecture trades one set of problems for another.

Debugging causality. There's no request trace anymore. A customer says "my order didn't go through" and you're grepping across six services for a correlation ID. You need distributed tracing (OpenTelemetry) and correlation IDs on every event from day one. Retrofitting this is miserable.

Eventual consistency the user can see. "I placed an order but it's not showing up." The event hasn't propagated yet. Your UI needs to handle this honestly — optimistic updates, polling, or just telling the user "processing, check back in a moment." Pretending it's instantaneous will create support tickets.

Testing gets weird. Async delivery, non-deterministic ordering, retries firing at random intervals. Integration tests for event-driven flows are harder to write and flakier to maintain. Contract tests on event schemas help, but they're not a substitute for end-to-end verification.

Monitoring is different. Consumer lag, dead letter queue depth, processing latency. These are your new first-class metrics. If you don't watch them, you'll find out about problems from angry users instead of dashboards.

🛠️ The dual-write problem

Here's a trap that bites almost every team the first time. Your service needs to save an order to the database AND publish an "OrderPlaced" event. Two writes to two different systems.

What if the DB write succeeds but the publish fails? You've got an order with no event. Downstream services never find out. What if you publish first and the DB write fails? You've announced something that didn't actually happen.

This is the dual-write problem, and "just retry" doesn't fix it. You can't get atomicity across a database and a message broker without some pattern.

The standard fix is the transactional outbox. Instead of publishing directly, you write the event to an outbox table in the same database transaction as your business data. One atomic write. A separate relay process polls the outbox (or uses CDC to tail the transaction log) and publishes events to the broker.

// Same DB transaction = atomic
async function placeOrder(order: Order, db: Pool) {
  const client = await db.connect();
  try {
    await client.query("BEGIN");
    await client.query(
      "INSERT INTO orders (id, user_id, total, status) VALUES ($1, $2, $3, $4)",
      [order.id, order.userId, order.total, "confirmed"]
    );
    // event goes in the same transaction
    await client.query(
      "INSERT INTO outbox (event_id, event_type, payload) VALUES ($1, $2, $3)",
      [crypto.randomUUID(), "OrderPlaced", JSON.stringify(order)]
    );
    await client.query("COMMIT");
  } catch (err) {
    await client.query("ROLLBACK");
    throw err;
  } finally {
    client.release();
  }
}

Both rows land or neither does. The relay picks up new outbox rows and publishes them. If the relay fails, it retries from where it left off. Consumers still need to be idempotent (since at-least-once delivery means duplicates are normal), but that's a topic for a dedicated post on idempotency patterns.

Scope and where to go from here

I've intentionally kept this post at the conceptual level. There are rabbit holes everywhere.

Message queues (Kafka, RabbitMQ, SQS) each have different delivery guarantees and ordering behavior. Choosing between them matters. Idempotent consumers need a deduplication strategy. Retries need backoff and dead letter queues. And when multiple services need to coordinate rollbacks, you're in saga territory — a pattern that's been around since Garcia-Molina and Salem named it in 1987.

Each of those deserves its own post. For now, the mental model is what matters: events decouple, topology shapes your failure modes, and the dual-write problem will bite you if you don't address it upfront.

If you're already using Kafka and want to understand how partitions and consumer groups affect your event flow, I wrote about that in depth here.

📌 Takeaways

Events are facts (past tense, immutable). Commands are directives. Don't confuse them.
Pick the right event flavor: notification for low coupling, state transfer for resilience, sourcing for audit trails
Choreography works for simple fan-outs. Orchestration wins for complex multi-step flows with rollback needs.
The dual-write problem is real. Use a transactional outbox.
Eventual consistency isn't a bug to hide. Design your UI around it.

More from me

More posts on distributed systems and backend architecture at arnavsharma.dev.

The Dual-Write Problem and How to Actually Fix It

Arnav Sharma — Thu, 13 Aug 2026 07:55:44 +0000

Distributed transactions: why BEGIN/COMMIT can't save you across services

Someone splits the monolith into services. Yesterday, orders, payments, and inventory all lived in one database. You wrapped everything in a transaction, committed, done. ACID handled the rest.

Now the order lives in one database, the payment in another, inventory in a third. You still write BEGIN and COMMIT. But there's no shared transaction manager anymore. That COMMIT only applies to the database you're talking to right now. The other two? They don't know. They don't care.

This is where things get ugly.

🛠️ The dual-write problem

Here's the situation. Your Order Service needs to do two things when an order is created: write to its database, and publish an event so downstream services (payment, inventory, notifications) can react. Two systems. Two writes. No shared transaction between them.

Three ways this blows up:

DB commits, event publish fails. Your database has the order. But the message broker never got the event. Payment never gets charged. Inventory never gets reserved. The systems silently diverge and nobody notices until a customer complains.

Event publishes, DB rolls back. The broker accepted your event. Downstream services start processing. But your database transaction failed. Now payment is charging for an order that doesn't exist.

Process crashes between the two writes. One write landed. Which one? Depends on the order you wrote them. Either way, you're inconsistent.

No retry logic fixes this. Retrying the publish doesn't help if the DB already rolled back. Retrying the DB write doesn't help if the event already fired. The problem isn't failure handling. The problem is that two independent systems can't commit atomically.

🧠 Two-phase commit: the textbook answer nobody uses

The academic solution is two-phase commit (2PC). A coordinator asks every participant "can you commit?" in Phase 1. Each participant acquires locks, writes to durable storage, and votes YES or NO. In Phase 2, if everyone voted YES, the coordinator sends COMMIT. Otherwise, ABORT.

Sounds clean. In practice, teams avoid it for good reasons.

It blocks. If the coordinator crashes between phases, every participant that voted YES holds its locks indefinitely. They can't commit (they don't know the decision). They can't abort (maybe the coordinator will come back and say COMMIT). They just wait. With locks held. Blocking every other transaction that touches those rows.

It requires XA. Every participant needs to implement the XA interface (prepare, commit, rollback). Your PostgreSQL supports it. Your message broker probably doesn't. Your managed cloud datastore? Almost certainly not. So you can't even use 2PC across the systems you actually have.

It kills availability. All participants must be online simultaneously. One slow service and the entire transaction is stuck. In a distributed system where partial failures are the norm, this is a non-starter.

So 2PC works in theory and in tightly controlled environments (like a single vendor database cluster). But for services communicating over a network? Not practical. 3PC exists as an academic improvement that adds a pre-commit phase to reduce blocking, but it still can't handle network partitions and almost nobody implements it.

⚡ The transactional outbox

Here's the trick. You can't atomically write to a database and a message broker. But you can atomically write to a database twice. Same database, same transaction.

Instead of publishing the event directly, you write it to an outbox table inside the same transaction as your business data:

BEGIN TRANSACTION;

  INSERT INTO orders (id, customer_id, status)
  VALUES ('ord-123', 'cust-1', 'PENDING');

  INSERT INTO outbox (id, aggregate_id, event_type, payload, created_at)
  VALUES (gen_random_uuid(), 'ord-123', 'OrderCreated',
          '{"orderId":"ord-123","customerId":"cust-1"}', NOW());

COMMIT;

One transaction. Both writes succeed or both fail. No dual-write problem.

A separate relay process picks up committed outbox rows and publishes them to your broker, Kafka, SQS, whatever you're using. The relay can poll the table on an interval, or you can use change data capture (CDC) to tail the database's transaction log directly.

If the relay crashes mid-publish, it restarts and re-publishes. This means consumers might see the same event twice. That's fine. They need to be idempotent anyway (a topic for its own post).

Simple pattern. Genuinely reliable. And it works with whatever broker you already have.

Sagas and compensating actions

The outbox solves "write + publish" atomicity. But what about operations that actually span multiple services? An order that needs to reserve inventory and charge payment and confirm the order?

That's where sagas come in. A saga is a sequence of local transactions, each in its own service. Each step publishes an event or command that triggers the next step. No global transaction. No distributed locks.

CreateOrder saga (orchestrated):

1. OrderService.createOrder(PENDING)
   → on failure: OrderService.rejectOrder()

2. PaymentService.authorizePayment()
   → on failure: PaymentService.refundPayment()

3. InventoryService.reserveStock()
   → on failure: InventoryService.releaseReservation()

4. OrderService.confirmOrder(CONFIRMED)
   → terminal step, no compensation needed

If Step 3 fails:
  → run compensate(Step 2): refund the payment
  → run compensate(Step 1): reject the order

Two flavors exist. Choreography: services react to each other's events with no coordinator. Works well for 2-3 steps, gets tangled fast after that. Orchestration: a central saga orchestrator sends commands and decides what to do next. Easier to trace and reason about.

But here's the thing people miss about sagas. Compensation is not rollback.

A database rollback erases the write. It never happened. A compensating action is a new forward operation. A refund is not an un-charge — the money moved to the merchant, now it moves back. There are processing fees, accounting entries, customer notifications. It takes time. The original charge still shows up in logs.

You have to design compensating actions explicitly for every step. And they have their own failure modes. What if the refund fails? Now you need retry logic for your compensations too.

I'll write a full saga implementation walkthrough in a future post. For now, the key insight: sagas give you eventual consistency across services, but they trade the simplicity of ACID for a lot of explicit failure handling.

The best first move is often the simplest

Before you reach for outbox patterns or saga orchestrators, ask yourself: does this operation actually need to span multiple services?

If Order Service and Payment Service always transact together, maybe they shouldn't be separate services. Maybe you split too early. A single database transaction is simpler, faster, and more reliable than any distributed pattern.

Redraw your service boundaries so the transaction stays local. That's not a failure of architecture. That's good design.

When you genuinely need cross-service coordination:

Outbox pattern for reliable event publishing from a single service
Saga when the operation truly spans multiple autonomous services with independent datastores

And whatever pattern you pick, idempotent consumers aren't optional. Every message will be delivered at least once. Design for it.

📌 Quick reference

Dual-write problem: you can't atomically write to two systems without a shared transaction
2PC: blocks on coordinator failure, requires XA, reduces availability. Avoid for service-to-service coordination
Transactional outbox: write the event to the same DB transaction as the business data; relay publishes it later
Sagas: sequence of local transactions with compensating actions on failure. Compensation is a new operation, not a rollback
Best first move: keep the transaction local by redrawing service boundaries

If you're working with event-driven services, the post on Kafka partitions and consumer groups covers how the messaging layer handles ordering and parallelism. And for routing requests to the right service in the first place, see API gateways.

Where else to find me

My writing and side projects all live at arnavsharma.dev.

CQRS Explained: It's Just Two Models (Not a Whole Architecture)

Arnav Sharma — Thu, 13 Aug 2026 07:54:21 +0000

CQRS explained: it's just two models (not a whole architecture)

Why does a pattern whose entire content is "use a different model for reads than for writes" end up meaning event sourcing, two databases, and a message bus in every conference talk? How did we get from "split your objects" to an architecture diagram with fifteen boxes?

I think we cargo-culted it. Someone saw a Greg Young talk where CQRS happened to sit next to event sourcing, and the two got welded together in collective memory. The actual pattern is almost disappointingly simple.

Let me un-weld them.

🔑 The minimal claim

Bertrand Meyer gave us Command-Query Separation in 1988. Method-level rule: a method either changes state (command) or returns data (query). Never both.

Greg Young extended that to the model level in 2010 and called it CQRS. His definition: "simply the creation of two objects where there was previously only one." You have a write model that handles commands and enforces business rules. You have a read model shaped for queries. That's it.

Not two databases. Not a message bus. Not eventual consistency. Young said this explicitly: "CQRS is not eventual consistency, it is not eventing, it is not messaging, it is not having separated models for reading and writing, nor is it using event sourcing."

So what is it? Separation at the model boundary. Your write side validates and persists. Your read side queries and returns. They don't share the same object or the same shape.

🧠 Four things CQRS is not

Let me kill these early because they're the reason people over-architect their first attempt.

It doesn't require event sourcing. Event sourcing is a persistence strategy where you store every state change as an event. CQRS works with a plain Postgres table. They pair well together, but neither requires the other. You can do CQRS without events. You can do event sourcing without separating read/write models.

It doesn't require a second database. Both models can live in the same database. Different tables, different views, even just different queries on the same table. A separate read store is one point on the spectrum, not the definition.

It doesn't require a message bus. You can call the write model, wait for it to finish, then query the read model synchronously. No Kafka, no RabbitMQ, no SQS. Messaging is an infrastructure choice you make when you need async decoupling, not a prerequisite for CQRS.

It's not a system-wide architecture. Fowler is blunt about this: "CQRS should only be used on specific portions of a system (a Bounded Context in DDD lingo) and not the system as a whole." Udi Dahan agrees. You apply it to the bounded context where read/write shapes have genuinely diverged. Your user settings page probably doesn't need it.

The implementation spectrum

Here's where it gets practical. CQRS exists on a spectrum from cheap to expensive.

Tier 1: same model, separate methods. You split your service into command handlers and query handlers. They might even hit the same database table. The separation is in your code organization, not your infrastructure. Cost: almost nothing.

// Command handler: validates, mutates, persists
async function makeCustomerPreferred(customerId: string, repo: CustomerRepo) {
  const customer = await repo.load(customerId);
  if (customer.orderCount < 10) {
    throw new Error("Customer doesn't qualify for preferred status");
  }
  customer.status = "preferred";
  customer.preferredSince = new Date();
  await repo.save(customer);
}

// Query handler: reads a flat view, zero domain logic
async function getPreferredCustomers(db: Pool) {
  return db.query<{ id: string; name: string; since: Date }>(
    "SELECT id, name, preferred_since FROM customers WHERE status = 'preferred'"
  );
}

Tier 2: separate DTOs. The write side uses rich domain entities. The read side uses flat DTOs shaped for the screen. Same database, but the query handler doesn't hydrate your full domain model just to render a list.

Tier 3: separate read models, same database. You add a materialized view or a denormalized table that's optimized for reads. The write side updates the canonical tables; a trigger or background job refreshes the read view. Still one database. But now your read queries are fast without N+1 problems or complex joins.

Tier 4: separate read database, async sync. Dedicated read store (Elasticsearch for search, Redis for hot data, a read replica shaped differently from the primary). Kept in sync via domain events or CDC. This is where eventual consistency enters the picture. And this is where most of the complexity lives.

Most teams should stop at tier 2 or 3. Tier 4 earns its cost only when you have genuine read/write asymmetry (thousands of reads per write) or when your read shapes are so different from your write schema that joins become the bottleneck.

⚡ The eventual-consistency cost

Once you cross into tier 4, a user can write something and immediately read stale data. The projection hasn't caught up yet. "I updated my profile but it still shows the old name." Sound familiar?

This is the read-your-own-writes problem, and you can't wish it away. But people have solved it repeatedly. Four approaches that actually work in production:

Route to primary after write. After a user mutates data, pin their reads to the write-side database for a few seconds (session flag or cookie). Everyone else reads from the eventual-consistent projection. Simple. Works.

Version tokens. The write returns a version number or logical timestamp. The client sends it on the next read. The routing layer picks a replica that's at or past that version. If none qualifies yet, it falls back to primary.

Optimistic UI merge. The client knows what it just wrote. It merges that pending state into whatever the read model returns until the projection catches up. React Query's optimisticUpdate is basically this.

Read from write model for "own" data. A user's own profile or cart reads directly from the write model. Other users' views of that data come from the projection. You're routing by ownership, not by time.

None of these are free. Each adds code paths and edge cases. But they're well-understood patterns, not research problems.

When to actually use it (and my default answer)

Fowler's warning is worth quoting: "you should be very cautious about using CQRS … the majority of cases I've run into have not been so good, with CQRS seen as a significant force for getting a software system into serious difficulties."

Strong words from someone who usually hedges.

My position: the default answer is don't. Start with a single model. If your read shapes are almost identical to your write shapes — which they are in most CRUD apps — separated models just double your code for no benefit.

CQRS earns its complexity when:

You have genuinely different read and write shapes (a dashboard aggregating data from multiple write models)
Read/write traffic is wildly asymmetric (analytics dashboards hit 10,000x more than the admin panel writing data)
Write-side invariants are complex enough that polluting the domain model with display concerns makes it worse
You're already in an event-driven system where projecting events into read models is natural

That last point is where CQRS pairs with the event-driven patterns I'll cover in a dedicated post on event-driven architecture. If your system already publishes domain events, building read projections from those events is a small incremental step. But adopting events and CQRS and a message bus all at once for a CRUD app? That's resume-driven development.

If you're working with API gateways that route between services, CQRS might make sense at the service boundary. And if you're already running Kafka with partitioned consumers, projecting events into a read store is straightforward. But those are preconditions, not reasons to adopt CQRS from scratch.

📌 Key takeaways

CQRS is two models. One for writes, one for reads. That's the whole pattern.
It doesn't require event sourcing, a second database, or a message bus. Those are infrastructure choices you might make, not prerequisites.
The spectrum runs from "separate methods on one class" to "fully async separate databases." Most apps should stay on the cheap end.
Eventual consistency is only a problem at tier 4. If you're there, use route-to-primary, version tokens, or optimistic UI to handle read-your-own-writes.
Default answer: don't use it. Wait until you feel the pain of divergent read/write shapes before you split the model.

More writing

I park all my writing at arnavsharma.dev if you want to read more.

Your Index Only Scan Is Lying: Covering Indexes and the Visibility Map

Arnav Sharma — Thu, 13 Aug 2026 07:53:12 +0000

Covering Indexes Explained: Why "Index Only Scan" Still Hits the Heap

You ran EXPLAIN ANALYZE. You saw Index Only Scan in the output. You assumed the query never touched the table. Case closed, right?

Then you looked one line lower. Heap Fetches: 4,827. Your "index only" scan hit the heap almost five thousand times. Not so index-only after all.

This trips up a lot of developers. The plan node says one thing, the runtime metric says another. I won't rehash what an index is or how B+ trees work (I've touched on index fundamentals before). This post is about the covering trick and the gotcha that makes it unreliable on write-heavy tables.

⚡ What covering means and the fetch it skips

A covering index is one that contains every column a query needs. SELECT columns, WHERE columns, ORDER BY columns. All of them. When the engine can answer entirely from the index, it skips the table (the heap).

That heap visit is expensive. Random I/O. The index gives you a pointer to a row, then the engine jumps to that heap page, pulls the row, extracts the column. Ten thousand rows means ten thousand random fetches.

A covering index kills that step. Everything lives in the index's leaf pages already.

In Postgres 11+, you get the INCLUDE clause for this:

-- Query: SELECT amount FROM orders WHERE customer_id = 42 AND status = 'shipped';
CREATE INDEX idx_orders_cover ON orders (customer_id, status) INCLUDE (amount);

customer_id and status are key columns. They live in both internal and leaf pages, they're searchable, and they determine sort order. Column order still matters here. The leftmost prefix rule applies to keys exactly as it does for any composite index.

amount is a payload column. It's stored only in leaf pages. You can't filter on it. You can't sort by it. It's just along for the ride so the engine doesn't have to go back to the heap. Think of it as a stowaway.

InnoDB doesn't have INCLUDE. You'd tack amount on as a trailing key column. But InnoDB gives you something free: every secondary index already carries the primary key columns in its leaf entries. PK columns are always "covered" without you doing anything.

🔑 The visibility map problem

Here's the part most people miss. Postgres can't actually guarantee a pure index-only scan even with a perfect covering index.

Why? MVCC. Every row has visibility rules: which transactions can see it, whether it's been deleted but not yet vacuumed. That info lives on the heap page, not in the index. So Postgres needs a way to answer "is this tuple visible to my transaction?" without going to the heap.

The answer is the visibility map. A bitmap with one bit per heap page. When VACUUM confirms every tuple on a page is visible to all current transactions, it sets that page's all-visible bit. During an index-only scan, Postgres checks the VM bit:

Bit set → return data from the index. No heap access.
Bit not set → fetch the heap page anyway to check visibility.

And that second case is your Heap Fetches number. Every recently-modified page that VACUUM hasn't caught yet forces a heap visit. On a write-heavy table where pages are constantly dirtied, those bits get cleared faster than VACUUM can set them. Your covering index is technically correct but practically useless.

Index Only Scan using idx_orders_cover on orders
  Index Cond: (customer_id = 42 AND status = 'shipped')
  Heap Fetches: 42
  Buffers: shared hit=15
  Planning Time: 0.08 ms
  Execution Time: 1.2 ms

(illustrative numbers)

Heap Fetches: 0 means you got a true index-only scan. Any positive number means degraded pages forced heap access. The diagnostic is simple: run EXPLAIN ANALYZE, check Heap Fetches. If it's high relative to rows returned, VACUUM isn't keeping up.

But MySQL doesn't have this problem. InnoDB stores row versions inline using undo logs, and secondary indexes point to the clustered index via the PK. No separate visibility check needed. When MySQL's EXPLAIN shows Using index in the Extra column, it really does mean "table not accessed."

Don't confuse that with Using index condition. Different thing entirely:

+----+------+---------------+---------+------+-------------+
| id | type | key           | key_len | rows | Extra       |
+----+------+---------------+---------+------+-------------+
|  1 | ref  | idx_status_amt| 5       |  120 | Using index |
+----+------+---------------+---------+------+-------------+

(illustrative)

Using index = covering, query answered from index alone. Using index condition = Index Condition Pushdown (ICP), where a filter is pushed to the storage engine to evaluate against the index, but the table is still fetched for non-indexed columns. Similar names, different behaviour.

📌 Key takeaways

A covering index holds every column the query needs, skipping heap fetches entirely. In Postgres, INCLUDE lets you add payload columns that aren't searchable or sortable.
Index Only Scan in the plan doesn't guarantee zero table access. Check Heap Fetches in EXPLAIN ANALYZE. That's the real metric. High numbers mean VACUUM hasn't caught up with writes.
In InnoDB, secondary indexes carry the PK for free. Using index means covering; Using index condition means ICP. They aren't the same.
Wide covering indexes cost you: bigger pages, more memory pressure, slower writes. Worth it for hot read paths on stable tables. Not worth it when the visibility map can't stay ahead.

If you're tuning a slow read query and EXPLAIN already shows an index scan, adding one or two columns via INCLUDE is often the cheapest win. But only if your table is vacuumed regularly. Otherwise you're paying index maintenance cost for a benefit you never get.

Somewhat related — API gateways solve a similar "one extra hop" problem at the network layer, where caching at the edge saves the round-trip to the origin the same way a covering index saves the trip to the heap.

Where else to find me

You'll find my other posts and projects at arnavsharma.dev.