Read Replicas Promised Scale. They Also Gave You Stale Data.

You added a read replica to take pressure off the primary. Reads got faster, the primary stopped sweating, everyone was happy — until a user updated their profile, hit refresh, and saw the old value stare back at them. Or an order flipped to "paid" in the admin panel while the customer-facing page still said "pending" for ten seconds. None of that is a bug in your code. It's the shape of asynchronous replication, and almost nobody designs for it up front.

This isn't a post arguing against replicas. It's a post about the one cost nobody puts on the whiteboard: a read replica gives you throughput, not freshness. Those are different promises.

What "asynchronous replication" actually means

With a typical primary-replica setup (Postgres streaming replica, MySQL read-only replica, a cloud reader endpoint, etc.), the replica applies changes after they've committed on the primary. There's a lag — usually milliseconds, sometimes seconds, occasionally minutes when something backs up.

During that window, the replica is serving data that is simply not current. Not wrong, exactly. Just older than the primary by the length of the lag.

primary:  UPDATE users SET name='Ada' WHERE id=7;  -- commits at T0
replica:  still has name='Bob' until the WAL/redo reaches it (~T0 + lag)

If a client reads from the replica in that gap, it gets 'Bob'. The write succeeded; the read just didn't see it yet. That's a read-after-write (read-your-own-write) violation, and it's the single most common surprise people hit after adding a replica.

Where the lag actually comes from

It's easy to blame "the network," but the sources are more varied:

• Apply-queue backlog. The replica can receive WAL faster than it can apply it, especially under heavy write load or large transactions.
• Long-running queries on the replica. A reporting query saturating I/O delays replication apply.
• One huge transaction. A multi-GB bulk update can stall the apply thread for its full duration.
• Backups / VACUUM / maintenance windows. These compete for the same resources.
• Network partition or replica restart. Now you're behind by however long it was down.

The point: lag is not a constant you can subtract away. It's a distribution, and the tail is what bites you.

The instinct that makes it worse: route everything to the replica

The tempting setup is "writes to primary, reads to replica" — send all SELECTs to the read endpoint and call it scaling. That's also the setup that produces the profile-refresh bug above, because some reads must be strongly consistent:

• The row you just wrote (read-your-own-write).
• Balances, inventory counts, payment/order status — anything where showing a stale value causes a real-world mistake.
• Anything another user just changed that the current user is actively acting on.

Pushing those through a replica "because it's a read" is how you get a customer who paid twice because the available-balance read lagged, or an admin who approves something they shouldn't because the status hadn't caught up.

What actually works (without throwing away the replica)

You don't need synchronous replication (which trades throughput for freshness — often the opposite of why you added a replica). You need to be selective about which reads need to be current.

1. Read-after-write: pin the session to the primary right after a write.
The simplest version: after a mutating request, keep that user's subsequent reads on the primary for a short window (a few seconds). The user sees their own change immediately; everyone else tolerates the lag. Many connection pools support a "primary for this session" flag; in a Spring app you can route through a replication-aware datasource that flips to primary for the remainder of the request or a short TTL.

2. Version / token handoff.
When you write, return a version or timestamp to the client. On the next read, the client sends it back; if the replica's applied position is behind that version, read from the primary instead. More precise than a fixed time window, and it survives the "user walked away and came back 20 minutes later" case.

# after a write, tell the client how far the data has progressed
response.headers["x-write-version"] = str(user.version)
# next read: only escalate to primary if the replica hasn't caught up
if replica_applied_version < int(request.headers["x-write-version"]):
    read_from(primary)

3. Route by data criticality, not by SQL verb.
Decide per query: is this read allowed to be slightly stale? Analytics dashboards, search indexes, public profiles, "last updated" lists — those are replica-friendly. Account balances, order state, anything transactional — primary. Encode that decision in your data-access layer, not in ad-hoc routing scattered across the codebase.

4. Don't forget the cache has the same disease.
A cache is just a faster, more aggressive replica. "Set a TTL" is not an invalidation strategy; it's a confession that you'll serve stale data for the length of the TTL. For read-after-write correctness, prefer write-through (update the cache on write) or explicit invalidation on the write path over relying on expiry. The same read-your-own-write logic applies: after you write, make sure the next read doesn't hit a stale cache entry.

When the replica is exactly the right tool

To be clear, replicas are great — for the right reads:

• Analytical and reporting queries that would otherwise hammer the primary.
• Full-text search indexes.
• Public, non-personalized content.
• Any read where "a few seconds old" is fine, which is most reads.

The mistake isn't using a replica. It's treating "replica" and "consistent" as the same word.

The metric you should be watching

If you run a replica and you're not tracking replication lag, you're flying blind. Every engine exposes it: Postgres via pg_stat_replication (or comparing pg_last_wal_receive_lsn / pg_last_wal_replay_lsn), MySQL via Seconds_Behind_* / replica-lag metrics. Plot it, alert on the tail, and — crucially — keep it visible where a human decides whether a stale read just caused a problem. A p99 lag of eight seconds tells you exactly how far behind a confused user might be. Concretely, on Postgres you can see the gap directly:

-- how far behind is each replica, in bytes of WAL not yet replayed
SELECT client_addr,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS bytes_behind
FROM pg_stat_replication;

The part nobody warns you about

Adding a read replica feels like free scale. It's not free — it's a trade where you give up "every read sees the latest write" in exchange for "reads don't load the primary." That trade is almost always worth it. But you have to make the trade consciously: enumerate the reads that must be current, route only those to the primary (or pin the session, or hand off a version token), and stop assuming SELECT means consistent. The replica solved your throughput problem. It quietly created a freshness problem you now own.