Read replicas are a standard scaling technique: route read-heavy traffic to one or more replica databases, kept in sync with a primary, freeing the primary to handle writes with less contention. The pattern is well understood and genuinely valuable for the right workloads. It's also frequently adopted as a default scaling step before the specific problems it introduces, primarily around replication lag, are fully understood by the team implementing it, which leads to a category of bugs that are confusing to diagnose specifically because they only manifest intermittently and are tied to timing rather than to a consistent, reproducible condition.
Replication lag is not a theoretical edge case, it's a constant, variable condition
Data written to a primary database takes some amount of time, often milliseconds, but sometimes considerably longer under load, to propagate to read replicas. During this window, a query against a replica can return data that doesn't yet reflect a very recent write to the primary. This isn't a rare failure condition, it's the normal, expected operating behavior of an asynchronously replicated system, which means any application logic that assumes reads will always reflect the most recent writes is making an assumption that read replicas structurally don't guarantee.
The specific danger is that this assumption often works fine in testing and in typical low-load conditions, where replication lag tends to be minimal, and only becomes a visible problem under real production load, when replication lag can grow meaningfully, or during specific timing-sensitive sequences, a write immediately followed by a read of that same data, that happen to occur before replication has caught up.
The read-after-write problem is the most common practical failure mode
A specific, frequently encountered pattern: a user submits a form, the application writes the new data to the primary database, and then immediately redirects to a page that reads the just-written data, potentially from a replica. If replication hasn't yet caught up by the time that read happens, the user sees a page that doesn't reflect the change they just made, sometimes appearing as though their submission failed or was lost entirely, even though the write actually succeeded and simply hasn't propagated to the replica yet.
This specific failure mode is genuinely common in systems that route reads to replicas without deliberate handling of the read-after-write case, and it's a particularly confusing bug to debug from a support or QA perspective, since the underlying data is actually correct and the write genuinely succeeded, the problem is purely in the timing of when a subsequent read happens to occur relative to replication catching up.
Common mitigations, and their own trade-offs
Routing reads immediately following a write to the primary rather than a replica, at least for a short window or for the specific data just written, avoids the read-after-write problem directly but requires the application to explicitly track which reads need this special handling, adding real implementation complexity rather than being a transparent, automatic solution.
Session-level read consistency, routing all reads within a given user session to the primary for some period after that session performs a write, reduces the read-after-write problem's visibility to end users at the cost of reducing the actual read-scaling benefit of having replicas at all for that session's subsequent activity, since it's effectively bypassing the replica specifically when the user is most actively interacting with recently changed data.
Checking replication lag explicitly and falling back to the primary if lag exceeds a threshold provides a more general solution but requires the infrastructure to actually expose current replication lag in a way the application can check efficiently, which isn't available or straightforward in every database setup, and adds its own runtime overhead and complexity to every read path that needs this check.
None of these mitigations are free, and choosing among them requires understanding the actual read-after-write patterns present in a specific application, rather than assuming a single, universal solution applies cleanly to every read-replica use case.
Not every read workload actually benefits from replica routing
A read replica genuinely helps with workloads that are read-heavy, latency-tolerant of eventual consistency, and where the read-after-write concern either doesn't apply or can be handled with acceptable added complexity, analytics queries, reporting dashboards, and search functionality are commonly good fits. Workloads where a user directly expects to see the immediate effect of their own recent action, most core transactional application flows, are frequently a poor fit for naive replica routing without the specific consistency handling described above, and forcing these workloads onto replicas primarily because "read replicas improve scalability" as a general principle, without checking whether the specific workload's consistency requirements actually tolerate the trade-off, is a common source of confusing, hard-to-reproduce production bugs.
Monitoring replication lag needs to be a first-class operational concern
Once read replicas are in use for any workload with genuine consistency sensitivity, replication lag itself needs to be actively monitored and alerted on as its own operational metric, separate from monitoring the health of the primary and replica instances individually. A replica that's technically up and responding to queries, but with replication lag that's grown significantly beyond normal, represents a real degradation in the guarantees the application is implicitly relying on, even though nothing about the replica's own health metrics would necessarily indicate a problem on its own.
Teams that adopt read replicas without building this specific monitoring often don't discover growing replication lag until it manifests as user-visible data inconsistency complaints, at which point diagnosing the actual root cause, distinguishing "the write genuinely failed" from "the write succeeded but hasn't replicated yet" from application logs and user reports alone, is considerably harder than it would have been with direct visibility into replication lag as an explicit, monitored metric from the start.
The underlying question worth asking before adopting read replicas
Read replicas are a genuinely valuable and common scaling pattern, and this isn't an argument against using them. It's an argument for adopting them deliberately, with explicit consideration of which specific workloads actually tolerate eventual consistency, what mitigation strategy will handle the read-after-write case for workloads that don't tolerate it well, and how replication lag will be monitored as an ongoing operational concern, rather than adopting replicas as a default scaling step and discovering these considerations reactively once confusing, intermittent consistency bugs start appearing in production.
Top comments (1)
A time-based “stick this session to primary for N seconds” policy is easy to ship, but it is only a guess about replication progress. A stronger contract is causal: after a write, capture the commit position (for PostgreSQL, an LSN) and attach it to the session/request context. Replica reads are allowed only once that replica has replayed at least that position; otherwise route to primary or return an explicit retryable freshness state.
That also makes monitoring more actionable. Wall-clock lag can look small while a replica is far behind in bytes during a burst, and it can look large during an idle period without affecting the user’s dependency. Track replay position/byte distance and the age of a replicated heartbeat, then test the routing rules during replica pause, catch-up, and failover. The useful SLO is “can this read satisfy its required consistency token?”, not simply “replica lag is under 500 ms.”