For most of a year, every query our application made — reads and writes alike — went to the same primary Postgres instance. It worked fine until it didn't: our primary was sitting at a sustained 85-90% CPU during business hours, query latency was creeping up across the board, and every time we ran a reporting query for an internal dashboard, customer-facing endpoints got measurably slower for the few seconds it took to run. We'd been putting off the obvious fix — read replicas — because it sounded like it would touch a lot of application code. It touched less than we expected, and the part that actually mattered wasn't the infrastructure, it was deciding which reads were safe to move.
The infrastructure part was almost boring. We stood up two read replicas using standard streaming replication, put them behind a connection string our ORM could route to separately from the primary, and confirmed replication lag stayed under 100ms under normal load. That part took an afternoon. The part that took two weeks was going through every code path that touched the database and classifying each query as either "must go to the primary" or "safe on a replica."
Read-your-own-writes is where this gets hard
The naive version of read replicas — send all writes to primary, send all reads to replicas — breaks the moment a user does something like update their profile and then immediately view their profile page, and the view happens to route to a replica that hasn't caught up yet. They see their old data and assume the save failed. We didn't want to solve this with a blanket "wait for replication" because that reintroduces the latency problem we were trying to fix. Instead we tagged specific request flows — anything immediately following a write in the same user session — to pin to the primary for a short window (we used 5 seconds, which covered our observed replication lag with a comfortable margin), while everything else defaulted to replicas.
We built this as a decorator on our query layer, not a config toggle scattered through the codebase. Every database call already went through a thin query-execution wrapper, so adding a read_preference parameter there — defaulting to replica, explicitly set to primary only where a caller needed strict consistency — meant we didn't have to hunt down every call site by hand. We did still have to review each one, but the review was "does this specific query need to be primary" rather than "how do I even route this differently."
The categories that had to stay on the primary surprised us a little
Obviously anything in a write transaction. Less obviously: any read used to make an authorization decision (we did not want a stale replica to be the reason a permission check passed when it shouldn't have), and any read feeding into a financial calculation, even one that felt low-stakes, because a few seconds of staleness compounding across a report was a correctness bug we didn't want to explain to anyone. Analytics queries, search, activity feeds, and the vast majority of page-load reads all moved to replicas without any observable behavior change for users.
Monitoring replication lag became a first-class metric, not an afterthought. Before this project we weren't watching lag at all because it didn't affect anything. Once application correctness depended on it staying low, we added alerting at 2 seconds (warning) and 10 seconds (page), and we load-tested what happens to our primary-pinning window if lag spikes beyond our 5-second assumption during, say, a bulk data migration. It does happen occasionally during large batch jobs, and our on-call runbook now includes "check replication lag" as a standard step whenever someone reports seeing stale-looking data.
The result was primary CPU dropping from the high 80s/low 90s down to around 20% under the same traffic, and query latency on customer-facing endpoints becoming noticeably more consistent, since it was no longer sharing capacity with every internal reporting query in the company. The infrastructure change was the easy part. Knowing exactly which reads could tolerate a few seconds of staleness and which absolutely could not — that's the part that actually determines whether a read-replica migration is safe or is a very slow-motion correctness incident.
We think through this same read/write consistency tradeoff on most systems we build at Edilec, since it comes up anywhere read volume outgrows what a single primary should carry — more on how we approach it at edilec.com.
Top comments (0)