How an autosave feature took down production — and five fixes that stuck

An editor I built saves your work as you type. That's the feature users love most, and one evening it was the reason the whole app stopped answering.

No single bug caused it. Five ordinary decisions, each reasonable on its own, lined up. Here they are, with what I changed.

1. Saving things that hadn't changed

The editor sent the cursor position and the current scene on a timer. The server wrote them every time — even when nothing had changed. And the "last saved" baseline on the client never advanced, so every tick looked like a new change.

At normal traffic this was invisible. When usage climbed, the database spent most of its time writing identical rows.

Fix: compare before you write. The client keeps a baseline and sends only real diffs; the server skips writes where the value is unchanged. A no-op save should cost nothing.

2. Read-before-write on every save

Each save first selected the row, then updated it. Two round trips where one conditional update would do — doubling the load exactly where it hurt.

Fix: a single upsert with a condition. Let the database decide, in one statement.

3. A blocking call inside async code

The API server was async, but one database path used a synchronous driver call. Under load, that one call blocked the event loop, so requests that didn't touch the database at all — health checks included — queued behind it.

Fix: no synchronous I/O on the event loop, enforced by a lint rule, not by memory.

4. Health checks that killed waiting pods

Because health checks were stuck behind the slow database, Kubernetes decided the pods were dead and restarted them. The fresh pods reconnected, retried pending saves, and hit the database even harder. The restarts made the outage worse.

Fix: split the probes. Liveness answers only "is the process alive?" and never touches dependencies. Readiness can check the database — and a failing readiness probe takes the pod out of traffic without killing it.

5. Scaling on the wrong signal, and no alarm

The autoscaler watched CPU. The pods weren't busy — they were waiting — so CPU stayed low and nothing scaled. When I did try to add capacity, I hit node-pool and quota limits I hadn't checked in months. And no alert fired: users noticed before I did.

Fix: scale on what the users feel (request latency, queue depth), keep quota headroom as a checked number, and alert on error rate and latency at the edge, not only on CPU and memory.

What I'd tell a team shipping a "save as you type" feature

• Writes are the expensive part. Budget them per user per minute and measure that number before launch.
• Make no-ops free. The client and the server both skip unchanged data.
• Treat liveness probes as dangerous. A probe that checks dependencies turns a slowdown into a restart storm.
• Autoscale on the bottleneck, not on the metric that's easiest to read.
• Test the alarm. If you've never seen the alert fire, assume it won't.

None of the five decisions was wrong when it was made. Together they turned a popular feature into an outage. The fixes are boring — and boring is what you want from autosave.

---

I'm Noam Tzuberi — ex-Head of AI and founder of WriteinaClick, an AI book-writing platform I built solo. I help SaaS teams ship AI features that hold up in production. Start with a 3-day AI feature audit + working prototype: https://contra.com/noam_tzuberi_78uj1psz.