DEV Community

ADITYA DUTT PANDEY
ADITYA DUTT PANDEY

Posted on Originally published at lab.adpandey.dev on

Our traffic grew seven times and we got faster. I don't fully trust that.

Here are tunyt's numbers from one stretch of this year:

Before After
Weekly requests 69K 485K
Latency 110 ms 59 ms
Apdex 0.55 0.89
Error rate 2.84% 0.32%
Deploy time 20+ min under 9 min

The error rate moved between June and August 2026, while traffic was still climbing.

Traffic went up about seven times and every other number got better. That's the kind of table that makes a great LinkedIn post, and it's exactly the kind of table I've learned to be suspicious of, my own included. So this post is about how we got there, and what I'm not willing to claim.

Decision 1: measure before touching anything

Before we changed a line, we made the server tell us how it was doing. A metrics endpoint reports, for every route, request duration at the 50th, 95th and 99th percentile. It also reports how many jobs are waiting in each queue and the state of every circuit breaker.

You usually already suspect what's slow, and suspicion is how you spend a week optimising the wrong thing. The measurements decide what we fix. What I think is slow doesn't get a vote.

Decision 2: per route, and in percentiles

An average across the whole API is a comfortable lie. Ninety fast endpoints will hide the one that's slow for every single person who touches it, and that one is usually the endpoint people came for. So every route gets its own numbers.

Within a route, we watch percentiles. The 50th is your typical user. The 95th is your unlucky one, the person on a slow phone and bad data who decides your app is broken and doesn't come back. The 99th is where the strangest problems hide. If you only watch the average, you're optimising for someone who doesn't exist.

Decision 3: one number for "does it feel fast"

Percentiles are precise but hard to watch day to day. So alongside them we track Apdex, a single score from 0 to 1.

You pick a target time. Requests faster than it count as satisfied. Requests up to four times slower count as tolerating. Anything slower is frustrated.

Apdex = (satisfied + tolerating / 2) / total

Enter fullscreen mode Exit fullscreen mode

Going from 0.55 to 0.89 means the bulk of requests moved into satisfied. It's a blunt number, but it's the closest single number I know to "does this feel fast?"

What changed in that window

I'll list what we did, and I won't pretend I can tie each number to one change.

  • Every list query got a ceiling. Pagination on endpoints, an explicit take everywhere else.
  • Loops of lookups became one query. N round trips turned into one IN query and a Map.
  • Hot reads got a cache with real invalidation. Every write path deletes what it changes.
  • Expensive reads got single-flight. One request rebuilds a missing value while the rest serve the last good copy, so a cache expiry can't stampede the database.
  • Analytics moved to rollups. Finished days come from a daily table, and only today is computed live.
  • The database got a budget. One connection limit for the whole deployment, split between the API and the workers, with capped fan-out.
  • Hot pages start on the server. Event pages fetch their data in parallel on the server and hand it straight to the client's cache, so the browser doesn't wait for a second round of requests.

Decision 4: treat deploy time as reliability

Deploys went from more than twenty minutes to under nine. Docker layer caching meant unchanged dependencies stopped being rebuilt. Lint and tests became parallel jobs instead of one long line. Model weights got baked into the base image instead of downloaded on every build.

Deploy time quietly controls how you ship. When a fix takes twenty minutes to reach production, people batch changes together, and big batches are harder to review, harder to roll back and more likely to break something. Nine minutes makes small changes cheap again. Deploy time is a reliability metric pretending to be a convenience.

Decision 5: refuse to take full credit

Some of the improvement came from the changes above. Some almost certainly came from things I didn't control: traffic patterns shifting, caches staying warmer at higher volume, effects I never measured.

If I told you the rollup table saved exactly 30 ms, I'd be making it up. I'd rather lose a little credit than gain a little dishonesty, because the moment someone catches you overclaiming one number, they stop believing the rest.

What I can say honestly is that the system got faster and more reliable while doing about seven times the work. Every change on that list is now a written rule the team and our agents check before shipping, so the numbers don't quietly slide back.

The table is real. The story that every number came from a fix I made is the part I won't tell you.

Top comments (0)