We almost shipped a “performance win” that was mostly HTTP 429s.
The dashboard said p95 was 136ms. It looked excellent. It was also mostly useless.
This is a postmortem of measurement failures from hardening an internal platform operations dashboard (Django + React). I’m keeping employer specifics out. The patterns travel.
The seductive number
When you’re chasing SLOs, a single latency percentile feels like truth. It’s clean. It’s graphable. Leadership likes it.
But a latency number without a status-code story is a rumor.
We hit that lesson three times in one milestone.
Lie #1 — Measuring the rate limiter
We pointed a load tool at the API with a single identity. Our per-user limit was 300 requests/minute.
Result: p95 ≈ 136ms. Celebratory Slack energy.
Then we looked at statuses.
98.6% of those responses were HTTP 429.
We hadn’t measured endpoint speed. We’d measured how fast the throttle rejected us.
Rule: Never report a latency SLO unless the run also asserts on success codes. A fast failure is not a fast success.
In the harness we added hard thresholds like: throttled_429 count must be under 1. If the rate limiter is the thing under test, say so. Don’t file it under “API p95.”
Lie #2 — Measuring permission denials
After fixing the identity/volume mix, another endpoint still showed ugly error rates. Digging in: load users sat in a bare role group. Every call returned 403.
Again: a latency figure computed over “you are not allowed.”
Rule: Seed realistic authz state next to realistic data volume. If your fixture user can’t do the job a real user does, your timing is fiction.
Lie #3 — Measuring an unauthenticated probe
We ran an availability drill. It reported 0.000% availability.
Panic — until we noticed the probe ignored session setup and ran unauthenticated. Thousands of 4xx. A broken probe, not an outage.
Rule: A run that didn’t exercise the assertion is void, not a verdict. Treat “we forgot auth on the probe” the same way you’d treat a skipped test.
The sibling problem on the accessibility side
Performance wasn’t the only place green checkmarks lied.
Our accessibility suite once passed dozens of unit tests while the product still shipped serious contrast failures. Why? color-contrast was disabled in the suite with a note that colors came from the design system.
Disabled checks are invisible defects.
Worse: re-enabling the rule inside jsdom didn’t save us. axe often returns incomplete there, not violations, so toHaveNoViolations() can still pass. The “fix” was a no-op until we used live-browser audits and numeric contrast tests against a registry of token pairings — and then guarded the registry so new literals couldn’t sneak past.
Rule: Audit what a suite switches OFF before you trust it. When a unit suite and a live audit disagree, the live audit wins.
What actually helped
- Assert statuses with timings. Latency and availability without 2xx/3xx discipline is cosplay.
- Seed volumes and roles together. We eventually measured against hundreds of instances and thousands of workflow intents — and users who had the right groups.
-
Prefer production-like topology for SLO proof. Single-process
runserver+ SQLite will invent tails that vanish under multi-worker + Postgres + Redis. Record which topologies cannot host the measurement. - Refuse the two tempting cheats: raising the threshold until it passes, or lowering load until it goes green. An unproven gate is a legitimate outcome. A fabricated pass is not.
- Read your own artifacts before filing the verdict. Twice we almost opened “fix” tickets against bugs that didn’t exist because a report contradicted its own HAR/screenshots.
A template you can steal
Before you paste a p95 into a slide, answer:
- What % of responses were successful for the intended operation?
- Did we measure the rate limiter, the authz layer, or the handler?
- Did the probe use the same auth path as a real user?
- Is the topology close enough to production that the number means something?
- What did we deliberately not claim?
If you can’t answer those in one paragraph, you don’t have a performance result yet. You have a screenshot of a graph.
Closing
The interesting bugs weren’t “the API is slow.” They were “our instruments were confidently wrong.”
Ship gates that fail closed when the measurement is dishonest. Your future self — and anyone who inherits the dashboard — will trust you more for an unproven clause than for a pretty lie.
If you’ve caught a green metric that was measuring something else, I’d like to hear the story in the comments.
Top comments (1)
The order-execution version of this is a fast API acknowledgement that gets reported as a fast trade. The acknowledgement only says the request was received; it says nothing about a later fill or rejection. I'd split that p95 into request-to-ack and request-to-terminal-order-state, then assert how many orders reached the intended terminal state. Otherwise the fastest path may be the reject path.