A support ticket comes in: "Checkout feels slow on Tuesdays." No error, no stack trace, no screenshot. You check the status page. Everything's green.
That gap between "the dashboard says fine" and "a paying customer says otherwise" is where most performance problems actually live. Here's how to close it without waiting for an engineer to have free time.
Up and fast are not the same promise
A server is "up" when it answers a request. It's "fast" when it answers before the person waiting notices the delay. A status page only checks the first one.
Picture that Tuesday checkout: request goes out, four seconds pass, confirmation appears. No error fired. Nothing crashed. The server did its job. The customer still thinks your app broke, because a four-second spinner during payment feels like death.
And the cost of that four seconds is real. Cart abandonment climbs sharply once load time crosses a few seconds, and even a tenth-of-a-second improvement can move conversion numbers. None of that shows up as a red light on a dashboard. It shows up as silence, as people quietly leaving.
A green status check tells you the patient is alive. It says nothing about whether they're in pain.
Averages hide the exact people you're looking for
This is the one lesson worth remembering above everything else in this post: averages lie, percentiles don't.
Say you've got ten checkout requests. Nine finish in under 100ms. One takes two full seconds. Average it out and you get a perfectly respectable-looking number, something like 257ms. That number describes literally none of the actual requests. It's worse than 90% of them and way better than the one that mattered.
This is why percentiles exist. p95 means 1 in 20 users waited longer than that number, it's your early warning sign. p99 means 1 in 100 users waited longer, and that's usually where churn actually lives. The average, meanwhile, is just a number that describes almost nobody.
Next time someone shows you a "response time looks fine" chart, ask for p95 and p99 instead. If the answer is "we only track the average," that's your first red flag right there.
Throughput and error rate won't save you either
Error rate tells you what percentage of requests failed outright. Worth watching, but your Tuesday problem never threw an error. It just took forever to succeed. A clean error rate can sit right next to a customer rage-quitting your checkout.
Throughput (requests over time) is more useful as a pattern-matcher. Steady traffic plus a climbing p99 is the fingerprint of a bottleneck hiding inside an app that technically works. If throughput is flat and response time is climbing, something changed on your end, not because more people showed up.
Is it your app or the network
Before you send an engineer chasing a phantom bug, rule this out: a phone on bad wifi will make your backend look slow even when it answered instantly. That's network latency, not application latency, and it's a completely different fix.
A quick sanity check: if even a trivial database round-trip consistently takes 100ms or more from your own app server, the problem probably isn't your query. It's the network between your app and the database.
Do this: separate "this user's connection is bad" from "our app is slow" before looping anyone in.
Don't do this: assume every slow-page report is a code problem.
*Tracing turns "checkout is slow" into "this step is slow"
*
A high p99 tells you pain exists somewhere. It doesn't tell you where. For that you need distributed tracing: give a single request a unique ID and record how long each hop takes as it moves from your app, to a payment API, to the database, and back.
Instead of a wall of graphs saying "checkout is slow," you get a timeline of one specific slow checkout, broken into steps, with actual seconds attached to each one. That's the difference between guessing and pointing.
It's almost always the database
Trace enough slow requests and they converge on the same ending, over and over: a database query nobody optimized.
Three usual suspects:
- N+1 queries — fetch 10 orders, then fire 10 more queries to get the customer for each one. Common in ORMs with lazy loading turned on by default.
- Missing indexes — without one, the database scans every row. Fine on a small table, brutal on a big one.
- Queries that don't scale — a query that runs in 50ms on your dev machine with 1,000 rows can take 15+ seconds in production with 10 million rows. This is exactly why the problem only shows up "on Tuesdays," when traffic peaks and the query finally buckles under load.
That third one is your whole Tuesday mystery, solved.
Two terms worth knowing before your next standup:
- Slow query log: the database's own record of every query that ran past a threshold you set. Just ask what's in it.
- Eager loading: the standard fix for N+1, fetching related data in one query instead of a loop.
The actual sentence to bring to your engineers: "Under Tuesday load, p99 on checkout spikes and the trace points at the database. Can we check the slow query log for an N+1 or a missing index?" That's a hypothesis they can act on in an hour, not a vague complaint.
Catching it before the support ticket
Once you know where to look, you can stop waiting for customers to tell you:
- Real user monitoring ties slow load times to actual sessions, devices, and networks, so you're not guessing whether it was your app or someone's hotel wifi.
- Synthetic monitoring runs scripted checkouts on a schedule, catching a slowdown as it builds instead of after it's already cost you carts.
- Threshold alerts mean you hear about a rising p99 from a rule, not from an angry email.
The checklist
- Take the vague complaint seriously and pin it to a specific flow.
- Check p95 and p99 on that flow. Never the average.
- Trace one slow request end to end to find which hop ate the time.
- Suspect the database first. Check the slow query log for N+1s or missing indexes.
And the tell that a team is running monitoring without ever finding anything: they only report averages, they only watch up/down status, they hear about slowness from customers instead of alerts, and they can't trace a single request start to finish.
If your team is doing two or more of those, the monitoring isn't broken. It's just not being asked the right questions.

This is the condensed version. The full writeup digs deeper into Apdex scoring, SLA/SLO targets built on p99 instead of averages, and how fragmented tooling quietly adds hours to incident resolution. Worth a read if you want the complete picture: How to Find Application Performance Monitoring Bottlenecks
Top comments (0)