A few years ago, I was working on a backend service that looked completely healthy.
Tests were green. CPU usage was normal. Memory looked fine. No obvious exceptions were showing up in the logs.
And yet, every few hours, requests would suddenly slow down.
Not fail.
Just become painfully slow.
The API normally responded in around 80–120 ms. During the problem, some requests were taking 8–12 seconds.
Then everything would recover by itself.
Those are the bugs I dislike the most.
If something crashes, at least you have a starting point.
When a system becomes slow only occasionally, you first have to prove which part of the system is actually lying to you.
The first mistake: looking at application code
The team initially started reviewing recent commits.
That was reasonable.
We checked database queries, serialization, background jobs, caching, and a few recently added endpoints.
Nothing looked suspicious.
One engineer suggested adding more application logs.
Another thought the database might need additional indexes.
Someone else suspected garbage collection.
All possible explanations.
But there was one problem.
We were guessing.
So instead of changing code, I asked a simpler question:
Where is the time actually being spent?
That question sounds obvious, but it is surprisingly easy to skip.
Follow the request
We traced a slow request through the system.
Application processing time was roughly 70 ms.
Database time was around 20 ms.
External API calls were normal.
But the client had waited almost 10 seconds.
That immediately changed the investigation.
The application wasn't slow.
Something before the application was.
We started looking at the connection pool and eventually found the real problem.
Under certain traffic patterns, connections weren't being returned to the pool quickly enough.
Most requests worked normally.
But occasionally the pool became exhausted.
New requests then waited for a free connection.
The actual query might take 15 ms.
The request could still take 10 seconds.
That's an important distinction.
A slow database query and waiting 10 seconds to obtain a database connection can look almost identical from the user's perspective.
They are completely different engineering problems.
This is something I ask in interviews
Sometimes I'll give an engineer a situation like this:
"An API endpoint normally responds in 100 ms. A few times per hour it takes 10 seconds. CPU and memory look normal. What would you investigate?"
I'm usually not looking for a specific answer.
I'm looking at how they think.
Junior engineers often immediately name technologies:
"Redis."
"Database indexing."
"Use a queue."
"Add more servers."
Senior engineers usually start asking questions.
Is the latency inside the application or outside it?
Does the slowdown affect every endpoint?
What changed recently?
Is it correlated with traffic?
Are connections, threads, sockets, or workers being exhausted?
What do p50, p95, and p99 latency look like?
Can we trace one slow request end to end?
Those questions are much more valuable than randomly choosing an optimization.
Performance problems are often waiting problems
A common misconception is that a slow system must be doing expensive computation.
Very often, it isn't doing anything.
It's waiting.
Waiting for a database connection.
Waiting for a lock.
Waiting for another service.
Waiting for DNS.
Waiting for disk.
Waiting for a worker.
Waiting for a network timeout.
This is why simply checking CPU usage can be misleading.
A server can be almost idle while users are experiencing terrible latency.
The machine isn't busy.
Your requests are stuck.
The fix was small
Once we understood the problem, the actual fix wasn't particularly impressive.
We corrected the connection lifecycle, adjusted pool configuration, and added metrics around connection acquisition time.
The interesting part wasn't the code change.
It was the debugging process.
Before the fix, we only measured query execution time.
Afterward, we also measured how long requests waited to acquire a connection.
That metric would have exposed the problem almost immediately.
And that's something I try to teach engineers I mentor:
Observability isn't just about collecting more data. It's about measuring the boundaries where work can wait.
A useful interview habit
When you're given a production problem during a technical interview, don't rush to solve it.
Start by narrowing the search space.
Instead of saying:
"I would optimize the database."
Try:
"I'd first determine whether the latency is happening in the database, application, network, or while waiting for a shared resource."
That one sentence tells the interviewer something important.
You're not debugging by intuition alone.
You're building evidence.
Final thought
The longer I work in software, the less impressed I am by engineers who can immediately produce ten possible solutions.
I'm more interested in the engineer who asks the one question that eliminates nine of them.
Production systems are noisy.
Metrics can be misleading.
Logs can look clean while users are still suffering.
The job isn't always to know the answer immediately.
The job is to reduce uncertainty until the answer becomes difficult to miss.
Top comments (0)