Our service dashboard had a panel called database time, and for thirteen weeks it showed a p99 of about nine hundred milliseconds at peak. We tuned queries, added two indexes, rewrote a join, and none of it moved the line. There was a quote on my desk for an instance upgrade at two thousand four hundred pounds a month and I was a day or two from approving it.
What stopped me was a number that disagreed. The database's own statistics view had the same statements at three to five milliseconds, executed millions of times, with no outliers to speak of. The instance sat at twelve percent CPU through the busiest hour of the week. Two measurements of the same work differing by a factor of two hundred, and I had believed the one further from the evidence for two and a half months.
Our tracing wraps the repository method. The span opens before the pool hands us a connection and closes after the rows are mapped, so it contains the wait for a connection as well as the query. Every millisecond a caller spent queueing for the pool was being recorded, graphed and labelled as time spent in the database. The queue existed because of one endpoint: the despatch note export opened a transaction, then called the rendering service over HTTP inside it, and sat holding a connection for up to eight seconds while waiting for a PDF. Forty of those a minute against a pool of twenty per pod is enough to make every other query in the process look slow.
The instrumentation change was an hour's work and settled it. Acquiring a connection and executing a statement are now two spans with two names, and the panel split into two lines that told the whole story on the first render. An architecture test fails the build if a transaction is open across an outbound HTTP call. The export reads, commits, then renders. And the alert that used to watch query duration now watches time spent waiting for the pool.
A measurement that spans two systems gets attributed to the far one. Put a boundary in your instrumentation wherever a resource changes hands, or your graphs will keep naming a suspect that was never in the building.
– Sergey Shinder
Top comments (0)