The service came back after we restarted it, the graphs recovered, and somebody wrote "memory exhaustion, service restarted" in the incident record. The ticket was closed the same afternoon. Six weeks later it happened again, and this time it took the payment queue down with it, and I had the unpleasant experience of reading my own team's notes from the first occurrence and realising they contained no information whatsoever about why the memory had filled up.
Restarting something is not a diagnosis. It is a way of ending an incident, which is a legitimate and often correct thing to do at two in the morning, but it tells you nothing except that the process had accumulated state it could not shed. We had confused the action that restored service with the explanation for the failure, and once that sentence was written down, the organisation treated the matter as understood. Nobody was being lazy. The pressure to close incidents is real, the queue was long, and a resolved ticket looks the same in a report whether or not anybody learned anything.
What made it worse is that the workaround got better with practice. After the second occurrence somebody added a scheduled restart, which genuinely reduced the impact, and that quietly removed the last incentive anyone had to look further. A problem that is painful gets investigated. A problem that has been sanded down to a mild inconvenience gets absorbed into the furniture, and then it sits there until the day the underlying growth rate outruns the workaround.
The change I pushed for was small and procedural rather than clever. An incident cannot be closed with only an action in the resolution field; it needs a sentence about cause, and if we do not have one, the honest entry is "cause unknown, mitigated by restart" with a follow-up item attached. That phrasing is uncomfortable to write, which is exactly why it works. Unknown causes accumulate visibly. Resolved tickets do not.
The real leak, in the end, was a connection pool that was never being returned to under one specific error path. It had been in the code for two years. What kept it hidden was not its subtlety, but our own tidy paperwork.
– Serguey Shinder
Top comments (0)