TL;DR
A slow memory leak took down one of my production services at 2am. I spent the first hour guessing and the next hour actually fixing it — once I stopped guessing and started using Claude Code to work through heap snapshots systematically. Here's what actually happened, and the 4 lessons I took away about using an AI coding agent for real production debugging instead of toy examples.
The Problem
It started with a Slack alert: memory usage on one of my Node.js services climbing steadily, no plateau, no GC recovery. Classic slow leak. The kind that's fine for six hours and then, right around 2am, tips over into OOM kills and a crash loop.
I'd fixed leaks before, but always the easy kind — an obvious unbounded cache, a forgotten setInterval. This one wasn't obvious. The service had grown over two years, had a dozen contributors, and the leak only showed up under real traffic patterns I couldn't easily reproduce locally.
My first instinct was to just ask Claude Code to "find the memory leak." That went about as well as you'd expect — it read through the codebase, found three plausible-looking candidates (an event listener that might not be getting cleaned up, a cache with no eviction policy, a closure capturing a large object), and presented them all with roughly equal confidence. None of them turned out to be the actual cause.
That's the trap. An agent that's good at reading code will always be able to find something that looks leak-shaped, because most nontrivial codebases have a few sketchy patterns lying around. Plausible isn't the same as correct, and I almost shipped a "fix" for the wrong thing.
I actually wasted close to an hour on the closure candidate specifically. It was the most "interesting" looking one — a callback capturing a large request object — so both the agent and I gravitated toward it first. We patched it, redeployed to staging, watched memory for twenty minutes, and it kept climbing at basically the same rate. That's the moment it clicked that we were pattern-matching on "code that looks like it could leak" instead of "code that is actually leaking," and those are very different searches.
How I Solved It
The fix wasn't a smarter prompt. It was giving the agent real data to work from instead of asking it to reason from the source code alone.
Step 1 — capture the evidence first.
I took two heap snapshots from the running process, twenty minutes apart, using the built-in Node inspector:
node --inspect=9229 server.js
# then, from Chrome DevTools -> Memory tab, or programmatically:
node -e "
const v8 = require('v8');
const fs = require('fs');
fs.writeFileSync('heap-1.heapsnapshot', v8.getHeapSnapshot());
"
Two snapshots matter because a single snapshot just shows you what's alive — it can't show you what's growing. The diff between two snapshots is where a leak actually reveals itself.
Step 2 — hand the agent the diff, not the vibes.
I loaded both snapshots into Chrome DevTools' comparison view, exported the top retained-size deltas as JSON, and gave that to Claude Code alongside the relevant source files:
Here's a heap snapshot diff between t=0 and t=20min under steady load.
The top 5 growing constructors by retained size are:
1. Array (+340MB, +1.2M objects)
2. RequestContext (+180MB, +40k objects)
3. Closure (+90MB)
...
Cross-reference these against the source and tell me which
allocation site is consistent with this growth pattern.
This changed the conversation completely. Instead of "here are three things that look suspicious," I got: "RequestContext growing by 40k objects over 20 minutes lines up almost exactly with your request volume — that object should be short-lived per-request, so if it's retained, something's holding a reference past the response lifecycle. Let's check what holds onto it."
Step 3 — trace retainers, not code paths.
The actual bug: a request-scoped logger was being pushed into a module-level array for "debug replay" during a previous incident, and nobody had ever removed the cleanup step — except the cleanup step had a bug where it only ran on the happy path, not on early returns. Every request that hit an early-return branch (about 15% of traffic) leaked its RequestContext forever.
// Before — the bug
function handleRequest(req, res) {
const ctx = new RequestContext(req);
debugReplayBuffer.push(ctx);
if (shouldShortCircuit(req)) {
return res.status(304).end(); // ctx never gets removed from the buffer
}
// ...normal handling...
cleanupContext(ctx); // only reached on the "normal" path
}
// After — cleanup runs regardless of exit path
function handleRequest(req, res) {
const ctx = new RequestContext(req);
debugReplayBuffer.push(ctx);
try {
if (shouldShortCircuit(req)) {
return res.status(304).end();
}
// ...normal handling...
} finally {
cleanupContext(ctx);
}
}
Claude Code found the actual finally-shaped fix once it had retainer evidence to reason from — it wasn't guessing at "add a finally block somewhere," it traced the exact object identity from the snapshot diff back to this one function.
What made this step work wasn't just the fix itself — it was that the agent could point at the exact class name (RequestContext) from the diff and grep for every place that class got constructed and stored. That's a search a human can absolutely do by hand, but it's tedious and easy to stop early once you find one plausible site. Handing that grep-and-cross-reference grind to the agent, with the constructor name as the anchor, is where most of the actual time savings came from — not from the agent having some special insight the profiler didn't already have.
Step 4 — verify against a third snapshot before calling it done.
I didn't trust "looks right" — I redeployed to a canary, waited 20 minutes under load, took a third heap snapshot, and diffed it against a healthy baseline. Flat growth curve. That's when I actually believed the fix.
flowchart LR
A[Alert: memory climbing] --> B[Snapshot at t0]
B --> C[Snapshot at t0+20min]
C --> D[Diff: top growing retainers]
D --> E[Agent cross-references diff + source]
E --> F[Fix + finally-block cleanup]
F --> G[Canary deploy]
G --> H[Third snapshot confirms flat growth]
Lessons Learned
"Find the bug" from source code alone is a bad prompt for production issues. Static reasoning over code will always surface plausible-looking candidates because most codebases have several. Feed the agent the actual runtime evidence — heap diffs, profiler output, request traces — and the search space collapses fast.
Two snapshots beat one, every time. A single heap snapshot is a photo. A diff between two snapshots under load is a video. If you only ever take one, you're asking the agent (and yourself) to spot growth in something that has no time dimension.
Don't ship the first "consistent with the data" theory without a redeploy-and-reverify loop. The agent's explanation sounded right immediately, and it was tempting to just merge. The canary + third-snapshot step is what actually separates "sounds right" from "is right" — that verification step is on you, not the agent.
The agent is excellent at cross-referencing retainer data against source, but you still own the incident. Deciding what data to capture, when the fix is safe to ship, and what to canary against — that judgment call didn't move. What changed is how fast I could go from "three plausible guesses" to "one verified root cause."
Keep debug-only scaffolding (like that replay buffer) on a very short leash. The root cause here wasn't even the "real" feature code — it was a debugging aid from a past incident that quietly became a permanent leak. If you leave temporary instrumentation in prod, put an expiry on it.
I'd add a sixth, softer lesson: watch for the moment you start treating the agent's first plausible answer as the answer. The closure candidate we chased for an hour wasn't a bad guess — it was a reasonable guess dressed up with enough detail that it felt verified when it hadn't been. The tell, in hindsight, was that neither of us had actually looked at real growth data before committing to it. Once "did we check the diff" became a hard gate before "let's fix this," the false starts basically stopped.
What's Next
I'm turning the snapshot-diff-plus-source-cross-reference flow into a repeatable checklist for the rest of the team, since "just ask the agent to find the leak" was the failure mode that cost me the first hour. Next up is trying the same evidence-first approach on a CPU profiling case (a slow endpoint, not a leak) to see how well the pattern generalizes beyond memory.
Wrap-up
If you've been asking your AI coding agent to debug production issues purely from source and getting plausible-but-wrong answers, try feeding it real runtime data instead — snapshots, diffs, traces — before you ask it to theorize. It's a small change that made a big difference for me.
If this was useful, follow me here on Dev.to — I'm writing up more of these war stories as I go, warts and all.
Top comments (3)
The closure-candidate hour is the part that rings truest. An agent with source-only access will always find something leak-shaped in a nontrivial codebase, and plausible gets dressed up as verified faster than you'd think. The heap diff flips the search from pattern-matching code that could leak to tracing what actually grew.
One thing I'd add from the agents-as-untrusted-tools school: the canary plus third-snapshot step isn't just verification. It's the thing that separates an agent-generated fix from a fix you own. The agent proposes; the canary confirms. Skip that loop and you're shipping agent output on the same evidence the agent used to generate it.
This is a good use case for AI coding help because the valuable work is narrowing the search space. I would still keep the final proof grounded in runtime evidence: heap snapshots, reproduction steps, before-after metrics, and the smallest patch that explains the leak.
This is a test comment about the heap snapshot debugging approach.