Connecting LLMs to my debugging flow to fix a memory crash

#programming #debugging #artificialintelligen #llm

Every engineer has a “Mystery Case” story. 🔍

For a long time, mine was a service that would run perfectly for weeks, and then, always at the most unexpected times, would violently consume memory and die.

I didn’t just ignore it. I fought it.

I analyzed logs, optimized code to reduce allocation pressure, and even claimed a “False Victory” once, deploying a fix I swore was the root cause. The crash stopped for a week, then came back.

We kept restarting the system by hand to keep it running, but we didn’t know what was really causing the problem. We tried hard to find out why, but checking things manually wasn’t enough. There was just too much data to look through, which hid the important facts. Trying to piece it together ourselves took way too much time and effort. Because we didn’t have a better tool to help us, we got stuck.

Recently, I finally solved the case. Not by traditional debugging. I did it by using AI to connect a conversation between my varios “disconnected” tools.

Here is the breakdown of the investigation.

Phase 1: Evidence Gathering (Finding the Needle in the Telemetry Haystack)

I exported the raw metric data and treated the AI as a Pattern Matcher.

My Prompt: “Analyze this dataset. Find the exact timestamps where memory allocation spikes > 20% in under 60 seconds.”
The Result: It identified two specific seconds in time.

I took those timestamps and asked the AI to generate a targeted query for my log aggregator (which have its own agent). The logs lit up. Every single memory spike aligned perfectly with a specific “System Refresh Event.”

In hindsight, this looks obvious. But in a codebase with millions of lines, “obvious” is a luxury you only get after you know exactly where to look.

Phase 2: The Interrogation (The “Chat-to-Profiler” Bridge)

Knowing when it happened was half the battle. I needed to know what was exploding.

The crash was happening deep in our core infrastructure. This wasn’t “bad code”; it was battle-tested bedrock logic that has scaled with us for years, making any modification a high-stakes operation requiring surgical precision.

In previous attempts, analyzing a production dump meant a deep, manual dive into a memory profiler. While modern profilers are powerful, they still require you to do all the heavy lifting. This time, I used a Model Context Protocol (MCP) to turn my profiler into a conversational partner. Instead of hunting through heap snapshots myself, I had a dialogue:

AI: “I detect a high volume of duplicate objects on the heap.”

Me: “That’s impossible, those should be cached and reused.”

AI: “The cache references are unique. They are not being reused.”

It wasn’t magic. I had to guide the AI, filtering out hallucinations and refining the context, but it handled the syntax while I focused on the semantics. It pointed me to a race condition I had looked at a dozen times but never truly saw.

Phase 3: The Implementation (Architecting the Cure)

The root cause was a classic “Stampede”: clearing old data before the new data was ready.

I knew the concept of the fix (a “Relay Race” pattern), but implementing high-concurrency caching logic in a critical subsystem is risky.

I used the AI to implement the solution :

The Prompt: “Refactor this cache logic to support a ‘Versioned Handoff’. Ensure thread safety during the swap between Version 1 and Version 2.”
The Result: The AI generated the boilerplate for the atomic swapping mechanism.

But I didn’t just copy-paste. I established an “AI Tribunal” (Github Copilot ,running cluade, for logic, Gemini for architecture) and performed a rigorous human code review to ensure the locking mechanism was sound before it ever touched the staging enviroment.

The Takeaway

Don’t replace yourself; multiply yourself. I used AI to handle the “grunt work” of parsing data and generating boilerplate.
Orchestrate, don’t just chat. Connect your tools. Let the metrics talk to the logs, and let the profiler talk to the code.
Respect the “Boring” Solution. The fix wasn’t a fancy new framework; it was a simple, boring Relay Race pattern.