I Ran Forensics on 1,629 AI Coding Transcripts to Find Why My Agent Stopped Listening

Everyone has a theory about why their AI coding agent got worse. "The model got nerfed." "They quantized it." "Mercury's in retrograde." Theories are cheap because the evidence is sitting right there on your disk and almost nobody reads it.

Claude Code writes every session to a .jsonl transcript. A month of heavy use left me with 1,629 of them, 1.1 GB of ground truth about exactly what my agent did and exactly how I reacted. So I stopped theorizing and went digging.

Step 1: let the rage be the index

I needed a cheap signal for "the agent failed here." I had a perfect one: the moments I swore at it.

A frustrated user is a labeled dataset. So my first pass was a grep across every transcript for profanity and the tells of a person at the end of their patience — "fucking moron," "I already told you," "why do you keep," "stop." That collapsed 1,629 files down to about 40 sessions worth reading, ranked by how many times I'd lost it.

The ranking alone was informative. The worst session had 28 hits. The pattern wasn't random across the month either, which led to step two.

Step 2: plot it against time and version

Each transcript record carries a timestamp and the exact Claude Code version. I bucketed my profanity rate — swears per human message — by day and by version.

The result killed my favorite theory. My frustration rate ran near zero in late April — 0 incidents across 776 messages over three days — then climbed steadily through May to 3%, then 7% on some builds. I update to the latest version almost daily, so "a bad build" was my prime suspect.

The version data refused to cooperate. The rate bounced around between adjacent versions with no clean trend. High-rage days were simply my high-volume days, the marathon sessions. The decay tracked how long and how hard I worked a session, never which build I was on. That single chart redirected the whole investigation away from the model and toward my own setup.

Step 3: fan out sub-agents to classify every failure

Reading 25 long sessions carefully is a day I didn't have. So I scripted a small extractor that pulled, for each rage moment, the prompt before it, what the agent actually did in between, and a few flags: did my re-grounding context load at session start, was this near a compaction, did the agent write files to a temp directory.

Then I ran a workflow: one sub-agent per session, each classifying every failure into a fixed taxonomy — ignored an existing system, asked instead of executing, redid finished work, broke something without verifying, wrote durable files to temp. Four more agents audited the mechanism: the hooks, the context budget, the temp-directory contradiction, the version timeline. A final agent synthesized. Thirty agents, 1.8 million tokens, about eight minutes.

The taxonomy came back lopsided. The dominant failure, by a wide margin, was "ignored an existing system" — 41% of sessions. The agent re-deriving a job it had done a dozen times as if it were brand new: scraping a site instead of connecting directly, reciting a metric from memory instead of pulling it live, proposing to build a tool that already existed.

That points at one thing. The agent didn't have the knowledge of my systems in front of it when it needed it.

What the evidence actually showed

Three findings, all mechanical, all fixable, none of them "the model."

The re-grounding context almost never loaded. Across the prior month, the block that tells the agent who I am and which systems exist was detectable in roughly 0% of sessions. The hook meant to inject it only fired on fresh launches and resumes, never after a /clear or a compaction — and /clear is how I start most work. The agent ran blind by default.

The guardrails were suggestions. My rule against writing scratch files to a temp directory was advisory, and my auto-approve settings waved it through silently. I'd written 458 durable files to that temp directory across 48 sessions without one of them being stopped.

The context was full of corpses. A memory plugin I'd stopped using months earlier had left 4,648 dead instruction-file fragments scattered across my repos, each injecting stale noise into sessions. (That one earned its own post.)

Compaction, the thing everyone blames, was a non-factor. It was absent from 26 of the 27 worst sessions. The meltdowns happened in normal, short sessions, caused by missing context and unenforced rules.

Step 4: fix it, then build a canary

Fixing it was the easy part once I knew the targets. I widened the hook to fire on /clear and compaction, added a CLAUDE.md fallback that reliably reloads, converted the temp-directory rule into a hook that returns a hard deny, and exorcised the dead plugin.

The part I care about more is making sure I never wait a month to notice the next slide. So I built a weekly drift-check. It scans the same transcripts, computes the same canaries — profanity rate this week versus a trailing baseline, how often re-grounding context loaded, how many files leaked to temp, whether my instruction files have crept back over budget — and posts a red or green summary to Discord every Monday. If any number drifts, I hear about it in days.

The baseline run came back red, which is correct: it's measuring a week that still contains the pre-fix damage. Next week's run is the real test. If the temp-write count drops to zero and the context-load rate climbs, the fixes held. If they don't, the canary tells me, and I go dig again.

The takeaway

Your AI coding tool is one of the most instrumented things you run. It logs every action you and the agent take, with timestamps and versions, in plain text. When it starts failing you, the answer to "what changed" is almost never a vibe and almost always a pattern you can grep for.

Stop guessing. Read the transcripts. Let the swearing be your index.