How I Triaged 8,400 Production Errors Into 11 Real Bugs With Claude Code

TL;DR

My error tracker had 8,400 events a week across ~340 distinct issues, and nobody on the team actually triaged them. I built a small pipeline that feeds structured error data plus repo context into Claude Code and forces it to return a verdict per issue — and it surfaced 11 genuine bugs that had been hiding under the noise for months. Here's the setup, the prompt structure, and the five things I got wrong before it worked.

The Problem

Every team I've worked on has the same dead ritual: someone sets up an error tracker in week one, everyone watches the dashboard for about a month, and then the volume outgrows human attention and the tab quietly stops getting opened.

That's exactly where we were. Concretely:

• 8,400 events per week, across roughly 340 distinct issue groups
• Top 10 issues by volume were all the same three things: a bot hammering a deprecated endpoint, a ResizeObserver loop limit exceeded browser warning, and network aborts from users closing tabs mid-request
• The interesting stuff — a null deref that only fired for accounts created before a 2024 schema change — sat at rank 180 with 6 events

The math is what kills you. Say a careful engineer needs 4 minutes per issue to open the stack trace, find the corresponding code, and decide whether it's real. That's 22 hours to get through 340 issues once. Nobody has 22 hours, so the honest team behavior is: read the top 5, ignore the rest, and wait for a customer to complain about the rest.

I wanted to know whether the tail was actually worth reading. Not "can AI fix my bugs" — just: can an agent do the boring 4-minute pass, 340 times, well enough that I only look at what survives?

How I Solved It

The whole thing is about 200 lines of Python 3.13 and one carefully-shaped prompt. Four stages:

flowchart LR
    A[Error tracker API] --> B[Normalize to JSON]
    B --> C[Cluster by cause]
    C --> D[Agent verdict per cluster]
    D --> E{Real bug?}
    E -->|yes| F[Reproduce + failing test]
    E -->|no| G[Auto-mute with reason]

Stage 1: Get structured data, not screenshots

My first attempt was embarrassingly lazy — I pasted dashboard screenshots into a session and asked "what looks real here?" The answers were confident and useless, because a screenshot has the top frame of a stack trace and nothing else.

Pull the real payload instead. Every error tracker has a REST API; mine gives me issues plus their latest event:

import json, os, urllib.request

BASE = "https://errors.example-tracker.com/api/0"

def fetch(path: str):
    req = urllib.request.Request(
        f"{BASE}{path}",
        headers={"Authorization": f"Bearer {os.environ['TRACKER_TOKEN']}"},
    )
    with urllib.request.urlopen(req) as res:
        return json.loads(res.read())

def issue_payload(issue):
    event = fetch(f"/issues/{issue['id']}/events/latest/")
    frames = [
        f for f in event["stacktrace"]["frames"]
        if f.get("in_app")  # third-party frames are noise for triage
    ]
    return {
        "id": issue["id"],
        "title": issue["title"],
        "culprit": issue["culprit"],
        "count": issue["count"],
        "users_affected": issue["userCount"],
        "first_seen": issue["firstSeen"],
        "last_seen": issue["lastSeen"],
        "release": event.get("release"),
        "frames": [
            {"file": f["filename"], "line": f["lineno"], "fn": f["function"]}
            for f in frames[-6:]  # deepest 6 in-app frames
        ],
        "message": event.get("message", "")[:2000],
    }

The in_app filter matters more than anything else here. An unfiltered React stack trace is 40 frames of framework internals and 3 frames of my code, and every token spent on framework internals is a token not spent reasoning about my code.

Stage 2: Cluster by cause, not by fingerprint

Error trackers group by a fingerprint — usually a hash of the exception type plus the top frame. That's a syntactic grouping, and it splits one bug into many issues constantly. In my dataset, one date-parsing bug appeared as 9 separate issues because it threw from 9 different call sites.

So before triage, I have the agent do a cheap clustering pass over just the metadata (no code reading yet):

You will receive a JSON array of error issues.
Group them by ROOT CAUSE, not by exception type or stack frame.

Two issues share a root cause if fixing one line of code would
plausibly resolve both. Different call sites into the same broken
helper = same cause. Same exception type from unrelated modules
= different causes.

Return JSON: [{ "cause_label": str, "issue_ids": [int], "why": str }]
If you are unsure, keep them separate. Over-splitting is cheap;
over-merging hides bugs.

340 issues collapsed into 112 causes. That alone cut the expensive stage by two thirds.

Stage 3: Make it read the actual code

This is the step that turned the output from plausible to useful. For each cause cluster, I run Claude Code inside the repo so it can open the files named in the frames:

claude -p "$(cat prompts/triage.md)" \
  --append-system-prompt "You are triaging one production error cluster. \
Read the referenced files before forming any opinion. Never guess at \
code you have not opened." \
  < clusters/${cluster_id}.json

The difference is stark. Without repo access, on a TypeError: Cannot read properties of undefined (reading 'timezone'), I get:

This suggests the user object may be undefined. Consider adding a null check before accessing timezone.

With repo access, on the same error:

formatSlot() at src/scheduling/slots.ts:88 reads user.prefs.timezone. prefs is populated by hydrateUser(), which early-returns at line 41 when user.status === 'pending'. Invited-but-not-activated users therefore reach formatSlot() with prefs undefined. The 6 events all carry a release after 2026-06-02, which is when the invite flow started rendering the schedule preview.

One of those is a fortune cookie. The other is a bug report I can act on.

Stage 4: Force a verdict, and make "not a bug" a real option

My second big mistake: my first prompt asked "what's the fix for this error?" — and a model asked for a fix will always produce a fix. I got beautiful null checks for errors that were bots probing /wp-admin.

The fix is a schema where "this isn't worth fixing" is a first-class, equally valid answer:

VERDICT_SCHEMA = {
    "type": "object",
    "required": ["classification", "confidence", "evidence"],
    "properties": {
        "classification": {
            "enum": [
                "real_bug",           # our code is wrong
                "environment",        # browser quirk, extension, network abort
                "hostile_traffic",    # scanners, bots, probing
                "already_fixed",      # code path no longer exists on main
                "insufficient_data",  # cannot decide from what was provided
            ]
        },
        "confidence": {"enum": ["high", "medium", "low"]},
        "evidence": {
            "type": "array",
            "items": {"type": "string"},
            "description": "file:line references that justify the verdict",
        },
        "user_impact": {"type": "string"},
        "suggested_fix": {"type": ["string", "null"]},
    },
}

Two rules in the prompt do the heavy lifting:

Every evidence entry must be a file:line you actually opened. If you cannot cite code you have read, the classification must be insufficient_data.

insufficient_data is a correct and respected answer. A wrong real_bug costs an engineer an hour; an honest insufficient_data costs nothing.

Out of 112 clusters: 61 hostile traffic or environment, 28 already fixed (dead code paths still throwing from cached bundles), 12 insufficient data, 11 real bugs.

Stage 5: No fix without a failing test

For the 11 survivors, I didn't let the agent open a PR with a patch. It had to first write a test that fails on main for the reason described in the verdict:

claude -p "Write a failing test that reproduces this verdict. \
Run it. It MUST fail with the error described in the verdict, \
for the described reason. Do not modify source code in this step. \
If you cannot make it fail for that reason, output REPRO_FAILED \
and stop."

3 of the 11 came back REPRO_FAILED. Two of those three were misdiagnoses that read completely convincingly — the reasoning was internally coherent and pointed at the wrong function. The reproduce-first gate is the only thing that caught them, and it's the single most valuable rule in this whole pipeline.

The remaining 8 became PRs. 7 merged. Total agent cost for the run: about $14.

Lessons Learned

1. Syntactic grouping is not causal grouping. Your error tracker groups by stack hash because that's what it can compute cheaply. One bug scattered across 9 issues looks like 9 low-priority nuisances; merged, it's a P1. The cheap metadata-only clustering pass was the highest leverage 20 lines in the project.

2. Volume is the worst possible priority signal. My noisiest issue had 3,100 events and zero user impact. My most expensive bug had 6 events and blocked every invited user from seeing their schedule. Sort by users_affected × "is this on a path where someone spends money," never by raw count.

3. A stack trace without the source is a horoscope. Both are vague enough to feel true and unfalsifiable enough to be safe. Give the agent the repo, tell it to open the files, and require file:line citations — the quality jump isn't incremental, it's categorical.

4. If you don't make "no action" a valid output, you'll get action. This generalizes way past error triage. Any time you ask a model for a fix, a finding, or a recommendation, you have to build an equally respectable escape hatch or you're just measuring its willingness to produce output. Naming insufficient_data and explicitly saying it was a good answer cut my false positives more than any prompt tuning did.

5. Reproduce before you fix — no exceptions. 3 of 11 verdicts evaporated at the reproduction step. Confident, well-cited, coherent, and wrong. The failing test is the only artifact in this whole chain that can't be argued with, and it's what makes the output trustworthy enough to stop reviewing every verdict by hand.

What's Next

Two directions I'm working on:

• Run it on new issues, not batches. The 340-issue backfill was the interesting experiment, but the real value is a triage verdict attached to an issue within an hour of it first appearing, while the release that caused it is still obvious.
• Feed verdicts back as calibration data. I now have 112 verdicts and 8 merged fixes. That's a small but real eval set for testing whether a prompt change makes triage better or just different — which is a question I currently answer by vibes.

The broader lesson I keep re-learning: agents are excellent at the boring 4-minute pass you'd never do 340 times, and mediocre at the judgment call you'd make in 10 seconds. Build for the first thing and gate the second.

Wrap-up

If your error tracker has a tail you've never read, there's a decent chance there's a real bug in it. Mine had 11.

If you try this, I'd genuinely like to know your hit rate — drop it in the comments, especially if it's low, because that's the more interesting data point.

Follow me here on Dev.to for more write-ups on building with AI coding agents, and if you want to try the pipeline yourself, Claude Code plus your tracker's REST API is the entire dependency list. 🚀

Comments

[Skillselion]

The in_app frame filter is the quiet hero here. In our own agent pipelines the verdicts only became trustworthy once we forced a falsifiable next step per item: either concrete reproduce steps referencing a file the model saw, or a mute reason a human could veto in one line. A bare real/not-real verdict drifts optimistic on exactly the kind of rank-180 null deref you describe, because nothing punishes a wrong "not real". In our runs, most of the precision gain came from tightening that verdict schema. Did you keep the auto-mute reasons anywhere reviewable? That log sounds like the most valuable artifact the pipeline produces.

[Kane Lim]

This is a really solid approach to production error triage. I especially like the decision to treat insufficient_data as a valid outcome and require a reproducible failing test before allowing a fix. That adds an important verification layer instead of letting the agent turn every error into a speculative code change.

The causal clustering idea is also interesting because fingerprint-based grouping can easily fragment the same underlying defect across multiple call sites. Combining production telemetry with repository context seems much more useful than analyzing stack traces in isolation.

One question I’m curious about: how are you planning to measure the false-negative rate bugs that the pipeline incorrectly classifies as noise or insufficient data?

And for the continuous version, would you consider feeding the final human/PR outcome back into the pipeline as an evaluation dataset to continuously measure triage accuracy?