LLM-as-Judge: Auto‑Annotate & Triage AI Failures

Turn Override Signals into Fixes: An LLM-as‑Judge Pipeline for Auto‑Annotation and Triage

User overrides are one of the richest — and most underused — signals you have for improving LLM-driven features. Rather than treating overrides as anecdotes, you can instrument, sample, and automatically annotate traces so engineering teams get ship-ready, prioritized work. In this article I walk through a concrete pipeline I use in production that leverages LLM-as-judge automatic annotation to convert high-override traces into prioritized tickets with example sessions attached.

Why override signals matter

When a user rejects or modifies an AI suggestion you shipped, that interaction is a direct expression of friction: something in the model, prompt, or orchestration produced an unwanted outcome. Unlike synthetic benchmarks, overrides capture real user intent, real state, and repetitive pain. Treat the sequence suggestion_shown → suggestion_accepted/modified/overridden in Mixpanel, Amplitude, or your telemetry as a defect funnel. The traces with the highest override rate are your goldmine.

But raw traces are noisy. An engineer needs to know why the suggestion failed, which step in the trajectory matters, and whether fixes exist that don’t require a full model retrain. That's where an LLM-as-judge and a step-wise rubric come in.

Pipeline overview — from trace to ticket

High level steps:

• Instrument suggestion_shown / accepted / modified events and capture full session traces (model calls, tool calls, user edits). Use an observability tool that records trajectories (LangSmith, Langfuse, or your event store).
• Aggregate and compute override rates per flow, prompt, or UI surface. Treat the top-N items by override rate as your defect funnel.
• Sample traces from those top-N buckets and feed them to an LLM-as-judge using a step-wise rubric (trajectory-aware evaluation, not outcome-only).
• The judge emits: verdict (faulty / ok), failure type, failed step pointer, one-line rationale, and a confidence score.
• Cluster labeled failures, rank clusters by impact (override frequency × time cost × confidence), and auto-open prioritized tickets with one exemplar session attached.

The outcome: engineers receive tickets with clear evidence, a labeled failure mode, a pointer to the failed step, and a tiny rationale — everything needed to triage quickly.

Designing a step-wise rubric (why it matters)

Outcome-only evaluation (show the judge request + final answer) misses silent failures: wrong tool used, skipped precondition, ignored retrieved evidence, or poor state-tracking. A step-wise rubric forces the judge to inspect the trajectory at each transition and label which step went wrong and why.

Good rubric properties:

• Per-step questions (e.g., "Was the tool call appropriate?", "Did the agent respect prior state?", "Was the returned output used correctly?").
• Mutually exclusive failure categories that map to engineering ownership: prompt, state-tracking, tool misuse, hallucination, guardrail lapse.
• A required one-line rationale and a failed-step pointer (e.g., "tool_call[3] used wrong params").
• A confidence score and structured fields (avoid free-form verdicts).

G-Eval and trajectory-judge style approaches show that step-rubric judges dramatically increase recall for silent failures and improve localisation.

Automating the judge: structured outputs and calibration

When you run an LLM-as-judge at scale, enforce structured outputs (JSON schema / function call mode). Don’t parse prose with regex. Request a typed object like:

{ "verdict": "faulty|ok", "failure_type": "string", "failed_step": "string", "confidence": 0.0-1.0, "rationale": "string" }

Also ask for a confidence value and, where feasible, probability-weighted scoring (G-Eval style) rather than argmax tokens. That improves calibration and reduces silent metric corruption.

Code example (Python-like pseudocode)

This example shows sampling the top-N high-override traces, running an LLM-as-judge, and opening tickets for high-confidence faults.

# pseudo-code: adapt to your SDK and LLM provider
for bucket in top_n_override_buckets:
    traces = sample_traces(bucket, k=50)
    for trace in traces:
        verdict = judge.label(
            trace,
            rubric="step_rubric",
            output_schema={
                "verdict":"string",
                "failure_type":"string",
                "failed_step":"string",
                "confidence":"float",
                "rationale":"string"
            }
        )

        if verdict.confidence > 0.9 and verdict.verdict == "faulty":
            # attach the raw trace/session so triage is fast
            create_ticket(
                title=f"{verdict.failure_type} — high-override cluster",
                body=compose_ticket_body(trace, verdict),
                labels=[verdict.failure_type, "auto-triaged"],
                example_trace=trace
            )

This keeps the loop simple: sample, judge, cluster, open tickets for high-confidence clusters.

Clustering and prioritization

After labeling, cluster failures using the judge's failure_type and surface similarity on the failed_step + rationale. Rank clusters by impact: frequency in the override funnel, average time cost per user (if available), and average confidence. TRIAGE-Bench–style ranking (smallest single-step fix first) is useful — it pushes teams to ship the minimal change that reduces override volume.

Attach the highest-confidence exemplar to the ticket so engineers can replay real sessions. That context speeds triage enormously — a one-line rationale plus failed-step pointer often saves 10x time.

Practical production notes and pitfalls

• Routing failures: if your rubric lacks categories for state-tracking or guardrail misfires, you'll get many flagged defects that have no owner. Iterate the taxonomy with product and infra teams.
• Shipping gate wiring: ensure the defects flagged by the judge actually surface in a fix-gating workflow. Otherwise they sit in dashboards and never get resolved.
• Sandbox/runtime access: giving the judge sandboxed execution or runtime access to a simulator (AutoTriage-style) measurably improves attribution and reduces mislabeling.
• Calibration: monitor judge confidence calibration (ECE). If confidence is uncalibrated, adjust prompts, use probability-weighted scoring, or bootstrap with human labels.
• Cost vs. value: step-rubric evaluation costs more per trace but often pays for itself by finding silent faults rules miss.

Start small, ship fast

You don’t need a full model retrain to start shipping value. Begin by surfacing clusters that repeatedly cost user time, attach real sessions, and commit the smallest single-step fix your TRIAGE-Bench ranking suggests. In my experience this shortens repair loops and forces clearer ownership: a single engineer can own a fix for a cluster tied to a failed-step.

If you try this: require the judge to include a one-line rationale and a failed-step pointer. That tiny context makes triage an order of magnitude faster.

Conclusion

LLM-as-judge automatic annotation is not magic. It’s an engineering discipline: instrument, sample, label with a step-wise rubric, cluster, and ship. When you treat high-override traces as your defect funnel and automate the evidence collection and ticket creation, you turn anecdote into action and shorten the path from user pain to product improvement.

What override signal in your product would you most want auto-triaged tomorrow? Share a use case and I’ll suggest a rubric tweak you can implement quickly.