DEV Community

Cover image for How to Know Your AI Agent Got Worse Before Your Users Tell You
Gabriel Anhaia
Gabriel Anhaia

Posted on

How to Know Your AI Agent Got Worse Before Your Users Tell You

#typescript #agents #ai #monitoring

Conventional monitoring answers "is it up". For an agent, it is always up. The
degradation you care about looks like: same status codes, same latency,
answers that are a bit worse, tool calls that are a bit more often wrong.

Nobody pages you. Three weeks later someone in support says "it's been useless
lately", and nobody can say when it started.

Two things fix that. Proxy signals that move before anyone complains, and a
fixed test set you can rerun to confirm what the proxies suggest.

Four proxies that move first

None of these measures quality directly. All of them correlate with it, and
all are free to collect.

1. Outcome distribution. Every run ends in exactly one terminal state.

export type Outcome =
  | "complete" | "refused" | "gave_up"
  | "max_turns" | "budget_exceeded" | "tool_failed" | "timeout";

metrics.increment("agent.outcome", 1, { outcome, model, promptVersion });
Enter fullscreen mode Exit fullscreen mode

The share of complete is the single most useful number on the dashboard. It
moves for real reasons — a degraded model, a broken tool, a prompt change, and
it moves days before anyone files a ticket.

2. Turns to completion. An agent that needs more steps for the same work
is getting worse at the work.

metrics.histogram("agent.turns", turns, { intent, model });
Enter fullscreen mode Exit fullscreen mode

Watch p50, not the mean. A p50 moving from 4 to 6 is a real regression even
while the mean stays flat.

3. Tool-call validity. How often the model produces arguments that fail
schema validation.

const parsed = schema.safeParse(call.input);
metrics.increment("agent.tool_args", 1, {
  tool: call.name,
  valid: String(parsed.success),
});
Enter fullscreen mode Exit fullscreen mode

This one is unusually sensitive. A model that has quietly changed will fail
schema validation at a measurably higher rate before its prose gets noticeably
worse.

4. Retry and self-correction rate. How often the agent calls the same tool
twice with slightly different arguments, or contradicts itself between turns.

const repeats = countRepeatedCalls(trace);        // same tool, similar args
metrics.histogram("agent.repeat_calls", repeats, { model });
Enter fullscreen mode Exit fullscreen mode

Four proxy signals moving before any user report<br> arrives.

The golden set confirms what proxies suspect

Proxies tell you something changed. They cannot tell you the answers got
worse. For that you need a fixed set of inputs with known-good outputs, rerun
on a schedule.

export type GoldenCase = {
  id: string;
  input: string;
  mustCall?: string[];          // tools that should be used
  mustNotCall?: string[];       // tools that must not be
  assert: (out: AgentOutput) => Promise<Verdict>;
};
Enter fullscreen mode Exit fullscreen mode

The assertion is the hard part, and it should be as deterministic as you can
make it. Prefer checkable properties over judging prose:

{
  id: "refund-under-limit",
  input: "I want a refund for order ord_4471, it arrived broken",
  mustCall: ["get_order", "refund_order"],
  mustNotCall: ["send_email"],
  assert: async (out) => {
    const call = out.toolCalls.find((c) => c.name === "refund_order");
    if (!call) return fail("no refund attempted");
    if (call.input.orderId !== "ord_4471") return fail("wrong order");
    if (call.input.amountCents !== 8900) return fail("wrong amount");
    return pass();
  },
}
Enter fullscreen mode Exit fullscreen mode

Structural assertions like these are stable across model versions and reruns.
Assertions on wording are not — they fail on paraphrase and train you to
ignore the suite.

For cases where output quality genuinely is prose, use a model grader with a
rubric, and accept that it is noisier. Run those cases three times and take
the majority.

Run it against production config, on a schedule 

// nightly, plus on every deploy that touches prompts, tools or model
export async function runGolden(): Promise<GoldenReport> {
  const cases = await loadCases();
  const results = await pMap(cases, async (c) => {
    const out = await runAgent(c.input, prodLikeCtx());   // real prompts, stub tools
    return { id: c.id, verdict: await c.assert(out), turns: out.turns };
  }, { concurrency: 4 });

  const passed = results.filter((r) => r.verdict.ok).length;
  metrics.gauge("golden.pass_rate", passed / results.length);
  return { results, passRate: passed / results.length };
}
Enter fullscreen mode Exit fullscreen mode

Stubbed tools with recorded responses, so the suite measures the model and the
prompt rather than the weather in your integrations. Real system prompt, real
tool schemas, real model id.

Nightly matters because the thing you are guarding against is change you did
not make. A provider-side change lands without a deploy, and a suite that only
runs in CI will not see it.

Alert on the delta, not the level 

const today = await runGolden();
const base = await loadBaseline();

const regressed = base.results.filter((b) =>
  b.verdict.ok && !today.results.find((t) => t.id === b.id)?.verdict.ok);

if (regressed.length >= 2) {
  await pager.notify(`golden: ${regressed.length} cases regressed`, {
    cases: regressed.map((r) => r.id),
    model: process.env.MODEL_ID,
  });
}
Enter fullscreen mode Exit fullscreen mode

Naming the specific cases is what makes the page actionable at 3am. "Pass rate
dropped to 87%" prompts a shrug; "refund-under-limit and
refund-over-limit-needs-approval both regressed" is a diagnosis.

Two cases rather than one because a single flaky case will fire eventually and
teach everyone to ignore the alert.

Version everything the output depends on 

logger.info("agent run", {
  runId, outcome, turns, costUsd,
  model: cfg.model,
  promptVersion: cfg.promptVersion,
  toolsetVersion: cfg.toolsetVersion,
  retrievalIndexVersion: cfg.indexVersion,
});
Enter fullscreen mode Exit fullscreen mode

When quality drops, the first question is what changed. Four version fields on
every run turn that into a query. Without them it is an archaeology exercise
across deploy logs and provider changelogs.

A golden-set run compared against its baseline, naming the specific cases<br> that<br> regressed.

The cheapest signal of all

Ask. One thumb up/down on the output, stored with the run id.

await db.feedback.create({ data: { runId, verdict, userId } });
Enter fullscreen mode Exit fullscreen mode

Response rates are low and the sample is biased toward the annoyed. It is
still the only signal that comes from the person who knows whether the answer
was useful, and joined to the run trace it turns a complaint into a
reproducible case, which is exactly what your golden set needs more of.

If this was useful

AI That Ships covers evaluation and
monitoring for AI features — proxy metrics, golden sets and structural
assertions, regression alerting, and the versioning that makes "what changed"
answerable.

AI That Ships — Taking AI Features to Production

The full series is at
xgabriel.com/ai-in-typescript.

Top comments (0)