DEV Community

Cover image for AI Agent Incident Response: What to Capture Before It Scrolls Away
Gabriel Anhaia
Gabriel Anhaia

Posted on

AI Agent Incident Response: What to Capture Before It Scrolls Away

#typescript #agents #ai #node

An AI incident does not look like a normal one. There is no stack trace, no
failing health check. There is a customer saying the agent told them something
false, or an audit log showing forty refunds in an hour that should have been
four.

The investigation then depends entirely on what you happened to be logging
before it happened. Nothing is reproducible — same input, different run, so
the trace is the evidence, and if it was not captured it does not exist.

The record has to be enough to reconstruct the decision

Per run, one row:

export type RunRecord = {
  runId: string;
  userId: string;
  startedAt: string;
  finishedAt: string;

  // what produced this output
  model: string;
  promptVersion: string;
  toolsetVersion: string;
  temperature: number;

  // what the agent saw
  input: string;
  retrievedDocIds: string[];        // ids, not bodies
  contextTokens: number;

  // what it did
  toolCalls: { name: string; argsHash: string; ok: boolean; ms: number }[];
  turns: number;
  outcome: Outcome;
  costUsd: number;

  // who allowed it
  approvals: { by: string; at: string; verdict: string }[];
  capabilities: string[];
};
Enter fullscreen mode Exit fullscreen mode

The four version fields are what turn "quality dropped last Tuesday" from an
argument into a query. retrievedDocIds is what lets you ask whether the
agent was reading a poisoned document — the single most common cause of an
agent behaving strangely for one tenant and fine for everyone else.

capabilities answers "how was it even able to do that", which is usually the
second question after "what did it do".

Store the full trace separately, with a shorter life

The record above is small enough to keep for a year. The full transcript is
not, and it is also the part containing customer data.

await traceStore.put(runId, {
  messages: redact(messages),
  toolResults: toolResults.map((r) => ({ ...r, body: truncate(r.body, 4_000) })),
}, { ttlDays: 30 });
Enter fullscreen mode Exit fullscreen mode

Thirty days covers essentially every investigation. Longer, and you are
holding customer conversations you have no product reason to hold.

Redaction at write time, not at read time:

const PATTERNS = [
  [/\b[\w.+-]+@[\w-]+\.[\w.]+\b/g, "[email]"],
  [/\b(?:\d[ -]*?){13,19}\b/g, "[card]"],
  [/\b(sk|pk|ghp|xox[baprs])[-_][A-Za-z0-9]{16,}\b/g, "[secret]"],
] as const;
Enter fullscreen mode Exit fullscreen mode

Imperfect, and still far better than the alternative. Anything genuinely
sensitive should be referenced by id and never enter the transcript in the
first place.

A compact run record kept long-term alongside a redacted full trace with a<br> short<br> TTL.

One id, threaded everywhere 

res.setHeader("X-Run-Id", runId);
logger.info("agent", { runId, ... });
await db.refund.create({ data: { ..., runId } });
await mailer.send({ ..., headers: { "X-Run-Id": runId } });
Enter fullscreen mode Exit fullscreen mode

The reason to put it on the effect — the refund row, the sent email — is
that incidents start from the effect. Someone found a wrong refund. Without a
runId column on that table, connecting it to a run is a timestamp-matching
exercise across two systems.

The first ten minutes

Contain before you diagnose. The flag from your rollout should still
exist:

await flags.update("agent-v1", { mode: "suggest" });
Enter fullscreen mode Exit fullscreen mode

One rung down puts a human in front of the output without taking the feature
away. If the effect is irreversible and ongoing, go to off.

Find the blast radius.

SELECT run_id, user_id, outcome, cost_usd
FROM run_records
WHERE started_at > now() - interval '6 hours'
  AND tool_calls @> '[{"name": "refund_order"}]'
ORDER BY started_at;
Enter fullscreen mode Exit fullscreen mode

Answering "how many" before "why" is what lets you tell support something true
in the first half hour.

Freeze the evidence. Traces have a TTL, and an investigation that runs
past it loses its own material:

await traceStore.hold(runIds, { until: addDays(new Date(), 180) });
Enter fullscreen mode Exit fullscreen mode

Do this early. It is the step most often remembered on day 29.

Reproduce from the checkpoint, not from the input

Re-running the input gives you a different run and tells you nothing.
Replaying the stored state is the closest thing to a reproduction available:

const history = [];
for await (const s of graph.getStateHistory({ configurable: { thread_id: runId } })) {
  history.push(s);
}

const before = history.find((s) => s.next.includes("refund_order"));
await graph.invoke(null, before.config);      // with side effects disabled
Enter fullscreen mode Exit fullscreen mode

With side effects disabled — a dry-run tool wrapper, because otherwise
investigating a wrong refund issues more wrong refunds.

Fork with one value changed to test a hypothesis. That is how you distinguish
"the model misread correct data" from "the data in state was already wrong",
and those have completely different fixes.

The three causes worth checking first

Injected instructions. Pull the retrieved documents for the run and read
them. Content written by the person who benefits from the agent's action is
the classic case, and retrievedDocIds is what makes it a two-minute check.

A silent upstream change. Compare model and promptVersion against runs
from before the incident window. Provider-side changes land without a deploy.

A capability that was too wide. Look at capabilities and ask whether the
action should have been possible at all. Frequently the honest finding is that
the model behaved reasonably given tools it should never have had.

Three first-line hypotheses checked against fields already present in the<br> run<br> record.

Write-ups that are worth writing

The action items that hold up are not "improve the prompt". Prompts are not a
control — they influence behaviour, they do not constrain it.

The ones that hold up look like:

  • narrowed the capability so the action is no longer possible unapproved - added an amount ceiling enforced in the tool, not requested in the prompt - added the failing case to the golden set so a regression is caught nightly - added the missing field to the run record so the next investigation is faster

That last one is the compounding item. Every incident reveals a field you
wished you had; adding it is how the next investigation takes an hour instead
of a day.

If this was useful

AI That Ships covers running AI
features in production — trace and audit design, redaction and retention,
containment, replay-based investigation, and the follow-ups that actually
prevent recurrence.

AI That Ships — Taking AI Features to Production

The full series is at
xgabriel.com/ai-in-typescript.

Top comments (0)