DEV Community

Akash Pal
Akash Pal

Posted on

Part 6: Observability for AI Agents: Tracing, Metrics, and Drift

#ai #typescript #observability #llm

Part 6 of a series building a support-ticket agent with no framework. Previous: Part 5 (guardrails). Repo: github.com/akash-pal/agent-from-scratch

"Run the eval set" and "is this agent healthy right now" are different questions, and it's easy to only build infrastructure for the first one. Eval sets run offline, on cases you already thought of. Production traffic doesn't ask permission to send you a ticket type you didn't anticipate. Observability is what tells you when that's happening — and it's also, unglamorously, what makes offline evaluation possible in the first place: you can't debug a failing eval case without knowing what the agent actually did, step by step.

The minimum trace payload

Every tool call in this build logs a structured record — src/trace.ts:

export interface TraceStep {
  trace_id: string;
  step_id: number;
  tool_name: string;
  args_hash: string;        // hashed, never raw args
  duration_ms: number;
  result_summary: string;
  model: string;
  token_usage: { input: number; output: number };
}
Enter fullscreen mode Exit fullscreen mode

Two details here that look small and aren't:

args_hash, not raw args. This trace log is meant to be safe to keep around, ship to a monitoring system, or paste into a bug report — none of which should require thinking about what secrets might be embedded in a tool call's arguments. Hashing means you can still confirm two calls used identical arguments (for debugging idempotency, for instance) without ever persisting the actual values:

export function hashArgs(args: Record<string, unknown>): string {
  return "sha256:" + createHash("sha256").update(JSON.stringify(args)).digest("hex").slice(0, 8);
}
Enter fullscreen mode Exit fullscreen mode

result_summary, truncated. Full tool results can be large (a kb_search returning full article bodies, for instance) — logging the whole thing on every step makes trace output unreadable and bloats whatever's storing it. summarizeResult takes the first few fields and truncates long values:

const MAX_FIELD_LEN = 70;
export function summarizeResult(result: Record<string, unknown>): string {
  const entries = Object.entries(result).slice(0, 4);
  return entries.map(([k, v]) => `${k}=${truncate(JSON.stringify(v))}`).join("  ");
}
Enter fullscreen mode Exit fullscreen mode

Readable in a terminal, structured everywhere else

The trace log doubles as CLI output — this build's whole point is being inspectable, so watching an agent run in real time matters. Early on, that meant a raw JSON blob per line, which is technically complete and practically unreadable. The fix was a small formatting pass, not a new logging system:

export function logTrace(step: TraceStep): void {
  const timing = `${step.duration_ms}ms, ${step.token_usage.input}→${step.token_usage.output} tok`;
  console.log(`  ${DIM}[${step.step_id}]${RESET} ${CYAN}${step.tool_name}${RESET} ${DIM}(${timing})${RESET}`);
  console.log(`      ${step.result_summary}`);
}
Enter fullscreen mode Exit fullscreen mode

Output in an actual terminal:

  [1] order_lookup (0ms, 1077→23 tok)
      order_id="ord_1005"  status="processing"  items=[...]  total_usd=45
  [2] kb_search (1ms, 1268→20 tok)
      articles=[...]  relevance_scores=[0.48,0.24,0.24]
Enter fullscreen mode Exit fullscreen mode

Colors auto-disable when stdout isn't a real TTY (process.stdout.isTTY), so piping this to a file or a CI log doesn't leave you with literal escape-code garbage — small thing, but the kind of small thing that makes the difference between a trace log people actually read and one they ignore.

Three production monitoring layers

Trace-per-step is the foundation, but three distinct layers sit on top of it, each answering a different question:

  • Trace logging — every run, every tool call: input, response, latency, cost. This is what you already have from the payload above.
  • Online metrics — aggregate numbers over time: success rate, escalation rate, average tool calls per run, average token cost per run. This is where you'd notice, for instance, that escalation rate crept from 8% to 15% over a week.
  • Drift detection — compare those metrics week over week. This one's easy to skip and shouldn't be: model provider updates cause silent regressions with zero code changes on your side. The agent that scored 95% on your eval set last month can start failing differently this month because the underlying model changed, not because anything you wrote did.

This repo doesn't implement online metrics or drift detection — it's a CLI reference build with no persistent request volume to aggregate — but the trace payload is written specifically so those layers could be built on top of it without changing the tracing code itself. That's the actual design goal: the minimum payload isn't "the metrics you need now," it's "the raw material any metrics system would need later."

Why this matters for debugging, not just monitoring

Go back to Part 3's eval failure:

[FAIL] hard_04 (hard)
    - trajectory: expected [order_lookup, refund_eligibility, issue_refund] as a subsequence, got [order_lookup, refund_eligibility]
Enter fullscreen mode Exit fullscreen mode

That failure message exists because of tracing, full stop. Without a structured, step-by-step record of what tools got called, "the eval failed" would be all you'd know — not why. The actual bug behind that failure (Part 5's phantom refund proposal) was findable specifically because the trajectory was visible, not just the final pass/fail.

What's next

Part 7: Iterating to Green: Real Bugs, and When You'd Actually Reach for a Framework → closes the series: the full iteration log — every real bug found running this agent against the eval set, what fixed each one, and when you'd actually reach for a framework instead of this raw loop.

Repo: github.com/akash-pal/agent-from-scratch

Top comments (1)

Collapse
 
reidmarlow profile image
Reid Marlow •

The args_hash choice is the bit I wish more agent demos copied. Full traces are useful until the first customer email, token, or raw tool payload leaks into a log that gets pasted into Slack. Do you also keep a separate private trace for replay, or is the hash-only version enough for this build?