The Observability You Need Before Your First 1,000 AI Users

#typescript #ai #observability #node

Book: AI That Ships
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Advice about LLM observability tends to assume you already have
scale, a tracing backend, and someone to own it. Before your first
thousand users you have none of those, and adopting a full tracing
stack is a week you do not have.

The good news is that most of the debugging value comes from five
things, they are cheap, and they go in a specific order — each one is
useful before the next exists.

1. A run id through everything

Nothing else works without this. One identifier per user-facing
operation, present on every log line from every layer.

import { AsyncLocalStorage } from "node:async_hooks";

export type RunCtx = { runId: string; userId: string; feature: string };
export const runCtx = new AsyncLocalStorage<RunCtx>();

export function withRun<T>(ctx: RunCtx, fn: () => Promise<T>) {
  return runCtx.run(ctx, fn);
}

export function log(event: string, fields: Record<string, unknown> = {}) {
  const ctx = runCtx.getStore();
  logger.info(event, { ...ctx, ...fields });
}

AsyncLocalStorage is what makes this bearable — no context
parameter threaded through every function, and it survives await
boundaries.

Without a run id you have log lines that cannot be assembled into a
story. A user says "it gave me a wrong answer at about 3pm" and you
have model calls, tool calls, and errors that cannot be related to
each other.

This is the one to do first, and it takes twenty minutes.

2. Tokens and cost per run

export function recordModelCall(model: string, usage: Usage, ms: number) {
  const cost = costOf(model, usage);
  const ctx = runCtx.getStore();
  if (ctx) {
    tallies.get(ctx.runId).cost += cost;
    tallies.get(ctx.runId).calls += 1;
  }

  log("model_call", {
    model,
    inputTokens: usage.input_tokens,
    outputTokens: usage.output_tokens,
    cacheReadTokens: usage.cache_read_input_tokens ?? 0,
    costUsd: +cost.toFixed(6),
    ms,
  });
}

Per call and accumulated per run. Cost is the metric that turns
vague worry into a decision — which feature is expensive, whether
cost per run is drifting upward, whether prompt caching is doing
anything.

Log cache tokens separately. A cache hit rate you assumed was high
and is not is common, and invisible if you only log a total.

3. Tool outcomes

export async function instrumentTool<T>(
  name: string,
  args: unknown,
  fn: () => Promise<T>,
): Promise<T> {
  const started = performance.now();
  try {
    const out = await fn();
    log("tool_call", {
      tool: name, ok: true,
      ms: Math.round(performance.now() - started),
      argKeys: Object.keys(args as object),
      resultBytes: JSON.stringify(out).length,
    });
    return out;
  } catch (err) {
    log("tool_call", {
      tool: name, ok: false,
      ms: Math.round(performance.now() - started),
      errorType: err?.constructor?.name ?? "unknown",
    });
    throw err;
  }
}

argKeys rather than the arguments. Key names tell you the shape;
values are where the PII is.

resultBytes earns its place quickly. A tool returning large results
is the usual cause of an unexplained cost increase, because the
result is resent in context on every subsequent turn.

Once you have this, "why is this run expensive" and "why is this run
slow" are usually answered by a single query grouping tool calls by
name.

4. Prompt version on every call

export const PROMPTS = {
  support: { id: "support", version: 12, text: "..." },
} as const;

log("model_call", {
  promptId: PROMPTS.support.id,
  promptVersion: PROMPTS.support.version,
  // ...
});

Without this, "quality dropped last Tuesday" is unanswerable. With
it, you can compare outcomes before and after a prompt change, and
you can answer "what produced this stored output" for any record you
kept.

A content hash works as well as a number and cannot be forgotten:

const promptHash = createHash("sha256")
  .update(PROMPTS.support.text).digest("hex").slice(0, 8);

5. One sampled transcript per failure class

Storing every transcript is expensive and mostly useless. Storing
none means every investigation starts from nothing.

Sample by outcome:

const SAMPLE_RATES: Record<Outcome, number> = {
  error: 1.0,          // keep all
  budget_stop: 1.0,
  turn_limit: 1.0,
  low_rating: 1.0,
  complete: 0.01,      // 1% of successes
};

export async function maybeStoreTranscript(run: RunSummary, msgs: MessageParam[]) {
  if (Math.random() > SAMPLE_RATES[run.outcome]) return;
  await transcripts.put(run.runId, {
    ...run,
    messages: redactAll(msgs),
    ttlDays: run.outcome === "complete" ? 7 : 30,
  });
}

Every failure, one percent of successes. Failures are what you
investigate; successes are the baseline you occasionally need to
compare against.

The successful sample matters more than it looks — most "the agent is
behaving differently" reports are resolved by comparing a failing run
to a normal one.

Redaction before anything leaves the process

const RULES = [
  { name: "card", re: /\b(?:\d[ -]*?){13,19}\b/g },
  { name: "email", re: /\b[\w.+-]+@[\w-]+\.[\w.]{2,}\b/g },
  { name: "bearer", re: /\bBearer\s+[\w-.]+/gi },
];

export function redactAll(msgs: MessageParam[]): MessageParam[] {
  return msgs.map((m) => ({
    ...m,
    content: typeof m.content === "string"
      ? redact(m.content)
      : m.content.map(redactBlock),
  }));
}

Two rules with no exceptions.

Redact before the data leaves the process, not in the logging
backend. A pipeline that redacts downstream has already copied the
raw value into a queue, a buffer, and possibly a vendor.

Never log raw tool arguments by default. The tool that takes an
email or an address is the one whose arguments end up in a
screenshot pasted into Slack.

Regex redaction is partial. It reduces exposure; it does not make a
transcript safe to share freely. Set a TTL and restrict access
accordingly.

The order, and why

Run id, then cost, then tool outcomes, then prompt version, then
sampled transcripts.

Each is useful before the next exists. Run id makes every later
signal joinable. Cost is what management asks about first. Tool
outcomes explain most slowness and most expense. Prompt version turns
"something changed" into a comparison. Transcripts are what you read
when the aggregate does not explain it.

Doing them in the other order — starting with full transcript
capture, which is the tempting one — gives you a lot of data you
cannot query and a privacy surface you did not plan.

What this is not

Not distributed tracing. Not spans, not a vendor, not
OpenTelemetry semantic conventions.

Those are worth adopting once you have several services, real
traffic, and someone who owns the backend. Before that, five
structured log fields and a sampled transcript store answer nearly
every question you will actually have — and they take an afternoon
rather than a sprint.

When you do move to tracing, the run id becomes your trace id and
almost nothing else changes. Which is the other reason to start
there.

If this was useful

AI That Ships covers the
operational side of shipping AI on Node — cost accounting, evals,
guardrails, deployment, and the observability that makes the rest
debuggable.