DEV Community: Raju Dandigam

From Flat Logs to Execution Trees: Debugging Modern AI Agents

Raju Dandigam — Wed, 19 Aug 2026 23:40:08 +0000

An agent trace is usually written as a sequence of events because append-only data is simple to produce:

span_started
span_started
span_ended
span_started
span_ended
span_ended

Developers do not want to debug that sequence directly. They want to see the causal structure:

research_agent
├─ search_web
├─ query_database
├─ call_finance_api
│  ├─ attempt_1          timeout
│  └─ attempt_2          ok
└─ summarize_results

The event stream is optimized for writing. The execution tree is optimized for understanding. Building a reliable tree requires more than sorting by timestamp: events may arrive out of order, siblings may run concurrently, spans may be incomplete, and retries may fail while the parent operation still succeeds.

The Minimum Event Contract

Every event needs stable trace and span identity. Start events establish parentage; end events establish outcome and duration.

type SpanKind = 'run' | 'model' | 'tool' | 'retrieval' | 'decision' | 'fallback';

type TraceEvent =
  | {
      event: 'span_started';
      traceId: string;
      spanId: string;
      parentSpanId: string | null;
      name: string;
      kind: SpanKind;
      timestampMs: number;
    }
  | {
      event: 'span_ended';
      traceId: string;
      spanId: string;
      timestampMs: number;
      status: 'ok' | 'error' | 'cancelled';
      errorCategory?: string;
      metadata?: Record<string, string | number | boolean | null>;
    };

Timestamps alone cannot establish parentage. Two events that occur next to each other may be siblings, unrelated concurrent work, or operations from different traces.

Assemble Events Into Spans

Start and end events may not arrive together. A buffered exporter can deliver the end first, and a crashed process may never deliver an end at all.

Represent assembly state explicitly:

type AssembledSpan = {
  traceId: string;
  spanId: string;
  parentSpanId: string | null;
  name: string;
  kind: SpanKind;
  startedAtMs: number;
  endedAtMs?: number;
  status: 'open' | 'ok' | 'error' | 'cancelled';
  errorCategory?: string;
  metadata: Record<string, string | number | boolean | null>;
};

type AssemblyDiagnostic = {
  code:
    | 'duplicate_start'
    | 'duplicate_end'
    | 'end_without_start'
    | 'span_left_open';
  spanId: string;
};

class SpanAssembler {
  private readonly spans = new Map<string, AssembledSpan>();
  private readonly pendingEnds = new Map<
    string,
    Extract<TraceEvent, { event: 'span_ended' }>
  >();
  readonly diagnostics: AssemblyDiagnostic[] = [];

  accept(event: TraceEvent): void {
    if (event.event === 'span_started') {
      if (this.spans.has(event.spanId)) {
        this.diagnostics.push({ code: 'duplicate_start', spanId: event.spanId });
        return;
      }

      const span: AssembledSpan = {
        traceId: event.traceId,
        spanId: event.spanId,
        parentSpanId: event.parentSpanId,
        name: event.name,
        kind: event.kind,
        startedAtMs: event.timestampMs,
        status: 'open',
        metadata: {},
      };

      this.spans.set(event.spanId, span);

      const pending = this.pendingEnds.get(event.spanId);
      if (pending) {
        this.pendingEnds.delete(event.spanId);
        this.applyEnd(span, pending);
      }

      return;
    }

    const span = this.spans.get(event.spanId);
    if (!span) {
      if (this.pendingEnds.has(event.spanId)) {
        this.diagnostics.push({ code: 'duplicate_end', spanId: event.spanId });
        return;
      }

      this.pendingEnds.set(event.spanId, event);
      return;
    }

    if (span.status !== 'open') {
      this.diagnostics.push({ code: 'duplicate_end', spanId: event.spanId });
      return;
    }

    this.applyEnd(span, event);
  }

  private applyEnd(
    span: AssembledSpan,
    event: Extract<TraceEvent, { event: 'span_ended' }>,
  ): void {
    span.endedAtMs = Math.max(span.startedAtMs, event.timestampMs);
    span.status = event.status;
    span.errorCategory = event.errorCategory;
    span.metadata = event.metadata ?? {};
  }

  finish(): { spans: AssembledSpan[]; diagnostics: AssemblyDiagnostic[] } {
    for (const spanId of this.pendingEnds.keys()) {
      this.diagnostics.push({ code: 'end_without_start', spanId });
    }

    for (const span of this.spans.values()) {
      if (span.status === 'open') {
        this.diagnostics.push({ code: 'span_left_open', spanId: span.spanId });
      }
    }

    return {
      spans: [...this.spans.values()],
      diagnostics: [...this.diagnostics],
    };
  }
}

This assembler buffers an end event until its start arrives. At finalization, unmatched ends and open spans remain visible as diagnostics. It does not invent timestamps or mark incomplete work successful.

For an unbounded live stream, pending events need a size limit and expiration policy. Otherwise malformed or hostile input can create an unbounded map.

Build a Forest, Not Just One Tree

A valid trace normally has one root, but a renderer should handle multiple roots and orphans without crashing.

type SpanNode = AssembledSpan & { children: SpanNode[] };

type TraceForest = {
  roots: SpanNode[];
  orphans: SpanNode[];
  duplicateIds: string[];
};

function buildForest(spans: AssembledSpan[]): TraceForest {
  const nodes = new Map<string, SpanNode>();
  const duplicateIds: string[] = [];

  for (const span of spans) {
    if (nodes.has(span.spanId)) {
      duplicateIds.push(span.spanId);
      continue;
    }

    nodes.set(span.spanId, { ...span, children: [] });
  }

  const roots: SpanNode[] = [];
  const orphans: SpanNode[] = [];

  for (const node of nodes.values()) {
    if (node.parentSpanId === null) {
      roots.push(node);
      continue;
    }

    const parent = nodes.get(node.parentSpanId);
    if (!parent) {
      orphans.push(node);
      continue;
    }

    parent.children.push(node);
  }

  const sortChildren = (node: SpanNode): void => {
    node.children.sort((a, b) => {
      return a.startedAtMs - b.startedAtMs || a.spanId.localeCompare(b.spanId);
    });
    node.children.forEach(sortChildren);
  };

  roots.sort((a, b) => a.startedAtMs - b.startedAtMs);
  roots.forEach(sortChildren);

  return { roots, orphans, duplicateIds };
}

Validate parent links for cycles before recursively sorting or rendering. A malformed trace where A is the parent of B and B is the parent of A can otherwise cause infinite recursion. Cycle detection can use a depth-first search with visiting and visited sets.

Orphans should appear in a separate “unattached spans” section with diagnostics. Hiding them makes instrumentation gaps look like missing work.

Tree Order Is Not Completion Order

Sort siblings by start time for a stable display, but do not imply that one caused the next. Parallel children can overlap completely.

A useful UI combines two views:

Tree: Parentage, retries, fallbacks, and handoffs.
Timeline: Overlap, waiting, time to first output, and critical latency.

tree                              timeline
research_agent                    |----------------------|
├─ search_web                     |-----|
├─ query_database                 |-------------|
├─ finance_api                    |--------|
│  └─ retry                           |----|
└─ summarize_results                            |------|

The tree explains why work happened. The timeline explains when it happened.

Do Not Sum Span Durations

Adding every span duration usually overstates trace time for two reasons:

Parent spans include the time of their children.
Parallel child spans overlap.

If a root lasts 2 seconds and contains two parallel 1-second tools, summing all three spans reports 4 seconds. The wall-clock run still lasted 2 seconds.

Use separate metrics:

Trace wall time: Root end minus root start.
Span duration: End minus start for one operation.
Self time: Span duration minus the union of child intervals, when that analysis is needed.
Critical path: The dependency path that determines the run’s completion time.

Computing a true critical path requires dependency semantics, not just parentage. Sibling tools may all be required, any one may be sufficient, or one may be cancelled after another succeeds. The trace schema needs to represent those decision rules before a UI can label a critical path confidently.

Model Retries and Fallbacks as Children

Retries are separate attempts with separate outcomes:

load_pricing                  ok
├─ attempt_1                  error: timeout
├─ attempt_2                  error: invalid_response
└─ fallback_to_cache          ok: age_minutes=18

The parent can legitimately succeed even when children fail. Do not automatically propagate the worst child status to the parent. The parent’s status should describe whether the operation fulfilled its contract; child statuses explain how.

Quality gates can still warn when a successful parent depended on stale fallback data or exceeded an attempt budget.

Preserve Partial Traces

Processes crash, clients disconnect, serverless invocations end, and exporters drop events. An open span is evidence, not clutter.

Render incomplete spans distinctly and include the reason when known:

generate_answer     open: completion event missing
stream_response     cancelled: client disconnected
tool_call           unknown: adapter ended before callback

Do not silently close every open span at the trace’s last timestamp. That invents duration and status. A UI may estimate a visible range, but it should label the estimate.

Compare Trees Through Invariants

Execution trees are useful regression artifacts, but exact snapshots are brittle. Compare durable properties:

Required and forbidden span kinds
Parent-child relationships
Attempt and fallback counts
Terminal status
Model and token budgets
Presence of open spans or orphans
Capability and adapter diagnostics

Ignore random IDs and normalize timestamps. For concurrent siblings, compare sets or parentage rather than one exact ordering.

Keep the Event Store Append-Friendly

Newline-delimited JSON works well for local traces because each event is independently appendable and replayable. A reader can stream events through the assembler rather than loading every run into memory.

Index or partition by trace ID when files become large. Bound retention and avoid storing raw prompts, tool payloads, retrieved documents, credentials, or user data by default. Tree reconstruction needs identity and lifecycle, not complete application content.

Validation Checklist

Before trusting an execution tree, verify:

All events belong to the expected trace.
Span IDs are unique.
Parent IDs resolve or appear as visible orphans.
Parent links contain no cycles.
Start and end events are not duplicated.
End times do not precede start times.
Open spans remain visible.
Numeric metadata is finite and bounded.
Missing adapter capabilities are reported.
Payload and privacy policy passed before storage.

Validation errors should be separate from agent errors. A malformed trace may describe a successful agent run while still being unusable for debugging.

Final Thought

Flat events are not the enemy; they are the practical storage format. The mistake is treating their arrival order as the execution model.

Assemble lifecycle events into spans, validate identity and parentage, preserve incomplete data honestly, and render both tree and timeline views. Then retries, fallbacks, parallel work, and silent failures become properties of a system you can inspect rather than clues scattered through a terminal transcript.

That is the real shift from flat logs to execution trees: not more telemetry, but trustworthy structure.

Why console.log Isn't Enough When Building AI Agents

Raju Dandigam — Fri, 14 Aug 2026 23:04:50 +0000

An AI agent fails, so you add a few log statements:

console.log('starting agent');
console.log('tool result', result);
console.log('final answer', answer);

That works until the agent calls tools in parallel, retries one of them, falls back to cached data, and makes several model calls for different purposes. The terminal still contains the events, but it no longer explains the run.

The limitation is not console.log() itself. The limitation is flat, uncorrelated events. Agent debugging needs identity, parent-child relationships, lifecycle, and safe metadata. Without those, more log lines often create more noise rather than more understanding.

The Questions Flat Logs Cannot Answer Naturally

Consider this output:

10:00:01 search started
10:00:01 search started
10:00:02 model started
10:00:02 search completed
10:00:03 search timed out
10:00:03 cache fallback used
10:00:04 model completed

Several important questions remain:

Which search completed and which timed out?
Were the searches siblings or was one a retry?
Which model call depended on which search result?
Did the model begin before retrieval finished?
Did the final answer use live or cached data?
Which user request produced these events?

Timestamps describe when events were written. They do not describe causality.

Agent Runs Have More Than One Shape

Some request paths are short and sequential, and flat logs are perfectly adequate. Agents become harder to observe because their control flow is often dynamic:

A model selects a tool at runtime.
Several retrieval strategies run concurrently.
A failed tool is retried with different arguments.
A fallback returns stale but valid data.
One agent hands work to another.
A stream begins before the complete result or usage is known.

The useful representation is usually a tree:

support_agent
├─ classify_question
│  └─ model_call
├─ retrieve_context
│  ├─ vector_search
│  └─ keyword_search
├─ check_account
│  ├─ billing_api       timeout
│  └─ cached_account    fallback
└─ generate_answer
   └─ model_call

The two model calls now have different roles. The fallback belongs to check_account, and the searches are parallel children of retrieval. The same events are easier to reason about because their relationships are explicit.

Silent Failures Make Structure Essential

The hardest agent failures do not always throw exceptions. A workflow can complete successfully while using the wrong path.

Imagine a quote agent that reports an item as available. Every top-level operation says success, but the inventory service timed out and the agent used a cache that was a day old. The response is syntactically valid and the HTTP status is 200. The behavior is still wrong for the current user.

A trace can make the path visible:

generate_quote                ok
├─ find_product               ok
├─ check_inventory            ok
│  ├─ live_inventory          error: timeout
│  └─ cached_inventory        ok: age_hours=24
└─ compose_quote              ok: inventory_source=cache

Flat logs can capture all of these facts, but only if every line carries enough context to reconstruct the relationships. At that point, you are already building a tracing model.

Add Correlation Before Adding Volume

The first improvement is to give every run and step stable identity.

type AgentEvent = {
  traceId: string;
  spanId: string;
  parentSpanId: string | null;
  event: 'started' | 'completed';
  name: string;
  kind: 'run' | 'model' | 'tool' | 'retrieval' | 'fallback';
  timestamp: string;
  status?: 'ok' | 'error' | 'cancelled';
  durationMs?: number;
  metadata?: Record<string, string | number | boolean | null>;
};

function writeEvent(event: AgentEvent): void {
  console.log(JSON.stringify(event));
}

console.log() is still the output mechanism. The difference is that the event has a contract. A local script, log processor, test, or trace viewer can group events by traceId and rebuild the tree from parentSpanId.

Structured events also make filtering reliable. Searching for a step name in prose logs is fragile; querying kind=tool and status=error is not.

Record Lifecycles, Not Messages About Lifecycles

Use one start and one completion event for each meaningful span. Completion should include status and duration.

const startedAt = Date.now();

writeEvent({
  traceId,
  spanId,
  parentSpanId,
  event: 'started',
  name: 'search_docs',
  kind: 'retrieval',
  timestamp: new Date(startedAt).toISOString(),
});

try {
  const documents = await searchDocuments(query);

  writeEvent({
    traceId,
    spanId,
    parentSpanId,
    event: 'completed',
    name: 'search_docs',
    kind: 'retrieval',
    timestamp: new Date().toISOString(),
    status: 'ok',
    durationMs: Date.now() - startedAt,
    metadata: { resultCount: documents.length },
  });
} catch (error) {
  writeEvent({
    traceId,
    spanId,
    parentSpanId,
    event: 'completed',
    name: 'search_docs',
    kind: 'retrieval',
    timestamp: new Date().toISOString(),
    status: 'error',
    durationMs: Date.now() - startedAt,
    metadata: {
      errorCategory: error instanceof Error ? error.name : 'UnknownError',
    },
  });

  throw error;
}

This is verbose when written manually, which is why real tracing libraries provide span helpers and async context propagation. The example exposes the information those helpers manage.

Capture the Smallest Useful Metadata

Useful metadata explains behavior without copying the payload:

Model name, input tokens, output tokens, and finish reason
Tool name, attempt number, and outcome category
Retrieval result count, score range, and context token count
Timeout, cancellation, or fallback status
Cache age and data-source category
Validation and authorization outcome

Avoid raw user messages, prompts, model output, tool arguments, tool results, headers, credentials, and retrieved documents by default. They increase risk and often make traces harder to scan.

For example, this metadata is enough to identify a broken context assembly step:

retrieve_docs       result_count=5
build_context       context_tokens=0
generate_answer     input_tokens=412 output_tokens=96

The trace narrows the problem without storing any document text.

Use One Vocabulary

Random log wording creates accidental complexity:

tool finished
tool done
completed tool
search returned

Choose controlled names and statuses instead. A small vocabulary such as started, ok, error, cancelled, and blocked is easier to aggregate and test. Use a separate error category for timeout, validation, authorization, rate limit, or dependency failure.

Consistency matters more than clever formatting.

When Structured Logs Are Enough

Structured logs may be all you need when the agent is small, runs in one process, has a few sequential operations, and does not need a visual timeline or distributed context.

They are a good starting point when:

You are still learning the workflow.
Runs are short and low-volume.
One developer inspects the output locally.
Parent-child relationships are simple.
Retention and team-wide analysis are not required.

The important step is to establish the event contract early. Moving structured events to a richer sink later is much easier than parsing years of inconsistent prose logs.

When to Move to Tracing

Use a tracing system when the agent has concurrent branches, retries, handoffs, streaming lifecycles, several services, or shared CI rules. Tracing adds capabilities that a terminal stream does not provide naturally:

Automatic context propagation
Parent-child span management
Timelines and execution trees
Trace-level sampling and retention
Cross-service correlation
Open-span and incomplete-run detection
Aggregation by model, tool, status, or error category

OpenTelemetry, framework-native integrations, local trace tools, and hosted observability products all use variations of this model. The destination can change; the core mental model remains the same.

A Practical Progression

Name the major agent boundaries.
Emit structured JSON rather than prose messages.
Add trace, span, and parent identifiers.
Record start/end lifecycle, status, and duration.
Use metadata-first capture and controlled vocabularies.
Add async context propagation when manual parent IDs become error-prone.
Move to a trace viewer or backend when volume and collaboration justify it.

This progression avoids a large platform investment before the execution model is understood.

Final Thought

console.log() remains useful. It is available everywhere and can be a perfectly good sink for structured local events. What it cannot provide by itself is the causal model of an agent run.

Do not respond to a confusing agent by printing more uncorrelated payloads. Add identity, lifecycle, parentage, and safe metadata. Once those relationships are visible, parallel tools, retries, fallbacks, and silent failures become much easier to explain.

The next article will focus on the representation itself: how execution trees differ from flat event streams and how to reconstruct them reliably from span data.

The Adapter Pattern: Unified Tracing Across AI SDK, LangChain, and OpenAI Agents

Raju Dandigam — Thu, 13 Aug 2026 00:05:01 +0000

An adapter layer becomes strategically useful when several teams need one observability contract but cannot, or should not, standardize on one agent framework.

AI SDK, LangChain.js, OpenAI Agents SDK, and direct model clients organize execution differently. One emphasizes generation and streaming, another exposes hierarchical callbacks, another has agent runs and handoffs, and a direct client exposes only provider requests unless the application adds its own spans.

Unified tracing should preserve those differences while translating the common lifecycle into one model. Done well, teams can share execution-tree tooling, CI quality gates, privacy policy, and telemetry export without coupling every consumer to every framework.

Unify Semantics, Not APIs

The frameworks do not need a shared callback interface. They need a shared answer to a smaller set of questions:

What is the root operation?
Which model, tool, retrieval, decision, and handoff spans occurred?
What was each span’s parent?
How did it end?
Which usage and timing metrics are available?
Which facts are unavailable from this integration?

The framework adapter converts its native lifecycle into those semantics. Consumers never call framework hooks directly.

AI SDK lifecycle ---------┐
LangChain callbacks ------+--> framework adapters --> trace core
OpenAI Agents tracing ----+
direct client wrappers ---┘
                                      |
                         +------------+------------+
                         |            |            |
                    execution UI   CI rules   telemetry sinks

A Practical Mapping Matrix

The exact public APIs change over time, so keep the mapping conceptual and verify it against the supported framework version.

Normalized concept	AI SDK-style integration	LangChain-style integration	OpenAI Agents-style integration	Direct client
Root run	Application request or generation	Chain, graph, or agent run	Agent trace or runner invocation	Manual application span
Model span	Generation or stream lifecycle	LLM/chat-model callback	Model generation item/span	Provider request wrapper
Tool span	Tool execution lifecycle	Tool callback/run	Function-tool execution	Application tool wrapper
Retrieval span	Tool or explicit application span	Retriever callback/run	Tool or custom span	Application wrapper
Handoff	Application-defined transition	Graph/chain transition	Native handoff lifecycle	Application span
Parent context	App context plus source IDs	Parent run identifiers	Trace/span context	`AsyncLocalStorage` or explicit context
Token usage	Final generation usage when exposed	Model callback metadata when exposed	Run/model usage when exposed	Provider response usage

This matrix is a design guide, not a promise that every version exposes every cell. The adapter’s capability declaration is the authoritative record.

Keep the Core Contract Framework-Neutral

The common model should represent lifecycle and capability without importing framework types.

type Capability =
  | 'model_lifecycle'
  | 'tool_lifecycle'
  | 'retrieval_lifecycle'
  | 'handoff_lifecycle'
  | 'parent_relationships'
  | 'streaming_completion'
  | 'token_usage'
  | 'cancellation';

type AdapterDescriptor = {
  id: string;
  adapterVersion: string;
  framework: string;
  supportedFrameworkRange: string;
  capabilities: Partial<Record<Capability, boolean>>;
};

type NormalizedAttribute = string | number | boolean;

type NormalizedSpan = {
  schemaVersion: 1;
  traceId: string;
  spanId: string;
  parentSpanId: string | null;
  name: string;
  kind: 'run' | 'model' | 'tool' | 'retrieval' | 'decision' | 'handoff';
  startedAt: string;
  endedAt?: string;
  status?: 'ok' | 'error' | 'cancelled';
  attributes: Record<string, NormalizedAttribute>;
  source: {
    adapterId: string;
    adapterVersion: string;
    sourceId: string;
  };
};

The source block is essential for support. When a span looks wrong, developers need to know which adapter and native event produced it.

Do Not Collapse Everything to the Lowest Common Denominator

A base schema should contain portable fields, but frameworks may expose valuable extra data. Use namespaced extension attributes rather than adding a new top-level field for every integration.

const span: NormalizedSpan = {
  // portable fields omitted
  attributes: {
    'model.name': 'example-model',
    'model.input_tokens': 840,
    'model.output_tokens': 126,
    'adapter.ai_sdk.finish_reason': 'stop',
  },
};

Portable consumers read model.*. Framework-specific diagnostics may read adapter.ai_sdk.*. Extension values must still follow the same privacy and size policy as core attributes.

Document extension keys and treat changes as adapter-version changes. Otherwise a shared schema slowly becomes an undocumented collection of framework internals.

Choose One Authoritative Capture Path

Duplicate instrumentation is one of the easiest ways to corrupt a trace. For example, a framework may already emit a model span while a provider-client wrapper emits another span for the same request.

agent run
├─ generate_answer        framework span
│  └─ provider_request    client wrapper span
└─ generate_answer        accidental duplicate

Nested spans may be intentional when they represent different layers. Duplicate peer spans are not.

Define precedence for each operation:

Prefer a stable framework lifecycle when it exposes correct parentage and completion.
Add a lower-level client span only when it represents a distinct network operation.
Disable overlapping automatic instrumentation when the adapter owns the span.
Attach a stable operation ID so duplicates can be detected in tests.

Do not deduplicate by span name and timestamp alone. Concurrent model calls may legitimately share both.

Register Adapters Explicitly

Automatic framework detection sounds convenient but can activate multiple adapters in a monorepo or after a transitive dependency is added. Explicit registration makes ownership visible.

type AdapterSession = {
  descriptor: AdapterDescriptor;
  stop(): Promise<void>;
};

type AdapterFactory<Options> = {
  descriptor: AdapterDescriptor;
  start(options: Options, core: TraceCore): Promise<AdapterSession>;
};

class AdapterRegistry {
  private readonly factories = new Map<string, AdapterFactory<unknown>>();

  register<Options>(factory: AdapterFactory<Options>): void {
    if (this.factories.has(factory.descriptor.id)) {
      throw new Error(`Adapter already registered: ${factory.descriptor.id}`);
    }

    this.factories.set(
      factory.descriptor.id,
      factory as AdapterFactory<unknown>,
    );
  }

  get(id: string): AdapterFactory<unknown> {
    const factory = this.factories.get(id);
    if (!factory) throw new Error(`Adapter not registered: ${id}`);
    return factory;
  }
}

Application configuration can then select one or more adapters deliberately. The registry should reject conflicting ownership of the same capture surface unless the configuration explains the intended nesting.

Preserve Context Across Framework Boundaries

An application may call a LangChain workflow from an AI SDK tool, or hand work to another service that uses a direct client. Adapter-local IDs are not enough.

Inside one Node.js process, the trace core can use AsyncLocalStorage to provide the active normalized context. An adapter reads that context when the framework does not supply a parent.

Across services or queues, propagate a standard trace carrier in request headers or message metadata. The receiving service validates the carrier and starts its framework run as a child or linked trace according to policy.

type TraceCarrier = {
  traceId: string;
  parentSpanId: string;
  sampled: boolean;
};

interface ContextBridge {
  inject(carrier: TraceCarrier): Record<string, string>;
  extract(headers: Record<string, string | undefined>): TraceCarrier | null;
}

Do not put prompts, user identifiers, or framework state in the carrier. It is correlation data, not a portable context dump.

Model Handoffs Explicitly

Handoffs deserve their own span kind because they change which agent owns the task. Treating a handoff as an ordinary tool call hides an important control-flow decision.

A normalized handoff span can record:

Source and destination agent roles from controlled vocabularies
Handoff reason category
Whether the destination accepted or rejected the task
Duration until the destination began work
Link to the destination root span

Do not record the complete transferred conversation by default. The trace needs the relationship, not a duplicate of the handoff payload.

Frameworks without a native handoff concept can emit the same span from an application wrapper. That keeps cross-framework analysis consistent.

Make Missing Data Visible

Unified tracing fails when consumers interpret “unavailable” as zero. If an adapter cannot observe token usage, cancellation, or tool internals, emit a diagnostic and declare the capability false.

Quality gates can then express policy:

Fail when a required capability is missing.
Skip a rule with a visible reason.
Warn when an optional metric is unavailable.

The choice belongs to the consumer’s policy, not the adapter. The adapter’s responsibility is honest data.

Maintain a Shared Conformance Suite

Every adapter should run the same behavioral fixtures:

Fixture	Required invariant
Single model call	One root, one model child, one completion each
Parallel tools	Siblings share the expected parent
Tool retry	Attempts remain distinct and ordered
Model stream	Completion or cancellation closes the span once
Handoff	Source and destination traces are linked correctly
Framework error	Controlled status and category are preserved
Duplicate callback	Diagnostic emitted; no duplicate completion
Abrupt shutdown	Open spans and dropped events are reported

Native callback fixtures test translation quickly. A small integration matrix should also run against the minimum and maximum supported framework versions.

Publish adapter compatibility separately from the core release. Framework updates should not require an unrelated trace-core version bump.

Treat Adapters as Maintained Products

An adapter has an ongoing compatibility cost. Give each integration:

A clear owner
A supported framework-version range
A changelog for semantic mappings
Capability and privacy documentation
Fixture and integration tests
A deprecation policy
Diagnostics that identify the adapter version

Without ownership, adapters tend to keep compiling while silently losing events after framework lifecycle changes.

Migrate One Consumer at a Time

Adopting unified tracing does not require replacing existing observability immediately.

Define the normalized schema and privacy policy.
Add one adapter in shadow mode.
Compare trace counts, parentage, status, and usage against the existing integration.
Move local rendering or one CI rule to normalized events.
Add the next framework adapter and run the conformance suite.
Switch shared consumers after the data is trusted.
Remove duplicate legacy instrumentation.

Shadow mode should avoid exporting sensitive data twice. Compare normalized metadata, not raw payloads.

When Unified Tracing Is the Wrong Choice

One small application with one stable framework may be better served by its built-in telemetry. An adapter platform is justified when several teams need shared rules, common privacy controls, migration flexibility, or one observability backend.

The pattern should remove repeated integration work. If it adds more maintenance than the framework differences it isolates, the boundary is premature.

Final Thought

Unified tracing across AI SDK, LangChain, OpenAI Agents, and direct clients is not achieved by forcing every framework event into an identical callback shape. It comes from a stable semantic contract, honest capability reporting, explicit context propagation, and adapters that own framework-specific volatility.

Choose one authoritative capture path, preserve handoffs and parentage, keep extensions namespaced, and operate each adapter with a conformance suite and compatibility policy. Teams remain free to choose the framework that fits their application while the organization gains one trustworthy language for execution, quality gates, and observability.

Trace Any TypeScript Agent Framework With Adapters

Raju Dandigam — Sun, 09 Aug 2026 15:45:28 +0000

TypeScript teams rarely standardize on one AI framework forever. One service may use Vercel AI SDK, another LangChain.js, another OpenAI Agents SDK, and a mature system may call provider clients directly.

Those implementations expose different callback, telemetry, and streaming surfaces. Observability becomes expensive when every dashboard, test rule, and CI report understands each framework independently.

An adapter layer isolates that variation. Framework-specific code captures source events; the adapter translates them into one versioned trace model; the rest of the system operates on normalized events.

framework callbacks or wrappers
            |
            v
     framework adapter
            |
            v
 versioned trace events
     |       |       |
     v       v       v
 local UI  CI gates  telemetry export

The goal is not to pretend every framework is identical. The goal is to preserve a common set of observable facts without leaking framework details into every consumer.

Put the Boundary in the Right Place

A tempting interface is runWithTrace(input) -> { result, events }. It works in a demo but creates several problems:

Streaming runs may not have one finite completion point.
Buffering every event in memory does not scale.
Callback-driven frameworks already own the run lifecycle.
A framework may emit activity after the initial method returns.
Returning events couples capture, storage, and application results.

A stronger boundary translates source events as they arrive and sends normalized events to the tracing core.

Define the Normalized Model First

Keep the shared model small, explicit, and versioned.

type SpanKind = 'run' | 'model' | 'tool' | 'retrieval' | 'decision';

type TraceEvent =
  | {
      schemaVersion: 1;
      event: 'span_started';
      traceId: string;
      spanId: string;
      parentSpanId: string | null;
      name: string;
      kind: SpanKind;
      timestamp: string;
      attributes: Record<string, string | number | boolean>;
    }
  | {
      schemaVersion: 1;
      event: 'span_ended';
      traceId: string;
      spanId: string;
      timestamp: string;
      status: 'ok' | 'error' | 'cancelled';
      durationMs?: number;
      errorCategory?: string;
      attributes: Record<string, string | number | boolean>;
    }
  | {
      schemaVersion: 1;
      event: 'adapter_diagnostic';
      traceId: string;
      adapter: string;
      code: string;
      message: string;
    };

The core model intentionally avoids raw prompts, outputs, tool arguments, and tool results. A separate capture policy can approve selected payload fields, but adapters should not add them implicitly.

Use controlled attribute keys for model usage, tool outcomes, retry counts, and framework identity. A common schema is useful only when adapters agree on the meaning and units of those fields.

Declare Adapter Capabilities

Not every framework exposes the same information. Hiding those gaps produces misleading traces.

type AdapterCapabilities = {
  modelLifecycle: boolean;
  toolLifecycle: boolean;
  parentRelationships: boolean;
  streamingLifecycle: boolean;
  tokenUsage: boolean;
  cancellation: boolean;
};

interface TraceEmitter {
  emit(event: TraceEvent): void;
}

interface FrameworkAdapter<SourceEvent> {
  readonly id: string;
  readonly version: number;
  readonly capabilities: AdapterCapabilities;

  accept(event: SourceEvent, emitter: TraceEmitter): void;
  close(emitter: TraceEmitter): void;
}

Capabilities let consumers distinguish “zero tool calls occurred” from “this adapter cannot observe tool calls.” Quality gates can then fail, skip, or warn according to policy.

Normalize a Generic Source Event

The following source union represents the kinds of callbacks many frameworks expose. Actual framework adapters translate their native callbacks into this internal source shape first.

type SourceEvent =
  | {
      type: 'start';
      sourceId: string;
      parentSourceId?: string;
      name: string;
      kind: SpanKind;
      timestampMs: number;
    }
  | {
      type: 'end';
      sourceId: string;
      timestampMs: number;
      status: 'ok' | 'error' | 'cancelled';
      errorCategory?: string;
      attributes?: Record<string, string | number | boolean>;
    };

Framework-specific code is now responsible for only one translation: native callback to SourceEvent. IDs, lifecycle state, diagnostics, and the final trace schema can be shared.

Preserve Identity and Parentage

Source IDs cannot be assumed to match the trace system’s ID format. Maintain a stable mapping for the lifetime of one adapter session.

import { randomUUID } from 'node:crypto';

class NormalizingAdapter implements FrameworkAdapter<SourceEvent> {
  readonly id = 'generic';
  readonly version = 1;
  readonly capabilities: AdapterCapabilities = {
    modelLifecycle: true,
    toolLifecycle: true,
    parentRelationships: true,
    streamingLifecycle: false,
    tokenUsage: true,
    cancellation: true,
  };

  private readonly traceId = randomUUID();
  private readonly spanIds = new Map<string, string>();
  private readonly startedAt = new Map<string, number>();
  private readonly ended = new Set<string>();

  private spanId(sourceId: string): string {
    const existing = this.spanIds.get(sourceId);
    if (existing) return existing;

    const created = randomUUID();
    this.spanIds.set(sourceId, created);
    return created;
  }

  accept(event: SourceEvent, emitter: TraceEmitter): void {
    if (event.type === 'start') {
      this.handleStart(event, emitter);
      return;
    }

    this.handleEnd(event, emitter);
  }

  private handleStart(
    event: Extract<SourceEvent, { type: 'start' }>,
    emitter: TraceEmitter,
  ): void {
    if (this.startedAt.has(event.sourceId)) {
      this.diagnostic(emitter, 'duplicate_start', event.sourceId);
      return;
    }

    this.startedAt.set(event.sourceId, event.timestampMs);

    emitter.emit({
      schemaVersion: 1,
      event: 'span_started',
      traceId: this.traceId,
      spanId: this.spanId(event.sourceId),
      parentSpanId: event.parentSourceId
        ? this.spanId(event.parentSourceId)
        : null,
      name: event.name,
      kind: event.kind,
      timestamp: new Date(event.timestampMs).toISOString(),
      attributes: { adapter: this.id },
    });
  }

  private handleEnd(
    event: Extract<SourceEvent, { type: 'end' }>,
    emitter: TraceEmitter,
  ): void {
    if (this.ended.has(event.sourceId)) {
      this.diagnostic(emitter, 'duplicate_end', event.sourceId);
      return;
    }

    const start = this.startedAt.get(event.sourceId);
    if (start === undefined) {
      this.diagnostic(emitter, 'end_without_start', event.sourceId);
      return;
    }

    this.ended.add(event.sourceId);

    emitter.emit({
      schemaVersion: 1,
      event: 'span_ended',
      traceId: this.traceId,
      spanId: this.spanId(event.sourceId),
      timestamp: new Date(event.timestampMs).toISOString(),
      status: event.status,
      durationMs: Math.max(0, event.timestampMs - start),
      errorCategory: event.errorCategory,
      attributes: {
        adapter: this.id,
        ...(event.attributes ?? {}),
      },
    });
  }

  private diagnostic(
    emitter: TraceEmitter,
    code: string,
    sourceId: string,
  ): void {
    emitter.emit({
      schemaVersion: 1,
      event: 'adapter_diagnostic',
      traceId: this.traceId,
      adapter: this.id,
      code,
      message: `${code} for source event ${sourceId}`,
    });
  }

  close(emitter: TraceEmitter): void {
    for (const sourceId of this.startedAt.keys()) {
      if (!this.ended.has(sourceId)) {
        this.diagnostic(emitter, 'span_left_open', sourceId);
      }
    }
  }
}

This example chooses to report an end event with no start rather than inventing a start time. Another system may synthesize an incomplete span, but the policy should be explicit and consistent.

Adapt Framework Hooks, Do Not Leak Them

Each integration should be a thin layer:

function attachFrameworkHooks(
  framework: FrameworkRuntime,
  adapter: FrameworkAdapter<SourceEvent>,
  emitter: TraceEmitter,
): () => void {
  const unsubscribe = framework.subscribe((nativeEvent) => {
    const sourceEvents = translateNativeEvent(nativeEvent);
    for (const event of sourceEvents) adapter.accept(event, emitter);
  });

  return () => {
    unsubscribe();
    adapter.close(emitter);
  };
}

The exact subscription and callback APIs differ across Vercel AI SDK, LangChain.js, OpenAI Agents SDK, and direct provider clients, and they may change between versions. That volatility belongs in translateNativeEvent(), not in the trace store or quality-gate code.

An adapter may emit zero, one, or several normalized events for one native callback. For example, a combined framework event might close a tool span and open the next model span.

Streaming Needs More Than Start and End

Frameworks often expose stream start, first chunk, tool activity, completion, usage, and cancellation through different callbacks or promises. The adapter should preserve the lifecycle facts the normalized model supports:

Keep the model span open until completion, error, or cancellation.
Record time to first chunk as a bounded numeric attribute.
Attach token usage only when the framework reports final usage.
Treat client cancellation separately from provider error.
Ignore or count individual content chunks unless payload capture is explicitly enabled.

If the source cannot expose cancellation or final usage, set the relevant capability to false. Do not fill missing values with zero.

Normalize Semantics, Not Just Field Names

Two frameworks may use the word “tool” differently. One event may represent the model requesting a tool; another may represent the application executing it. Those are distinct operations.

A robust adapter design defines semantics for:

Model request versus streamed response
Tool selection versus tool execution
Framework retry versus application retry
Retrieval operation versus ordinary tool call
Cancelled, timed out, rejected, and failed states
Input, cached-input, and output token units

Document these mappings next to the adapter and include the adapter version in trace attributes. Field renaming without semantic alignment creates a common schema that cannot be compared safely.

Test Adapters With Event Fixtures

Adapter tests should not call real models. Feed representative native-event fixtures into the translation layer and assert normalized invariants.

test('normalizes parallel tool spans under one parent', () => {
  const emitter = new RecordingEmitter();
  const adapter = new NormalizingAdapter();

  for (const event of parallelToolFixture) {
    adapter.accept(event, emitter);
  }
  adapter.close(emitter);

  expect(emitter.diagnostics()).toEqual([]);
  expect(emitter.openSpanIds()).toEqual([]);
  expect(emitter.childrenOf('parallel_retrieval')).toHaveLength(3);
});

Maintain fixtures for success, error, retry, streaming completion, cancellation, duplicate callbacks, missing parents, missing usage, and abrupt shutdown. Run the same conformance suite against every adapter.

Use a small integration smoke test for each supported framework version to detect callback-surface changes. Keep that separate from the fast fixture suite.

Keep Analysis Framework-Independent

Once events are normalized, one set of consumers can:

Render execution trees
Enforce CI quality gates
Calculate model and tool usage
Detect retries and fallbacks
Export spans to OpenTelemetry
Produce local trace artifacts

Consumers may filter by adapter and adapter version when investigating integration-specific gaps, but their primary logic should depend on the normalized schema.

When an Adapter Layer Is Worth It

Adapters add code and a compatibility commitment. They are justified when a team has multiple frameworks, is migrating between frameworks, maintains shared quality gates, or wants a stable trace history across implementation changes.

For one small project with adequate built-in tracing, direct instrumentation may be simpler. Introduce the abstraction when framework variation creates repeated work or inconsistent data, not merely because an adapter pattern is available.

Final Thought

A useful tracing adapter does more than rename callback fields. It preserves identity, parentage, lifecycle, status, usage semantics, and capability gaps while keeping framework volatility at the edge of the system.

Design the normalized event contract first, make adapter limitations visible, validate every lifecycle transition, and test translations with fixtures. Then teams can choose the agent framework that fits their application while sharing one execution model for debugging, CI, and observability.

The final article in this sequence will compare the concrete responsibilities of adapters for AI SDK, LangChain, and OpenAI Agents-style integrations and show how to keep the shared core stable as those ecosystems evolve.

Fast Agent Quality Gates: Deterministic Rules Over LLM Judges

Raju Dandigam — Tue, 04 Aug 2026 14:35:34 +0000

Deterministic agent tests become much more valuable when they are expressed as reusable quality gates rather than one-off assertions scattered across test files.

A gate answers a narrow engineering question: Did the run validate before writing data? Did retries stay within policy? Was token usage recorded and within budget? Did every started span finish? The result should be stable, fast, and specific enough that a developer knows what to fix.

LLM judges still have a role in semantic evaluation. They should not be the only thing standing between a structural agent regression and production.

What Makes a Good Gate?

A practical quality gate has five properties:

Deterministic: The same normalized trace produces the same result.
Narrow: It evaluates one explicit contract.
Actionable: Failure identifies the rule, evidence, and expected state.
Versioned: Rule and trace-schema changes can be reviewed.
Portable: It can run locally, in CI, and against replayed fixtures.

“Answer quality was 6/10” is a signal, but it is not a narrow engineering contract. “The write tool ran before authorization completed” is.

Start With a Stable Trace Contract

The gate engine should consume normalized metadata rather than framework-specific callback objects.

type StepKind = 'run' | 'model' | 'tool' | 'retrieval' | 'policy' | 'fallback';

type TraceStep = {
  id: string;
  parentId: string | null;
  sequence: number;
  name: string;
  kind: StepKind;
  status: 'ok' | 'error' | 'blocked' | 'cancelled';
  attempt?: number;
  inputTokens?: number;
  outputTokens?: number;
  durationMs?: number;
  metadata?: Record<string, string | number | boolean | null>;
};

type AgentTrace = {
  schemaVersion: 1;
  fixture: string;
  status: TraceStep['status'];
  steps: TraceStep[];
};

Normalize volatile fields before evaluation. Random IDs can remain for parent-child checks, but timestamps, temporary paths, request IDs, and raw payloads should not influence deterministic results.

Define a Rule Interface

Rules should return structured evidence instead of throwing assertion errors directly.

type Severity = 'error' | 'warning';

type RuleResult = {
  ruleId: string;
  severity: Severity;
  passed: boolean;
  message: string;
  evidence?: Record<string, string | number | boolean>;
};

type TraceRule = {
  id: string;
  version: number;
  severity: Severity;
  evaluate(trace: AgentTrace): RuleResult;
};

function result(
  rule: TraceRule,
  passed: boolean,
  message: string,
  evidence?: RuleResult['evidence'],
): RuleResult {
  return {
    ruleId: `${rule.id}@${rule.version}`,
    severity: rule.severity,
    passed,
    message,
    evidence,
  };
}

Versioning a rule makes baseline changes explicit. If the meaning of max_model_calls changes, reviewers can see that the policy changed rather than assuming the agent regressed.

Gate 1: Required Steps

function requireSteps(required: string[]): TraceRule {
  return {
    id: 'required_steps',
    version: 1,
    severity: 'error',
    evaluate(trace) {
      const actual = new Set(trace.steps.map((step) => step.name));
      const missing = required.filter((name) => !actual.has(name));

      return result(
        this,
        missing.length === 0,
        missing.length === 0
          ? 'All required steps ran'
          : `Missing required steps: ${missing.join(', ')}`,
        { missingCount: missing.length },
      );
    },
  };
}

Required-step rules work well for validation, retrieval, policy checks, and mandatory cleanup. Do not require every implementation detail; gates should protect behavior that matters to users, cost, safety, or correctness.

Gate 2: Causal Order

For sequential dependencies, compare the recorder’s monotonic sequence. For concurrent work, assert parentage instead of completion order.

function requireOrder(before: string, after: string): TraceRule {
  return {
    id: `order:${before}:${after}`,
    version: 1,
    severity: 'error',
    evaluate(trace) {
      const left = trace.steps.find((step) => step.name === before);
      const right = trace.steps.find((step) => step.name === after);

      if (!left || !right) {
        return result(this, false, 'Cannot evaluate order: step missing');
      }

      return result(
        this,
        left.sequence < right.sequence,
        left.sequence < right.sequence
          ? `${before} occurred before ${after}`
          : `${after} occurred before required dependency ${before}`,
        { beforeSequence: left.sequence, afterSequence: right.sequence },
      );
    },
  };
}

Authorization-before-write and retrieval-before-generation are good causal gates. Ordering every trace step creates brittle tests and blocks harmless parallelization.

Gate 3: Forbidden Work After a Block

const noExternalWorkAfterBlock: TraceRule = {
  id: 'no_external_work_after_block',
  version: 1,
  severity: 'error',
  evaluate(trace) {
    const block = trace.steps.find((step) => step.status === 'blocked');
    if (!block) return result(this, true, 'Run was not blocked');

    const forbidden = trace.steps.filter((step) => {
      return (
        step.sequence > block.sequence &&
        (step.kind === 'model' || step.kind === 'tool')
      );
    });

    return result(
      this,
      forbidden.length === 0,
      forbidden.length === 0
        ? 'No model or tool work occurred after the block'
        : `External work continued after block: ${forbidden
            .map((step) => step.name)
            .join(', ')}`,
      { forbiddenCount: forbidden.length },
    );
  },
};

This is stronger than checking only the final status. A run can report blocked and still leak a model or tool call if orchestration continues incorrectly.

Gate 4: Attempts and Loops

Retries should include an explicit attempt field. Count attempts by operation and parent span instead of looking for consecutive names, because parallel events can interleave.

function maxAttempts(stepName: string, limit: number): TraceRule {
  return {
    id: `max_attempts:${stepName}`,
    version: 1,
    severity: 'error',
    evaluate(trace) {
      const attempts = trace.steps
        .filter((step) => step.name === stepName)
        .map((step) => step.attempt ?? 1);

      const maximum = attempts.length === 0 ? 0 : Math.max(...attempts);

      return result(
        this,
        maximum <= limit,
        maximum <= limit
          ? `${stepName} stayed within ${limit} attempts`
          : `${stepName} reached attempt ${maximum}; limit is ${limit}`,
        { maximumAttempt: maximum, limit },
      );
    },
  };
}

For broader loop detection, define a stable state signature such as planner_state + selected_tool + outcome. Fail when the same signature repeats beyond policy. Step-name pattern matching alone often produces false positives in legitimate iterative workflows.

Gate 5: Token Budgets

Missing usage data should not silently become zero. A cost gate cannot pass when it has no measurements.

function maxTotalTokens(limit: number): TraceRule {
  return {
    id: 'max_total_tokens',
    version: 1,
    severity: 'error',
    evaluate(trace) {
      const modelSteps = trace.steps.filter((step) => step.kind === 'model');
      const missingUsage = modelSteps.filter((step) => {
        return step.inputTokens === undefined || step.outputTokens === undefined;
      });

      if (missingUsage.length > 0) {
        return result(this, false, 'Model usage is missing', {
          missingUsageCount: missingUsage.length,
        });
      }

      const total = modelSteps.reduce((sum, step) => {
        return sum + (step.inputTokens ?? 0) + (step.outputTokens ?? 0);
      }, 0);

      return result(
        this,
        total <= limit,
        total <= limit
          ? `Token usage ${total} is within budget ${limit}`
          : `Token usage ${total} exceeds budget ${limit}`,
        { totalTokens: total, limit },
      );
    },
  };
}

Depending on the provider, cached input may need its own field and cost policy. Keep raw usage dimensions in the normalized trace so the gate does not rely on a lossy totalTokens value.

Gate 6: Trace Integrity

Before evaluating agent behavior, verify that the telemetry itself is trustworthy:

Step IDs are unique.
Every non-root parent exists in the same trace.
Sequence numbers are unique and monotonic.
Exactly one root step exists.
Terminal status is present.
Numeric metrics are finite and non-negative.
No forbidden payload keys appear.

A malformed trace should fail with an instrumentation error, not produce misleading policy results.

Run Rules and Produce One Report

type GateReport = {
  fixture: string;
  passed: boolean;
  failures: RuleResult[];
  warnings: RuleResult[];
  results: RuleResult[];
};

function evaluateTrace(
  trace: AgentTrace,
  rules: TraceRule[],
): GateReport {
  const results = rules.map((rule) => rule.evaluate(trace));
  const failures = results.filter((item) => {
    return !item.passed && item.severity === 'error';
  });
  const warnings = results.filter((item) => {
    return !item.passed && item.severity === 'warning';
  });

  return {
    fixture: trace.fixture,
    passed: failures.length === 0,
    failures,
    warnings,
    results,
  };
}

Write the report as JSON for automation and as a short Markdown summary for pull-request logs or CI annotations. Include rule versions, evidence, fixture names, and a link or path to the normalized trace artifact.

Compare Baselines Without Snapshot Brittleness

Exact trace snapshots are difficult to maintain. Prefer a summary of durable metrics:

type TraceBaseline = {
  fixture: string;
  requiredTools: string[];
  maximumModelCalls: number;
  maximumTokens: number;
  maximumAttemptsByTool: Record<string, number>;
};

Use both absolute and relative limits. A 50% token increase from 100 to 150 may be harmless; a 50% increase from 20,000 to 30,000 may be costly. Conversely, an absolute increase of 500 tokens is significant for a small workflow and noise for a very large one.

Require intentional baseline updates in the same pull request as the behavior change. The review should explain why the new budget or tool path is acceptable.

Separate Synthetic and Live Thresholds

Scripted orchestration tests should use fake clocks and exact budgets. Live-model and network tests have natural variance and need broader statistical thresholds.

Do not use one threshold for both. A local fixture that suddenly takes ten seconds likely indicates a bug. A real provider call crossing a narrow latency threshold once may only reflect transient infrastructure conditions.

For live runs, compare rolling distributions such as median and tail latency over enough samples. Keep those trend checks outside the fastest pull-request gate unless the project has the capacity to operate them reliably.

A CI Contract

A provider-neutral CI job can follow this sequence:

1. Run scripted agent fixtures
2. Validate every normalized trace
3. Evaluate the configured rule set
4. Write JSON and Markdown reports
5. Exit non-zero when error-severity rules fail
6. Upload reduced trace artifacts for failed fixtures
7. Retain artifacts for a short, explicit period

Use the project’s existing runtime-version file and package-manager lockfile rather than hard-coding setup details into the article or gate engine. Real-model credentials should be unavailable to the deterministic job.

Run semantic evaluations in a separate job with explicit authorization, cost controls, and a slower cadence.

Avoid Gate Inflation

Too many brittle gates make developers ignore the system. Add a rule only when it protects a meaningful contract and has an owner.

Use error for correctness, safety, or hard budget violations. Use warning for trends that need review but should not block immediately. Track warning age; a warning that never becomes actionable should be removed or converted into a real policy.

When a gate fails repeatedly for accepted behavior, fix the rule or the baseline. Do not normalize permanent red CI.

Final Thought

Agent quality gates work best when they treat execution as an engineering artifact. A normalized trace, a versioned rule set, and an evidence-rich report can catch missing validation, unauthorized work, retry storms, cost regressions, and broken instrumentation in seconds.

Use deterministic gates for contracts the trace can prove. Keep semantic judges for language and reasoning quality, where probabilistic evaluation is actually necessary. That separation makes CI faster and makes every failure easier to trust.

The next article will move from rules to integrations: how adapters translate different TypeScript agent frameworks into one trace model without coupling the core to any single SDK.

How to Test AI Agents Without Calling More LLMs

Raju Dandigam — Thu, 30 Jul 2026 16:35:04 +0000

AI-agent testing often starts with an expensive loop: call the agent, send its answer to another model, ask for a quality score, and hope the score is stable enough for CI.

LLM judges can be useful for genuinely semantic questions. They are a poor default for verifying tool order, retry limits, validation, budgets, state transitions, or error handling. Those behaviors have explicit contracts and can usually be tested with ordinary deterministic techniques.

The most effective agent test suite separates orchestration correctness from language quality. Test the first on every commit with fakes, traces, and rules. Evaluate the second with a smaller, calibrated model-and-human workflow.

What Can Be Deterministic?

An agent may produce variable language while still following a predictable execution contract. Useful deterministic checks include:

Input validation runs before any model or tool call.
Only authorized tools are available for the current state.
Retrieval completes before generation uses its results.
Tool arguments match a schema.
Retries stop after the configured limit.
A timeout or cancellation ends the run.
A fallback is used only for approved error categories.
Token and tool-call budgets are enforced.
Every started span is completed exactly once.
Sensitive payloads are absent from the trace.

None of these questions requires a second model. Most do not require a first model either.

Use a Testing Pyramid for Agents

Layer	Model access	Purpose
Pure unit tests	None	Routing, validation, budgeting, state reducers
Orchestration tests	Scripted fake	Tool order, retries, fallbacks, termination
Tool contract tests	None or sandbox	Schemas, timeouts, error mapping
Trace replay tests	None	Regression checks against recorded execution metadata
Model integration tests	Real model	Provider compatibility and a small set of end-to-end paths
Semantic evaluations	Real model and/or human	Helpfulness, groundedness, tone, nuanced correctness

The lower layers should contain most of the suite. They are fast, reproducible, and actionable. The upper layers are valuable, but they should be deliberately smaller.

Put Models and Tools Behind Interfaces

Dependency injection makes the orchestration testable without network calls.

type ModelReply =
  | { type: 'tool_call'; tool: string; args: unknown }
  | { type: 'final'; text: string; usage: { input: number; output: number } };

interface ModelClient {
  generate(input: {
    messages: Array<{ role: string; content: string }>;
    tools: string[];
  }): Promise<ModelReply>;
}

interface ToolClient {
  execute(name: string, args: unknown): Promise<unknown>;
}

interface Clock {
  now(): number;
}

Production code receives real implementations. Tests provide scripted implementations with known behavior.

Script Model Behavior Instead of Generating It

A scripted model returns a predefined sequence and fails if the agent makes an unexpected extra call.

class ScriptedModel implements ModelClient {
  private index = 0;

  constructor(private readonly replies: ModelReply[]) {}

  async generate(): Promise<ModelReply> {
    const reply = this.replies[this.index];
    if (!reply) {
      throw new Error(`Unexpected model call at index ${this.index}`);
    }

    this.index += 1;
    return structuredClone(reply);
  }

  assertConsumed(): void {
    if (this.index !== this.replies.length) {
      throw new Error(
        `Expected ${this.replies.length} model calls, received ${this.index}`,
      );
    }
  }
}

This fake does not imitate model intelligence. It controls the branch that the orchestration must handle. One script can request a tool and then return a final answer; another can repeatedly request an invalid tool call to verify termination.

Record a Test-Friendly Trace

The trace should expose behavior without coupling tests to prose output.

type TraceStep = {
  sequence: number;
  name: string;
  kind: 'run' | 'model' | 'tool' | 'policy' | 'fallback';
  status: 'ok' | 'error' | 'blocked';
  parentId: string | null;
  attempt?: number;
  metadata?: Record<string, string | number | boolean | null>;
};

type AgentTrace = {
  status: 'ok' | 'error' | 'blocked';
  steps: TraceStep[];
};

Use a monotonic sequence assigned by the recorder for causal assertions. Do not infer retry behavior from consecutive names: parallel work can interleave events, and unrelated steps can separate attempts.

Keep the trace schema stable and metadata-first. Tests should assert execution contracts, not depend on raw prompts or complete tool results.

Build Reusable Trace Assertions

Small domain-specific helpers make failures easier to understand than generic array comparisons.

function stepIndex(trace: AgentTrace, name: string): number {
  return trace.steps.findIndex((step) => step.name === name);
}

function expectStepBefore(
  trace: AgentTrace,
  first: string,
  second: string,
): void {
  const firstIndex = stepIndex(trace, first);
  const secondIndex = stepIndex(trace, second);

  if (firstIndex < 0) throw new Error(`Missing trace step: ${first}`);
  if (secondIndex < 0) throw new Error(`Missing trace step: ${second}`);
  if (firstIndex >= secondIndex) {
    throw new Error(`Expected ${first} before ${second}`);
  }
}

function countSteps(trace: AgentTrace, name: string): number {
  return trace.steps.filter((step) => step.name === name).length;
}

When operations run in parallel, assert parentage and required dependencies rather than total ordering. Two sibling tools may complete in either order and both be correct.

Test Retrieval Before Generation

test('retrieves policy before generating an answer', async () => {
  const model = new ScriptedModel([
    {
      type: 'final',
      text: 'Use the documented return process.',
      usage: { input: 420, output: 18 },
    },
  ]);

  const { trace } = await runSupportAgent(
    { model, tools: fakeTools, clock: fakeClock },
    'How do returns work?',
  );

  expectStepBefore(trace, 'retrieve_policy', 'generate_answer');
  expect(countSteps(trace, 'retrieve_policy')).toBe(1);
  model.assertConsumed();
});

The test says exactly what failed: a missing step, an invalid dependency order, or an unexpected model call. No quality score is needed.

Test Retry Limits by Attempt

test('stops a tool after two failed attempts', async () => {
  const tools = new FakeTools({
    lookup_invoice: [
      new TimeoutError(),
      new TimeoutError(),
      { invoiceId: 'should-never-be-returned' },
    ],
  });

  const { trace } = await runInvoiceAgent(
    { model: scriptedToolCaller, tools, clock: fakeClock },
    'Find the latest invoice',
  );

  const attempts = trace.steps.filter(
    (step) => step.name === 'lookup_invoice',
  );

  expect(attempts.map((step) => step.attempt)).toEqual([1, 2]);
  expect(tools.calls('lookup_invoice')).toHaveLength(2);
  expect(trace.status).toBe('error');
});

This catches an unbounded retry loop without waiting for a real service or model.

Test Guardrails as Control Flow

A blocked request should produce no downstream model or tool activity.

test('blocks unauthorized export before external calls', async () => {
  const { trace } = await runExportAgent(
    { model: modelThatMustNotRun, tools: toolsThatMustNotRun, clock: fakeClock },
    { action: 'export_all_accounts', authorized: false },
  );

  expect(trace.status).toBe('blocked');
  expect(stepIndex(trace, 'authorize_export')).toBeGreaterThanOrEqual(0);
  expect(trace.steps.some((step) => step.kind === 'model')).toBe(false);
  expect(trace.steps.some((step) => step.kind === 'tool')).toBe(false);
});

The test uses structured input instead of asking a model to recognize a particular malicious phrase. A separate security suite can test prompt-injection resistance with representative text fixtures.

Make Time Deterministic

Wall-clock assertions are often flaky under CI load. Inject a fake clock or scheduler and advance it deliberately.

class FakeClock implements Clock {
  private value = 0;

  now(): number {
    return this.value;
  }

  advance(ms: number): void {
    this.value += ms;
  }
}

test('uses fallback after the configured timeout', async () => {
  const clock = new FakeClock();
  const tools = new FakeTools({ primary_search: [new TimeoutError()] });

  const { trace } = await runSearchAgent(
    { model: scriptedToolCaller, tools, clock },
    'pricing',
  );

  expect(stepIndex(trace, 'fallback_search')).toBeGreaterThanOrEqual(0);
});

Keep a small number of real timing tests for integration boundaries. Do not make every unit test depend on an overloaded runner finishing within an arbitrary number of milliseconds.

Test Budgets at the Enforcement Point

Model fakes can return explicit usage values. That lets you verify budget logic without paying for tokens.

test('stops before exceeding the session token budget', async () => {
  const model = new ScriptedModel([
    { type: 'final', text: 'first', usage: { input: 700, output: 200 } },
    { type: 'final', text: 'second', usage: { input: 700, output: 200 } },
  ]);

  const { trace } = await runBudgetedAgent(
    { model, tools: fakeTools, clock: fakeClock, maxTokens: 1_200 },
    'continue',
  );

  expect(trace.steps.some((step) => step.name === 'token_budget_block')).toBe(true);
  expect(countSteps(trace, 'model_call')).toBe(1);
});

This proves the enforcement behavior. A separate provider integration test can verify that production usage fields are mapped correctly into the internal token model.

Replay Traces for Regression Tests

Recorded metadata traces can drive tests without replaying sensitive content or calling a model. Store a compact fixture describing model decisions, tool outcomes, usage, and expected invariants.

Avoid treating the entire trace as a snapshot that must match byte for byte. IDs, timestamps, and harmless implementation details make snapshots brittle. Normalize the trace and assert durable properties:

Required and forbidden steps
Parent-child relationships
Attempt limits
Terminal status
Budget totals
Policy outcomes

Update a fixture only when the behavioral contract intentionally changes.

Test Failure Paths, Not Only Happy Paths

Agents fail at boundaries. Include fixtures for:

Invalid model tool arguments
Tool timeouts and rate limits
Empty retrieval results
Cancellation during streaming
Partial tool success
Unauthorized tool selection
Context-budget exhaustion
Trace-sink failure
Fallback failure

Verify that the original error category is preserved, cleanup runs, and no further model or tool work occurs after a terminal state.

Where Real Models Still Belong

Deterministic tests cannot prove that an answer is clear, grounded, polite, or semantically correct. Real-model evaluation remains useful for those questions.

Use a curated dataset and an explicit rubric. Calibrate judge scores against human labels, track disagreement, and review threshold failures. Because model behavior can change, compare distributions and trends rather than treating one score from one run as an unquestionable fact.

A practical schedule is:

Trigger	Recommended checks
Local save	Unit and scripted orchestration tests
Every commit	Trace invariants, policy, retries, budgets, tool contracts
Pull request	Deterministic suite plus a small real-model smoke set
Nightly or weekly	Larger semantic evaluation and regression trends
Release candidate	Human review of high-risk or changed workflows

Teams with stricter cost or reliability requirements can move real-model tests out of pull requests entirely. The important point is to make that trade-off explicit.

Final Thought

Agent output is probabilistic, but agent architecture does not need to be opaque. Validation, tool access, retries, fallbacks, budgets, state transitions, and trace structure can all have deterministic contracts.

Put those contracts behind injectable interfaces, drive them with scripted dependencies, and assert behavior through a stable metadata trace. Save real models and LLM judges for the semantic questions that ordinary code cannot answer.

The next article will turn these ideas into fast CI quality gates with reusable trace rules, baseline comparison, and actionable failure reports.

Building TypeScript-Native Observability: Async Context and Execution Flow

Raju Dandigam — Tue, 28 Jul 2026 20:55:08 +0000

A useful agent trace is not a list of timestamps. It is a causal tree.

When a TypeScript agent retrieves documents in parallel, calls a model, retries a tool, and falls back to cached data, each operation needs a trace ID, its own span ID, and the correct parent span. Without those relationships, completion order is easily mistaken for execution structure.

This article builds a small Node.js tracer to demonstrate the core mechanics: immutable async context, parent-child spans, reliable finalization, and a pluggable sink. It is intentionally smaller than a production observability library, but the design avoids several common mistakes found in minimal examples.

Completion Order Is Not Causality

Imagine three tools running in parallel:

80 ms   search_tickets completes
100 ms  load_account completes
120 ms  search_docs completes

Those timestamps describe completion order. The execution tree describes why the operations existed:

research_agent
└─ parallel_retrieval
   ├─ search_docs
   ├─ search_tickets
   └─ load_account

Both views are useful, but only the tree preserves the relationship between the agent decision and its child tools.

The normal JavaScript call stack cannot serve as that tree. Async work may resume later, execute concurrently, or outlive the function that scheduled it. Tracing therefore needs an explicit logical context.

The Context We Need

Each asynchronous branch needs two values:

type TraceContext = {
  traceId: string;
  parentSpanId: string | null;
};

When a new span starts, it reads the current context, records parentSpanId, creates its own spanId, and runs child work inside a new context whose parent is that span.

In Node.js, AsyncLocalStorage provides the propagation primitive. It carries a value through normal asynchronous resources without adding trace parameters to every application function.

Do not mutate one shared context object. Parallel siblings would race to replace the current span. Create a new context value for every nested span instead.

Define a Small Event Model

Use separate start and end events so a consumer can identify spans that never completed.

type SpanKind = 'run' | 'model' | 'tool' | 'retrieval' | 'decision';
type SpanStatus = 'ok' | 'error';

type SpanStarted = {
  version: 1;
  event: 'span_started';
  traceId: string;
  spanId: string;
  parentSpanId: string | null;
  name: string;
  kind: SpanKind;
  startedAt: string;
};

type SpanEnded = {
  version: 1;
  event: 'span_ended';
  traceId: string;
  spanId: string;
  endedAt: string;
  durationMs: number;
  status: SpanStatus;
  errorCategory?:
    | 'timeout'
    | 'validation'
    | 'authorization'
    | 'dependency'
    | 'unknown';
  metadata?: Record<string, string | number | boolean | null>;
};

type SpanEvent = SpanStarted | SpanEnded;

The schema stores a controlled error category rather than an exception message or stack. Metadata should also be operation-specific in a production design; the flat record keeps this example readable.

Keep Persistence Behind a Sink

The tracer should not own a global event array. Long-running processes would retain every trace in memory, and tests could accidentally read events from unrelated runs.

Use a sink contract instead:

export interface TraceSink {
  enqueue(event: SpanEvent): boolean;
  flush(options?: { timeoutMs?: number }): Promise<void>;
}

enqueue() is deliberately synchronous and non-throwing. It places the event in a bounded buffer and returns false when the event cannot be accepted. The sink handles batching and persistence outside the application’s critical path.

A production sink should expose accepted, dropped, retried, and failed event counts. “Tracing must not crash the request” should not become “tracing may fail silently.”

Build the Tracer

The tracer below scopes context with AsyncLocalStorage, emits one end event from a finally block, and keeps sink failure separate from application failure.

import { AsyncLocalStorage } from 'node:async_hooks';
import { randomUUID } from 'node:crypto';

type ActiveContext = {
  traceId: string;
  parentSpanId: string | null;
};

type SpanOptions = {
  metadata?: () => SpanEnded['metadata'];
};

function errorCategory(
  error: unknown,
): SpanEnded['errorCategory'] {
  if (!(error instanceof Error)) return 'unknown';
  if (error.name === 'AbortError') return 'timeout';
  if (error.name === 'ValidationError') return 'validation';
  if (error.name === 'AuthorizationError') return 'authorization';
  return 'dependency';
}

export class Tracer {
  private readonly context = new AsyncLocalStorage<ActiveContext>();

  constructor(
    private readonly sink: TraceSink,
    private readonly onDroppedEvent: (event: SpanEvent) => void = () => {},
  ) {}

  private emit(event: SpanEvent): void {
    try {
      const accepted = this.sink.enqueue(event);
      if (!accepted) this.onDroppedEvent(event);
    } catch {
      this.onDroppedEvent(event);
    }
  }

  async run<T>(name: string, work: () => Promise<T>): Promise<T> {
    const traceId = randomUUID();

    return this.context.run(
      { traceId, parentSpanId: null },
      () => this.span(name, 'run', work),
    );
  }

  async span<T>(
    name: string,
    kind: SpanKind,
    work: () => Promise<T>,
    options: SpanOptions = {},
  ): Promise<T> {
    const parent = this.context.getStore();
    if (!parent) throw new Error('span() must run inside run()');

    const spanId = randomUUID();
    const startedAt = Date.now();

    this.emit({
      version: 1,
      event: 'span_started',
      traceId: parent.traceId,
      spanId,
      parentSpanId: parent.parentSpanId,
      name,
      kind,
      startedAt: new Date(startedAt).toISOString(),
    });

    let status: SpanStatus = 'ok';
    let failureCategory: SpanEnded['errorCategory'];

    try {
      return await this.context.run(
        { traceId: parent.traceId, parentSpanId: spanId },
        work,
      );
    } catch (error) {
      status = 'error';
      failureCategory = errorCategory(error);
      throw error;
    } finally {
      this.emit({
        version: 1,
        event: 'span_ended',
        traceId: parent.traceId,
        spanId,
        endedAt: new Date().toISOString(),
        durationMs: Date.now() - startedAt,
        status,
        errorCategory: failureCategory,
        metadata: options.metadata?.(),
      });
    }
  }
}

The application exception is rethrown unchanged. A sink problem never enters the try block that classifies application work, so an observability failure cannot turn a successful model call into a failed model span.

The metadata callback runs when the span ends, which is useful when token usage or result counts are not known at start time. Keep the callback deterministic and payload-free.

Trace Parallel Work

All three operations below inherit the parallel_retrieval span as their parent even though they complete at different times.

await tracer.run('research_agent', async () => {
  return tracer.span('parallel_retrieval', 'decision', async () => {
    const [documents, tickets, account] = await Promise.all([
      tracer.span('search_docs', 'retrieval', () => searchDocs()),
      tracer.span('search_tickets', 'tool', () => searchTickets()),
      tracer.span('load_account', 'tool', () => loadAccount()),
    ]);

    return { documents, tickets, account };
  });
});

Each call to span() creates a new immutable context for its own promise chain. Sibling branches never mutate a shared current-span value.

Make Retries Visible

A retry should be a child span, not an overwritten attempt count. That preserves the duration and error category of every attempt.

await tracer.span('load_pricing', 'tool', async () => {
  try {
    return await tracer.span('attempt_1', 'tool', () => {
      return callPricingApi({ timeoutMs: 1_000 });
    });
  } catch {
    try {
      return await tracer.span('attempt_2', 'tool', () => {
        return callPricingApi({ timeoutMs: 3_000 });
      });
    } catch {
      return tracer.span('fallback_to_cache', 'tool', loadCachedPricing);
    }
  }
});

load_pricing
├─ attempt_1          error: timeout
├─ attempt_2          error: dependency
└─ fallback_to_cache  ok

In real application code, catch only the errors that should trigger a retry or fallback. Authentication, validation, and cancellation errors usually need different handling.

Preserve Function Types

A wrapper should retain argument and result types:

type AsyncFn<Args extends unknown[], Result> = (
  ...args: Args
) => Promise<Result>;

function traced<Args extends unknown[], Result>(
  tracer: Tracer,
  name: string,
  kind: SpanKind,
  fn: AsyncFn<Args, Result>,
): AsyncFn<Args, Result> {
  return (...args: Args) => {
    return tracer.span(name, kind, () => fn(...args));
  };
}

const tracedSearch = traced(
  tracer,
  'search_database',
  'tool',
  async (query: string): Promise<string[]> => searchDatabase(query),
);

const results = await tracedSearch('pricing'); // string[]

Methods that depend on this, overloaded functions, and streams need specialized wrappers. Keep those adapters explicit instead of erasing their signatures with any.

Handle Detached and Streaming Work Explicitly

Async context answers “which trace does this work belong to?” It does not decide how long a trace should remain open.

Detached work such as an unawaited background task may continue after the root span ends. Streaming work may continue after a route handler returns a response. Both need an explicit lifecycle policy:

Await work that is part of the request’s success criteria.
Start a new linked trace for a durable background job.
End a streaming span on completion, error, or cancellation.
Flush at controlled lifecycle boundaries, not after every event.
Do not assume serverless runtimes will keep executing after the response is sent.

Context propagation and lifecycle management are related, but they are not the same problem.

Control Overhead and Backpressure

Trace meaningful boundaries rather than every helper function. Agent runs, retrieval, model calls, tools, policy decisions, retries, and fallbacks usually provide enough structure.

Use a bounded queue. When the sink is slower than the event producer, choose and document a policy: drop newest, drop oldest, apply limited backpressure, or disable tracing for the run. Never allow an unbounded buffer to consume the process.

Sampling should normally happen at the trace level. Independently sampling child spans creates broken trees. Keep complete traces for selected runs, and always consider retaining errors or high-latency outliers through a documented policy.

Test the Execution Model

A tracer is correct only if it survives concurrency and failure tests:

Run two traces concurrently and assert that their span IDs never mix.
Start three Promise.all() siblings and assert that they share the expected parent.
Throw from application work and verify the original error reaches the caller.
Force the sink to reject or throw and verify the application result is unchanged.
Cancel a stream and verify the span ends once with a cancellation status.
Flush the sink and assert that every started span has exactly one end event.
Run tests in parallel workers and verify isolation.

Also test the emitted data policy. Execution trees are valuable even when prompts, outputs, tool arguments, and retrieved content are absent.

Relationship to OpenTelemetry

The concepts map naturally to distributed tracing: a run is a trace, an operation is a span, metadata becomes attributes, and context propagation preserves parentage. A production implementation can translate this event model into OpenTelemetry or another backend.

Building the small version first is still useful. It makes the invariants visible: immutable context, one parent per span, exactly one finalization, bounded persistence, and application behavior independent of sink health.

Final Thought

The difficult part of TypeScript agent tracing is not generating IDs. It is preserving causal structure while asynchronous work branches, completes out of order, retries, streams, and occasionally outlives its original request.

AsyncLocalStorage provides a strong Node.js foundation, but reliable observability also needs lifecycle rules, sink isolation, backpressure, and concurrency tests. With those pieces in place, scattered events become an execution tree that developers can trust.

Why TypeScript AI Developers Need Native Tracing Tools

Raju Dandigam — Fri, 24 Jul 2026 16:16:02 +0000

TypeScript support is easy to claim. Publish an npm package, add a few type declarations, and a tracing product can put “JavaScript and TypeScript” on its integration list.

Native support is a higher bar.

AI applications in the TypeScript ecosystem run inside concurrent Node.js servers, serverless functions, edge runtimes, background workers, test runners, and streaming web frameworks. They cross promise chains, callbacks, tool adapters, async iterators, and package boundaries. A useful tracing tool must fit those execution models without breaking types or producing disconnected spans.

The question is therefore not only, “Does this tool have a TypeScript SDK?” It is, “Does it preserve how a TypeScript agent actually runs?”

The Runtime Is Part of the Product

Consider a small agent:

async function supportAgent(question: string) {
  const category = await classifyQuestion(question);
  const documents = await retrieveDocuments(question, category);
  return generateAnswer(question, documents);
}

The source looks sequential, but a production server may execute hundreds of these functions concurrently. A flat event stream cannot tell which retrieval or model call belongs to which request.

request A: classify -> retrieve -> generate
request B: classify -> retrieve -> generate

The tracer needs a request-level trace ID and a parent span for each nested operation. More importantly, those identifiers must remain available after every asynchronous handoff.

Async Context Must Be Correct Under Concurrency

In Node.js, AsyncLocalStorage is the usual foundation for request-scoped context. It propagates state through normal promise chains and many asynchronous resources, which is much safer than storing a current trace ID in a module-level variable.

import { AsyncLocalStorage } from 'node:async_hooks';

type TraceContext = {
  traceId: string;
  spanId: string | null;
};

const context = new AsyncLocalStorage<TraceContext>();

export function withTraceContext<T>(
  value: TraceContext,
  work: () => T,
): T {
  return context.run(value, work);
}

export function currentTraceContext(): TraceContext {
  const value = context.getStore();
  if (!value) throw new Error('No active trace context');
  return value;
}

await by itself does not cause context loss when AsyncLocalStorage is used correctly. Problems usually appear when instrumentation relies on global mutable state, registers work outside the active context, crosses an unsupported runtime boundary, or integrates with a library that manages its own scheduling.

A TypeScript tracing library should test at least these cases:

Multiple agent runs executing concurrently
Nested tools and model calls
Timers, event emitters, and queued callbacks
Detached background work
Retries that start new asynchronous branches
Tests running in parallel workers

The acceptance criterion is simple: every span belongs to exactly one trace and has the expected parent.

Streaming Changes the Span Lifecycle

Many AI routes return a stream before generation has finished. The HTTP handler may complete from the framework’s perspective while tokens, tool calls, and usage data are still in flight.

A model span should therefore not end merely because the route returned a Response. It should end when the stream completes, fails, or is cancelled.

type StreamHooks<T> = {
  onStart(): Promise<void> | void;
  onChunk(chunk: T): Promise<void> | void;
  onComplete(): Promise<void> | void;
  onError(error: unknown): Promise<void> | void;
  onCancel(): Promise<void> | void;
};

async function* traceStream<T>(
  source: AsyncIterable<T>,
  hooks: StreamHooks<T>,
): AsyncGenerator<T> {
  await hooks.onStart();
  let completed = false;

  try {
    for await (const chunk of source) {
      await hooks.onChunk(chunk);
      yield chunk;
    }

    completed = true;
    await hooks.onComplete();
  } catch (error) {
    await hooks.onError(error);
    throw error;
  } finally {
    if (!completed) await hooks.onCancel();
  }
}

Real integrations also need to avoid double-finalizing a span when an error and cancellation happen close together. They should record time to first chunk, completion status, tool activity, and final token usage without storing every chunk by default.

Streaming support is not a cosmetic feature. Without it, latency is measured incorrectly, cancellations disappear, and partial responses look like successful completions.

Runtime Compatibility Is a Matrix

“TypeScript runtime” can mean several different environments:

Environment	Important tracing constraint
Long-running Node.js service	Async context, concurrency, graceful flush on shutdown
Serverless function	Cold starts, short lifetime, bounded flush time
Edge runtime	Web APIs, limited or absent Node built-ins
Background worker	Detached jobs, queue context propagation, retries
Browser	Bundle size, user privacy, no server credentials
Test runner	Isolation across files and parallel workers

A library that imports node:async_hooks or node:fs from its main entry point may fail when bundled for an edge runtime even if those features are never called. Runtime-specific code should live behind explicit exports so bundlers can exclude it.

{
  "exports": {
    ".": "./dist/core.js",
    "./node": "./dist/node.js",
    "./web": "./dist/web.js"
  }
}

The core event model can be portable. Context propagation, persistence, and flush behavior may need runtime-specific implementations.

Framework Hooks Should Follow Real Lifecycles

Framework integrations are valuable when they attach to stable lifecycle hooks rather than patching private internals. For an AI route, useful boundaries include:

Request accepted and validated
Agent run started
Retrieval started and completed
Tool call requested, executed, retried, or rejected
Model stream opened, produced its first chunk, and completed
Client disconnected or cancelled
Final usage became available
Trace flush succeeded or timed out

Different frameworks expose these boundaries differently. A TypeScript-native tracer should make the manual API first-class, then add thin adapters for frameworks such as Vercel AI SDK, LangChain.js, or OpenAI Agents SDK. The adapter should translate framework events into one internal trace model instead of forcing the core to depend on every framework.

Type Preservation Is Part of Developer Experience

Instrumentation should not erase the function signature it wraps. A generic wrapper can preserve argument and return types:

type AsyncFunction<Args extends unknown[], Result> = (
  ...args: Args
) => Promise<Result>;

function withTracing<Args extends unknown[], Result>(
  name: string,
  fn: AsyncFunction<Args, Result>,
): AsyncFunction<Args, Result> {
  return async (...args: Args): Promise<Result> => {
    return traceStep(name, () => fn(...args));
  };
}

type AgentResult = {
  answer: string;
  sources: string[];
};

const tracedAgent = withTracing(
  'support_agent',
  async (question: string): Promise<AgentResult> => {
    return runAgent(question);
  },
);

const result = await tracedAgent('Where is my order?');
result.answer;  // string
result.sources; // string[]

Overloaded functions, methods that depend on this, and streaming return types need more careful adapters. A library should document those boundaries instead of falling back to any.

Strong types also improve trace quality. Tool names, span kinds, metadata fields, and completion states can be controlled unions, which catches instrumentation mistakes before runtime.

ESM, CommonJS, and Bundlers Still Matter

Modern TypeScript packages are consumed through ESM, CommonJS, transpilers, monorepo build systems, and framework bundlers. Tracing libraries are especially sensitive because they often initialize early and integrate across package boundaries.

A production-ready package should make these behaviors clear:

Which module formats are published and tested
Whether initialization has side effects
How duplicate package copies affect global registration
Whether Node-only modules can be excluded from web and edge bundles
How source maps affect stack and code-location metadata
Whether instrumentation works before and after bundling

Automatic monkey-patching can be convenient, but explicit wrappers and adapters are easier to reason about across module systems. If automatic instrumentation is offered, it should be optional and observable.

Vendor-Neutral Events Keep the Core Flexible

TypeScript-native does not need to mean backend-specific. The instrumentation layer can emit a small internal event model, while sinks translate those events to local files, OpenTelemetry, or a hosted platform.

type TraceSink = {
  write(event: TraceEvent): Promise<void>;
  flush(options?: { timeoutMs?: number }): Promise<void>;
};

This separation lets teams use local traces for development, short-lived artifacts in CI, and centralized observability in production without rewriting every adapter.

It also creates one place to enforce privacy policy. Framework integrations should emit approved metadata into the core; destinations should not decide what sensitive payloads to collect.

How to Evaluate a TypeScript Tracing Tool

Use representative tests rather than a single hello-world script.

Test	What a passing result looks like
Two concurrent requests	Separate trace trees with no mixed spans
Nested tool call	Correct parent and timing under the model or agent step
Streaming response	First-chunk, completion, error, and cancellation are distinct
Serverless invocation	Events flush within a bounded time without delaying every request
Edge build	Node-only modules are absent from the bundle
Type check	Wrapped functions preserve arguments and results
Parallel tests	Trace state is isolated by test and worker
Privacy check	Raw prompts and tool payloads are absent by default

Also inspect failure behavior. Observability should not crash the agent because a sink is unavailable, but silent data loss is not acceptable either. Libraries should expose dropped-event counts, flush failures, and backpressure policy.

A Better Definition of TypeScript-Native

A TypeScript-native tracing tool should be:

Async-aware: Correct under concurrent and nested execution.
Stream-aware: Accurate through completion, failure, and cancellation.
Runtime-aware: Explicit about Node, serverless, edge, browser, and worker support.
Framework-adaptable: Integrated through stable lifecycle hooks.
Type-preserving: Safe wrappers without unnecessary any.
Module-conscious: Predictable across ESM, CommonJS, and bundlers.
Privacy-conscious: Metadata-first with explicit payload capture.
Backend-flexible: Able to send one event model to multiple sinks.

An npm package is only the delivery mechanism. Native support is the accumulated quality of these runtime and developer-experience decisions.

Final Thought

TypeScript agents are asynchronous, streaming, and frequently deployed across more than one runtime. Their observability tools need to understand those constraints as first-class design inputs.

When evaluating a tracer, ask whether it preserves execution context, lifecycle, types, and privacy under the conditions your application actually uses. That is the difference between an SDK that compiles and instrumentation you can trust.

The next article will build the core mechanics directly: a TypeScript execution tree using AsyncLocalStorage, parent-child spans, safe completion handling, and pluggable trace sinks.

Metadata-Only Tracing: Privacy-First Observability for AI Agents

Raju Dandigam — Tue, 21 Jul 2026 19:14:31 +0000

Agent tracing is useful because it reveals execution structure: which step ran, which tool failed, where a retry occurred, how long a model call took, and how the token budget changed.

The easiest implementation is to capture every prompt, argument, result, and response. It is also the easiest way to turn an observability system into a second copy of sensitive application data.

Metadata-only tracing takes a different approach. It records the behavior of an agent without storing its raw payloads by default. The result is not zero-risk telemetry, but it is a much smaller and more governable data surface.

The Design Goal

A useful metadata trace should answer operational questions such as:

Which steps executed, and in what order?
Which model and tool operations succeeded or failed?
Where did latency accumulate?
How many retries and fallbacks occurred?
How many input, cached-input, and output tokens were used?
Did retrieval return results, and how much context was assembled?
Which validation or policy gate blocked the run?

It should not answer these questions unless a separate capture policy explicitly allows it:

What did the user say word for word?
What was the complete prompt or model response?
Which email address, account number, or authorization token was used?
What records or documents did a tool return?

That boundary keeps everyday traces useful without making full-fidelity capture the default.

Metadata-Only Does Not Mean Anonymous

Metadata can still be sensitive. A workflow name, fine-grained location, unique identifier, decision label, or rare error category may identify a person or reveal confidential business activity.

The relevant distinction is not “payload versus harmless metadata.” It is necessary, classified metadata versus unbounded content. Every field still needs a purpose, an owner, and a retention policy.

Avoid free-form metadata bags. A type such as Record<string, string | number> constrains value shapes, but it does not prevent a developer from adding email, prompt, or accessToken.

Define Metadata by Operation

Operation-specific types make the intended schema visible during code review and prevent arbitrary fields from spreading through the trace system.

type StepMetadata = {
  retrieval: {
    source: 'knowledge_base' | 'ticket_index';
    requestedTopK: number;
    resultCount: number;
    contextTokens: number;
  };
  model: {
    provider: 'openai' | 'anthropic' | 'google' | 'other';
    model: string;
    inputTokens: number;
    cachedInputTokens: number;
    outputTokens: number;
    finishReason: 'stop' | 'length' | 'tool' | 'other';
  };
  tool: {
    tool: 'lookup_order' | 'search_docs' | 'create_ticket';
    result: 'found' | 'not_found' | 'created' | 'rejected';
    retryCount: number;
  };
  policy: {
    policy: 'input_validation' | 'tool_authorization' | 'output_check';
    outcome: 'allow' | 'block';
    reason: 'valid' | 'invalid_shape' | 'not_authorized' | 'unsafe_output';
  };
};

type StepKind = keyof StepMetadata;

Controlled vocabularies are intentional. They make dashboards stable, reduce high-cardinality fields, and force new data collection to be reviewed as a schema change.

Model names may remain dynamic, but they should still be length-limited and normalized. User-controlled strings should not be copied into these fields.

Use a Small, Versioned Event Envelope

The trace envelope should support parent-child relationships without carrying application payloads.

type TraceStatus = 'ok' | 'error';

type StepCompleted<K extends StepKind = StepKind> = {
  version: 1;
  event: 'step_completed';
  traceId: string;
  spanId: string;
  parentSpanId: string | null;
  timestamp: string;
  kind: K;
  name: string;
  status: TraceStatus;
  durationMs: number;
  errorCategory?:
    | 'timeout'
    | 'rate_limit'
    | 'validation'
    | 'authorization'
    | 'dependency'
    | 'unknown';
  metadata?: StepMetadata[K];
};

Versioning matters because trace artifacts often outlive the code that produced them. A version field lets readers migrate or reject incompatible events rather than guessing their shape.

Do not include raw error messages or stack traces in the default event. Both frequently contain payload fragments, file paths, headers, or query values. Map exceptions to a controlled category and keep richer diagnostics behind a restricted capture mode.

Preserve Parent-Child Context in TypeScript

AsyncLocalStorage can carry trace and parent-span identifiers across promise chains without passing them through every function signature. The tracer below emits completion events and requires each operation to return metadata that matches its declared kind.

import { AsyncLocalStorage } from 'node:async_hooks';
import { randomUUID } from 'node:crypto';

type TraceContext = {
  traceId: string;
  spanId: string | null;
};

type StepOutput<T, K extends StepKind> = {
  value: T;
  metadata: StepMetadata[K];
};

export interface TraceSink {
  write(event: StepCompleted): Promise<void>;
}

const traceContext = new AsyncLocalStorage<TraceContext>();

function categorizeError(error: unknown): StepCompleted['errorCategory'] {
  if (!(error instanceof Error)) return 'unknown';
  if (error.name === 'AbortError') return 'timeout';
  if (error.name === 'ValidationError') return 'validation';
  if (error.name === 'AuthorizationError') return 'authorization';
  return 'dependency';
}

export async function runTrace<T>(
  work: () => Promise<T>,
): Promise<T> {
  return traceContext.run(
    { traceId: randomUUID(), spanId: null },
    work,
  );
}

export async function traceStep<K extends StepKind, T>(
  sink: TraceSink,
  kind: K,
  name: string,
  work: () => Promise<StepOutput<T, K>>,
): Promise<T> {
  const parent = traceContext.getStore();
  if (!parent) throw new Error('traceStep must run inside runTrace');

  const spanId = randomUUID();
  const startedAt = Date.now();

  try {
    const output = await traceContext.run(
      { traceId: parent.traceId, spanId },
      work,
    );

    await sink.write({
      version: 1,
      event: 'step_completed',
      traceId: parent.traceId,
      spanId,
      parentSpanId: parent.spanId,
      timestamp: new Date().toISOString(),
      kind,
      name,
      status: 'ok',
      durationMs: Date.now() - startedAt,
      metadata: output.metadata,
    });

    return output.value;
  } catch (error) {
    await sink.write({
      version: 1,
      event: 'step_completed',
      traceId: parent.traceId,
      spanId,
      parentSpanId: parent.spanId,
      timestamp: new Date().toISOString(),
      kind,
      name,
      status: 'error',
      durationMs: Date.now() - startedAt,
      errorCategory: categorizeError(error),
    });

    throw error;
  }
}

The sink can write to a local NDJSON file during development or export approved events to an observability backend. Capture policy belongs before the sink so changing destinations cannot silently increase what is collected.

Instrument an Agent Without Capturing Content

The model and retrieval operations can use sensitive values in memory while returning only bounded operational metadata to the tracer.

const answer = await runTrace(async () => {
  const documents = await traceStep(
    sink,
    'retrieval',
    'retrieve_support_docs',
    async () => {
      const value = await searchDocuments(userQuestion);

      return {
        value,
        metadata: {
          source: 'knowledge_base',
          requestedTopK: 5,
          resultCount: value.length,
          contextTokens: countDocumentTokens(value),
        },
      };
    },
  );

  return traceStep(
    sink,
    'model',
    'generate_support_answer',
    async () => {
      const response = await callModel(userQuestion, documents);

      return {
        value: response.text,
        metadata: {
          provider: 'other',
          model: response.model.slice(0, 80),
          inputTokens: response.usage.inputTokens,
          cachedInputTokens: response.usage.cachedInputTokens ?? 0,
          outputTokens: response.usage.outputTokens,
          finishReason: response.finishReason,
        },
      };
    },
  );
});

This trace can reveal an empty retrieval result, an oversized context, a length-limited response, or an unexpectedly expensive model call. It never needs the user question or document text.

What a Useful Trace Looks Like

support_agent  1,184 ms  ok
├─ retrieve_support_docs   96 ms  ok
│  source=knowledge_base resultCount=5 contextTokens=0
└─ generate_support_answer 1,072 ms ok
   inputTokens=640 cachedInputTokens=0 outputTokens=84 finishReason=stop

The zero-token retrieval context is immediately suspicious even though the trace does not expose the documents. Metadata narrows the investigation; a developer can then enable selective local capture for that step if the issue cannot be reproduced otherwise.

Enforce Runtime Limits Too

TypeScript types disappear at runtime, and trace data may come from JavaScript adapters or external libraries. Validate events before writing them:

Reject unknown keys and unsupported schema versions.
Cap names and other strings to a small maximum length.
Require finite, non-negative numeric values.
Limit metadata field counts and serialized event size.
Reject keys associated with payloads, credentials, headers, or free-form content.
Fail closed when validation cannot complete.

Schema validation libraries can enforce the structural rules. The operation-specific builders should still remain the primary policy boundary.

Test the Negative Requirements

Privacy requirements are often about what must never appear. Encode those expectations in tests.

const forbiddenKeys = [
  'prompt',
  'response',
  'args',
  'resultBody',
  'authorization',
  'cookie',
  'email',
] as const;

function assertNoForbiddenKeys(event: unknown): void {
  const serialized = JSON.stringify(event).toLowerCase();

  for (const key of forbiddenKeys) {
    if (serialized.includes(`"${key.toLowerCase()}"`)) {
      throw new Error(`Forbidden trace key: ${key}`);
    }
  }
}

Add representative secrets and personal data to test fixtures, run the agent, and assert that none of those values appear in the emitted trace. This is not a substitute for a broader security review, but it catches regressions when instrumentation changes.

Know When Metadata Is Not Enough

Metadata-only tracing is excellent for timing, topology, retries, token usage, policy outcomes, and broad error localization. It cannot explain every semantic failure.

When exact content is necessary, use a separate diagnostic mode with these constraints:

Enable it for a specific trace, step, or short time window.
Keep capture local or in a restricted incident environment.
Allowlist fields instead of recording complete objects.
Apply deterministic redaction and secret scanning.
Display that enhanced capture is active.
Delete the artifact automatically after a short retention period.

The escalation path should be obvious, but it should require intent.

A Practical Default

Start with a versioned event envelope, operation-specific metadata types, async parent-child context, controlled error categories, and runtime validation. Store timing, status, token usage, counts, and bounded labels. Keep prompts, outputs, tool payloads, retrieved text, headers, and environment values out of the default schema.

Metadata-only tracing will not answer every debugging question. It will answer a large portion of them while substantially reducing the amount of sensitive data your observability system must protect.

The next article will focus on the TypeScript runtime itself: async context propagation, module boundaries, serverless execution, and the hooks a tracing tool needs to handle cleanly.

Keep Your AI Agent Traces on Your Machine: A Local-First Approach

Raju Dandigam — Fri, 17 Jul 2026 19:32:17 +0000

Adding tracing to an AI agent changes more than the debugging experience. It also creates a new data path.

A trace can contain user messages, system instructions, retrieved documents, tool arguments, tool results, model output, identifiers, and internal business logic. Sending that trace to an external service may therefore be equivalent to sending the underlying application data.

The right first question is not simply, “Does this tracing SDK have a good dashboard?” It is, “What does it capture, where does it go, and who controls it?”

A local-first tracing design gives developers useful execution visibility while keeping export and sharing decisions explicit. It is not a rejection of hosted observability. It is a safer default for the development loop and a clearer boundary for sensitive data.

Why Agent Traces Need Their Own Data Policy

Traditional telemetry can also expose sensitive information, but AI traces are unusually payload-rich. The data needed to explain an agent’s behavior is often the same data the agent was asked to process.

Trace source	What it may reveal
User and model messages	Personal data, confidential requests, generated decisions
System prompts	Internal policies, workflow rules, security assumptions
Tool calls	Identifiers, query parameters, authorization context
Tool results	Customer records, database rows, third-party API data
Retrieval context	Private documents, proprietary knowledge, access-controlled text
Errors and retries	Stack traces, request fragments, infrastructure details

This does not mean that rich traces are always inappropriate. It means trace data needs an owner, a classification, a retention period, and an approved destination just like any other sensitive dataset.

What Local-First Tracing Means

In a local-first workflow, the initial trace is written to a developer machine, an isolated test workspace, a CI runner, or infrastructure controlled by the organization. Nothing is exported merely because tracing was enabled.

agent execution
    |
    v
capture policy
    |
    v
controlled local sink
    |
    +--> local inspection
    |
    +--> reviewed, reduced export --> approved shared platform

The important property is intentional movement. Developers can inspect the execution locally, reduce the data to what is needed, and share only an approved artifact.

Local-first is not the same as automatically secure. A laptop may be compromised, a workspace may be synchronized to a consumer cloud account, and a CI artifact may be visible to more people than expected. Local traces still require access controls, retention rules, and safe defaults.

Define Capture Modes Before Writing Code

A single on/off tracing switch is too coarse for most agents. Use explicit capture modes instead:

Mode	Captured data	Typical use
Metadata	Names, timing, status, token counts, safe categories	Default development and production telemetry
Selective payload	Approved fields from specific tools or steps	Focused debugging in a controlled environment
Full fidelity	Prompts, outputs, and raw payloads	Temporary incident investigation with short retention

Full-fidelity capture should require a deliberate configuration change, produce a visible warning, and expire automatically. It should never be enabled silently by a dependency update or a default environment variable.

Make the Trace Schema Safe by Construction

The most reliable redaction strategy is not capturing unnecessary data. A typed metadata event makes the safe path easy and prevents arbitrary payloads from drifting into normal traces.

type TraceStatus = 'ok' | 'error';
type TraceKind = 'agent' | 'model' | 'tool' | 'retrieval';
type SafeValue = string | number | boolean | null;

export type TraceEvent = {
  version: 1;
  traceId: string;
  spanId: string;
  parentSpanId?: string;
  timestamp: string;
  kind: TraceKind;
  name: string;
  status: TraceStatus;
  durationMs?: number;
  inputTokens?: number;
  outputTokens?: number;
  metadata: Record<string, SafeValue>;
};

Notice what the schema does not include: raw prompts, free-form tool arguments, complete tool results, or authorization headers. Those fields require a separate, more restrictive capture path.

For an order lookup, prefer an allowlisted projection:

type OrderLookupResult = {
  found: boolean;
  status?: 'processing' | 'shipped' | 'delivered';
  itemCount?: number;
};

function orderLookupMetadata(
  result: OrderLookupResult,
): TraceEvent['metadata'] {
  return {
    resultFound: result.found,
    orderStatus: result.status ?? 'unknown',
    itemCount: result.itemCount ?? 0,
  };
}

This event can answer whether the lookup ran, succeeded, returned a record, and produced the expected state. It does not need an email address, street address, payment details, or the complete database response.

Write NDJSON to a Controlled Local Sink

Newline-delimited JSON is a practical local format because it is append-friendly, streamable, and easy to inspect with standard tools. The following sink creates a private directory and file on platforms that support POSIX permissions:

import { appendFile, mkdir } from 'node:fs/promises';
import { dirname, resolve } from 'node:path';

export class LocalTraceSink {
  constructor(
    private readonly filePath = resolve(
      '.agent-traces',
      'traces.ndjson',
    ),
  ) {}

  async write(event: TraceEvent): Promise<void> {
    await mkdir(dirname(this.filePath), {
      recursive: true,
      mode: 0o700,
    });

    await appendFile(
      this.filePath,
      `${JSON.stringify(event)}\n`,
      {
        encoding: 'utf8',
        flag: 'a',
        mode: 0o600,
      },
    );
  }
}

For concurrent or high-volume tracing, put writes behind a queue or use a storage engine designed for parallel writers. Also verify existing file permissions at startup because the mode option applies when a file is created; it does not repair an already permissive file.

Add the trace directory to .gitignore, keep it outside folders that synchronize automatically, and do not serve it from a development web root.

.agent-traces/

Use Stable Metadata, Not Hidden Payloads

Safe metadata should describe execution, not smuggle raw content under a different key. Useful fields include:

Operation and tool names from a controlled vocabulary
Success, failure, timeout, or cancellation status
Duration, retry count, and fallback usage
Model name and token usage
Retrieval result count and score range
Input and output size in bytes or tokens
Validation result and error category
Approved business categories such as order_status

Be careful with hashes. A plain hash of a predictable identifier, such as an email address or short account number, can often be reversed by guessing. If cross-trace correlation is necessary, use a keyed HMAC managed outside the trace store, rotate the key, and document who can perform the correlation.

Treat Redaction as Defense in Depth

Regex replacement is useful for catching obvious patterns, but it is not a complete privacy control. Addresses, names, medical details, credentials, and business-sensitive text do not all have reliable patterns.

A stronger export pipeline works in this order:

Start from a metadata-only event schema.
Allowlist the exact payload fields approved for a specific workflow.
Apply structured redaction by field name and data type.
Scan the resulting artifact for secrets and sensitive patterns.
Fail closed when the scan or policy check cannot complete.
Require review before external or public sharing.

Redaction should produce a new artifact. Preserve the restricted source only for its approved retention period, and never overwrite it in a way that makes the audit trail ambiguous.

Protect the Local Boundary

Local traces deserve the same operational hygiene as other sensitive developer data:

Restrict directory and file permissions.
Encrypt the disk and lock the development session when unattended.
Keep traces out of source control, terminal paste services, and automatic backups unless those systems are approved.
Store encryption keys and service credentials outside trace files.
Apply a short default retention period and provide an explicit cleanup command.
Record when full-fidelity capture is enabled and who enabled it.
Disable payload capture in shared development environments by default.

Deleting old traces is part of the design, not housekeeping. A useful default is to expire development traces after hours or days, not months.

Build a Safe CI Workflow

CI is controlled infrastructure, but it is not necessarily private. Logs and artifacts may be available to repository contributors, external pull requests, support personnel, or integrated services.

A safer CI flow is:

test fails
  -> metadata trace stays in the isolated workspace
  -> export policy creates a reduced artifact
  -> secret and sensitive-data scans run
  -> artifact uploads only when policy passes
  -> artifact expires after a short retention window

Avoid printing raw traces to the build log. Apply extra restrictions to workflows triggered from forks, and make artifact access and retention explicit in repository settings.

Use Hosted Observability Intentionally

Production systems often need centralized metrics, alerts, trace correlation, and team-wide dashboards. Local-first tracing can coexist with those needs.

A practical split is:

Development: Metadata is stored locally; temporary payload capture is opt-in.
Pull requests: Only reduced, scanned artifacts are shared with reviewers.
Production: Approved metadata is exported to the observability platform; restricted payload capture uses a separate incident process.

Before exporting agent telemetry, review the destination’s access model, retention controls, deletion behavior, regional storage, subprocessors, and incident response process. Ensure the export matches your organization’s security and privacy requirements.

Questions to Answer Before Enabling Tracing

What data can each capture mode collect?
Does the SDK upload anything automatically?
Can raw prompts, outputs, and tool payloads be disabled?
Where are local and CI traces physically stored?
Who can access them, and how is that access revoked?
How quickly are traces deleted?
What policy controls export to a shared platform?
Can developers verify the exact artifact before external sharing?
What happens if redaction or scanning fails?

If the team cannot answer these questions, it is too early to enable full-fidelity tracing.

Final Thought

Observability should explain an agent’s execution without becoming an accidental copy of everything the agent touched. Start with typed metadata, keep the initial trace inside a controlled boundary, and make richer capture temporary and explicit.

The principle is simple: capture the minimum, inspect locally, and export intentionally.

The next article in this series will go deeper into metadata-only tracing: how to preserve execution structure, latency, token usage, retries, and error context without storing raw prompts or tool payloads.

Token Drift Explained: Why Your Agent Gets Slower and More Expensive

Raju Dandigam — Fri, 17 Jul 2026 00:20:06 +0000

Your agent feels fast during a demo. Then a real session reaches twenty turns, several tools have returned large payloads, and every response starts taking longer and costing more.

This pattern is often called token drift: the effective input context grows as an agent carries more conversation history, tool output, retrieved documents, and state into each model call. The model itself is not gradually becoming less efficient. The application is asking it to process more material on every turn.

Token drift is manageable, but only if context is treated as a budgeted system resource rather than an unlimited transcript.

What Token Drift Actually Means

Most conversational agents build each request from several sources: a system prompt, tool definitions, recent messages, retrieved context, durable memory, and sometimes a summary of older work. Whether the application sends that state on every request or a provider manages part of it, the model still has an effective context to process.

Consider an illustrative session:

Turn	Effective input	What changed
1	1,200 tokens	System prompt, tools, and one user message
8	6,900 tokens	Conversation history and two tool results
20	18,400 tokens	More history, retrieved documents, and accumulated state

The exact price and latency depend on the model, provider, cache behavior, and workload. The important signal is the trend: later calls repeatedly process a larger context.

Why Agents Accumulate Tokens So Quickly

Conversation history is only one source of growth. Production agents often accumulate tokens in several places at once:

Repeated transcripts: Every prior user and assistant message remains in context.
Tool schemas: Large tool descriptions and JSON schemas may be attached to every model call.
Tool results: Search results, stack traces, database rows, and API responses can be much larger than the user's request.
Retrieved documents: RAG pipelines sometimes add too many chunks or keep stale retrieval results across turns.
Retries and repairs: Failed tool calls and validation errors may be appended even after they stop being useful.
Duplicated memory: A summary, structured state, and raw transcript may all contain the same facts.

This is why limiting the number of chat messages is useful but incomplete. A ten-message conversation can still be expensive if one tool returned 30,000 tokens of JSON.

The Cost Curve

Suppose a request has a fixed cost of s tokens for system instructions and tools, and each turn adds roughly m tokens of new history. The input on turn t is approximately:

input(t) = s + (t * m)

Per-turn input grows roughly linearly. However, the cumulative input processed across n turns is approximately:

session input = (n * s) + m * n * (n + 1) / 2

That second term is quadratic. This distinction matters: the final request is not necessarily exponentially larger, but repeatedly resending a growing transcript can make the total session cost rise much faster than the number of turns.

Prompt caching can reduce the cost of repeated prefixes on providers that support it. It does not remove context-window limits, irrelevant-history problems, or the need to control tool and retrieval payloads.

Measure Drift Before You Trim

Use the token usage returned by the model API whenever possible. Character-based estimates are acceptable for admission control, but they are not a reliable billing metric across models or languages.

Track usage per model call, not only per user request. One agent turn may contain planning, tool calls, retries, and a final response.

type UsageSample = {
  turn: number;
  step: string;
  inputTokens: number;
  outputTokens: number;
  cachedInputTokens?: number;
  durationMs: number;
};

function averageInputGrowth(samples: UsageSample[]): number {
  if (samples.length < 2) return 0;

  const byTurn = new Map<number, number>();
  for (const sample of samples) {
    byTurn.set(
      sample.turn,
      (byTurn.get(sample.turn) ?? 0) + sample.inputTokens,
    );
  }

  const totals = [...byTurn.entries()]
    .sort(([a], [b]) => a - b)
    .map(([, tokens]) => tokens);

  if (totals.length < 2) return 0;

  const deltas = totals.slice(1).map((value, index) => {
    return value - totals[index];
  });

  return Math.round(
    deltas.reduce((sum, delta) => sum + delta, 0) / deltas.length,
  );
}

A useful trace should also record the model, operation, tool name, retry count, retrieval chunk count, and whether cached input was used. Avoid recording sensitive prompt content unless your privacy policy explicitly permits it.

Use a Context Budget, Not a Message Limit

A sliding window is a reasonable fallback, but a token budget is the stronger control because message sizes vary dramatically. Reserve space for the output and for any tool calls the model may make, then allocate the remainder to fixed context and recent turns.

Treat a complete agent turn as the unit of removal. Dropping an individual tool result while keeping the assistant message that refers to it can leave an invalid or confusing transcript.

type AgentMessage = {
  role: 'user' | 'assistant' | 'tool';
  content: string;
};

type ConversationTurn = {
  id: string;
  messages: AgentMessage[];
  tokenCount: number;
};

type ContextPlan = {
  turns: ConversationTurn[];
  omittedTurnIds: string[];
  estimatedInputTokens: number;
};

type ContextBudget = {
  maxContextTokens: number;
  reservedOutputTokens: number;
  systemPromptTokens: number;
  toolSchemaTokens: number;
  summaryTokens: number;
  retrievedContextTokens: number;
};

function planContext(
  turns: ConversationTurn[],
  budget: ContextBudget,
): ContextPlan {
  const fixedTokens =
    budget.systemPromptTokens +
    budget.toolSchemaTokens +
    budget.summaryTokens +
    budget.retrievedContextTokens;

  let remaining =
    budget.maxContextTokens -
    budget.reservedOutputTokens -
    fixedTokens;

  if (remaining <= 0) {
    throw new Error('Fixed context exceeds the available input budget');
  }

  const selected: ConversationTurn[] = [];

  for (let index = turns.length - 1; index >= 0; index -= 1) {
    const turn = turns[index];

    if (turn.tokenCount > remaining) break;

    selected.unshift(turn);
    remaining -= turn.tokenCount;
  }

  if (selected.length === 0 && turns.length > 0) {
    throw new Error('The latest turn is larger than the context budget');
  }

  const selectedIds = new Set(selected.map((turn) => turn.id));
  const omittedTurnIds = turns
    .filter((turn) => !selectedIds.has(turn.id))
    .map((turn) => turn.id);

  return {
    turns: selected,
    omittedTurnIds,
    estimatedInputTokens:
      fixedTokens +
      selected.reduce((sum, turn) => sum + turn.tokenCount, 0),
  };
}

The token counts in this example should come from the provider's tokenizer or a compatible tokenizer for the chosen model. Recalculate the plan whenever tools, retrieval results, or the summary changes.

Separate Conversation From State

Not every fact belongs in the transcript. Long-running agents become more reliable when they store important state explicitly:

type TaskState = {
  objective: string;
  constraints: string[];
  decisions: Array<{
    choice: string;
    reason: string;
  }>;
  openQuestions: string[];
  artifactIds: string[];
};

Structured state is compact, inspectable, and easier to validate than a prose summary. It also prevents a critical decision from disappearing simply because an old message fell outside the recent window.

A practical context often has four layers:

Stable instructions: The system prompt and safety constraints.
Structured task state: Goals, decisions, identifiers, and unresolved work.
Compressed history: A summary of older turns.
Recent turns: The verbatim messages and tool exchanges needed for continuity.

This layered design is usually more useful than importance-scoring individual messages. Heuristic scores can be difficult to explain, and they may discard a quiet but essential constraint.

Summarize Deliberately

Summarization is valuable when sessions last long enough that a recent window loses necessary context. It should not run blindly on every turn.

Create or refresh a summary only when older turns cross a threshold. Ask the summarizer to preserve facts, decisions, constraints, failures, unresolved questions, and references to external artifacts. Store the summary separately from the raw transcript so it can be reviewed or regenerated.

Summaries are lossy and can introduce errors. Keep canonical values such as account IDs, approval state, financial amounts, and file paths in structured storage rather than trusting generated prose.

Bound Tools and Retrieval Too

Many apparent memory problems are actually payload problems. A useful context policy should also:

Send only the tool definitions available in the current state.
Replace large tool results with a compact, typed projection.
Store full results externally and keep an ID or URL in context.
Limit retrieval by both relevance and token budget.
Deduplicate overlapping chunks before adding them to the prompt.
Remove failed retries once their diagnostic value has expired.

For example, an agent rarely needs an entire database response. It may need five selected fields, the result count, and an identifier it can use to fetch more data later.

Choose a Strategy by Workload

Workload	Good starting strategy
Short support chat	Recent-turn window with a hard input budget
Tool-heavy workflow	Compact tool schemas and bounded results
Long research session	Structured state, summary, and recent turns
Knowledge assistant	Token-budgeted retrieval plus recent turns
High-stakes workflow	Structured state with validation and auditable summaries

Most systems do not need a sophisticated memory-ranking algorithm on day one. They need visibility, a budget, and clear rules for what may enter context.

Production Quality Gates

Context reduction is successful only if it lowers resource use without damaging task quality. Compare the policy against representative sessions and track:

Input tokens per turn and per completed task
Cached versus uncached input tokens
Model-call count, retries, and tool failures
Median and tail latency
Task completion and human correction rates
Missing facts or constraints after compaction
Summary refresh frequency and cost

Run deterministic regression cases for critical facts and tool sequences. For example, verify that an agent still remembers an approval constraint after the originating turn has moved into summarized history.

A Practical Baseline

Start with a simple policy: measure real usage, reserve output headroom, cap the input context, keep structured task state, retain a small window of complete recent turns, and summarize older history only when necessary. Bound tool and retrieval payloads independently.

Token drift is not mysterious model degradation. It is accumulated application state. Once that state is measured and budgeted, cost and latency become predictable engineering trade-offs instead of late-session surprises.

The Hidden Cost of AI Agents: Tokens, Tools, Retries, and Latency

Raju Dandigam — Wed, 15 Jul 2026 22:48:10 +0000

AI agents look simple at first.

You take a model, add a prompt, maybe connect a tool, and it works. It feels like you are just making one API call and getting an answer.

That illusion disappears the moment the agent starts doing real work.

In production, an agent is not one call. It is a loop. It may call the model multiple times, retrieve memory, execute tools, retry on failure, and refine its own output.

That is where cost shows up.

Not just in tokens, but in latency, complexity, and system load.

The mental model most people miss

Most tutorials present agents like this:

User → Model → Response

Real systems look more like this:

User → Runtime → Model → Tools → Memory → Validation → Model → Response

And that flow may repeat several times.

Each step adds cost.

Not just money. Time, complexity, and failure risk.

1. Token cost is not just one request

The first hidden cost is tokens.

Developers often assume a single request with a fixed prompt. In reality, an agent may call the model multiple times within one run.

For example:

One call to decide what to do
One call after retrieving memory
One call after tool execution
One call to generate the final response

Each call includes input tokens and output tokens.

If you also include memory retrieval, the prompt gets larger over time. That increases token usage even further.

A simple way to think about it:

const totalTokenCost =
  (numberOfCalls) *
  (inputTokens + outputTokens);

The problem is that numberOfCalls is not always predictable. It depends on how the agent behaves.
This is why cost can grow faster than expected.

2. Tool calls are not free

Tool usage is often treated as a free extension of the model. It is not.
Each tool call adds:
Network latency
External API cost
Failure scenarios
Additional model calls after execution
For example, a simple flow might look like:

Model decides → Call tool → Tool responds → Model interprets → Continue

Even if the tool itself is cheap, the surrounding orchestration is not.
In many cases, the cost of using a tool is not the tool itself, but the extra model calls and latency it introduces.

3. Retries multiply everything

Retries are where costs start to compound.
If a tool fails, the agent may retry. If the model returns invalid output, the system may retry. If validation fails, the system may retry again.
Each retry is not just one extra call. It repeats the entire step.
A simple retry loop might look like this:

for (let i = 0; i < 3; i++) {
  try {
    return await callTool(input);
  } catch {
    continue;
  }
}

Now imagine this happening inside an agent loop.
One failure can lead to:
Multiple tool calls
Multiple model calls
Longer execution time
Retries are necessary, but without limits, they become expensive quickly.

4. Latency grows with each step

Latency is often underestimated.
Each model call takes time. Each tool call adds network delay. Each retry increases total execution time.
Even if each step is fast, the combined latency can be noticeable.
A simple breakdown:
Model call: 500ms to 2s
Tool call: 200ms to 1s
Memory retrieval: 100ms to 300ms
Now combine them across multiple steps.
An agent that feels “instant” in a demo can easily take several seconds in production.
This is not always a problem, but it becomes important for user experience.

5. Memory adds both value and cost

Memory is powerful, but it is not free.
Every time you retrieve memory, you add:
Query cost (vector search or database lookup)
Additional tokens (more context sent to the model)
Complexity in prompt construction
If memory is not filtered carefully, it can:
Increase token usage significantly
Add irrelevant context
Reduce model accuracy
The key is not more memory, but better memory.
Retrieve only what is relevant to the current goal.

6. Reflection and “thinking” steps increase usage

Many modern agent patterns include reflection.
The agent may:
Evaluate its own output
Re-plan its next step
Summarize intermediate results
These patterns improve quality, but they also add more model calls.
For example:

Model → Draft response
Model → Critique response
Model → Refine response

This can double or triple token usage.
Reflection is useful, but it should be used intentionally.

7. Cost is not just money

When people talk about cost, they usually mean API pricing.
In practice, cost also includes:
Latency. Slow agents reduce user experience.
System load. More calls mean more infrastructure usage.
Failure surface. More steps increase the chance of something going wrong.
Debugging complexity. More moving parts make issues harder to trace.
This is why cost should be treated as an architectural concern, not just a billing concern.

A simple way to think about it

If I had to simplify everything into one idea, it would be this:
Each new capability multiplies cost.
More tools → more calls
More memory → more tokens
More retries → more loops
More reasoning → more model usage
Agents do not scale linearly. They scale multiplicatively.

What actually works in practice

In real TypeScript systems, I try to keep things controlled.
Limit the number of steps. Do not let the agent run indefinitely.
Keep prompts small. Only include relevant context.
Use tools intentionally. Do not expose everything.
Set retry limits. Avoid infinite loops.
Track usage. Measure tokens, latency, and failures.
These are not optimizations. They are guardrails.

The real takeaway

AI agents are powerful, but they are not free abstractions.
Every decision you add to the system has a cost. Every layer adds complexity. Every retry multiplies usage.
The goal is not to remove these features. The goal is to use them intentionally.
A simple, controlled agent that solves a specific problem is often more valuable than a complex agent that tries to do everything.
Because in production systems, efficiency and reliability matter more than flexibility.
And understanding the hidden cost is the first step toward building something that actually scales.