Start with a Frustrating Scenario
Your Agent runs for three minutes, then throws an error.
You stare at the logs — nothing useful. You don't know which tools it called, where it got stuck, or where the tokens went. You certainly don't know why it failed.
That's what missing observability looks like in practice.
Why Observability Matters More for Agents
With a traditional service, you can add breakpoints and read stack traces. Agents are different.
Agents are non-deterministic. The same input might produce a completely different sequence of tool calls. You can't set a breakpoint inside "the model's decision-making" — that's a black box.
Debugging requires log reconstruction. The model's 'reasoning' only manifests in the text and tool calls it outputs. You need to capture all of that so you can rebuild the inference chain after the fact.
Token costs are opaque. Across a multi-turn Agent conversation, which step is the most expensive? Is it the long tool result in turn three? The oversized system prompt? Without metering, you can't even identify where to optimize.
Production incidents need evidence. When a user reports a bad answer, you need to reconstruct the full execution chain from that moment — which tools ran, what they returned, what the model saw.
Session Logs: The Most Complete Observability Data Source
Recall from article 05: a dsh Session is an append-only log of typed events.
That design isn't only about persistence — the log itself is the most complete observability source you have.
Every Session event contains:
Structure of every SessionEvent:
- type: event type (e.g. 'tool/call', 'turn/end', 'assistant/message')
- seq: monotonically increasing sequence number (starts at 0)
- time: Unix timestamp in milliseconds
- data: typed event payload (structure varies by type)
This means: as long as you can read the Session log, you can reconstruct the entire execution — what happened at each step, how long it took, whether anything failed.
Real-Time Listening: session/event
You don't have to wait for the log to finish before analyzing it. dsh provides the session/event event, which fires for every entry as it's written:
// Listen to all events on a Session in real time
ctx.on('session/event', (session, event) => {
// This callback fires for every event
console.log(`[${event.type}] seq=${event.seq} time=${event.time}`)
// Check for tool calls
if (event.type === 'tool/call') {
// event.data.name is the tool name
// event.data.arguments is a raw JSON string (model output, not yet parsed)
// event.data.callId is the unique ID for this invocation
console.log(` Tool: ${event.data.name}`)
console.log(` Args: ${event.data.arguments}`)
}
// Check for tool execution results
if (event.type === 'tool/result') {
// Detect failure by looking for isError: true in the content blocks
const isError = event.data.message.content.some(
b => b.type === 'tool_result' && b.isError
)
console.log(` Result: ${isError ? 'ERROR' : 'OK'}`)
}
})
During development, this listener is invaluable — you see what the Agent is doing in real time, not just after it finishes.
Token Metering: ctx.tokenMeter
Knowing 'what happened' is step one. Knowing 'how much it cost' is equally important.
dsh provides ctx.tokenMeter to measure the token pressure of the current Session:
// TokenMeasurement interface (from packages/llm/token-meter/src/types.ts)
interface TokenMeasurement {
// How many events this measurement consumed (for caching, avoids recomputation)
readonly logRevision: SessionLogOffset
// Total token pressure for the current request (input + output combined)
readonly totalTokens: number
// Token count of the current surface (history visible to the model)
readonly surfaceTokens: number
// Signed delta of surface tokens vs. the last successful request
// (can be negative if context shrank)
readonly surfaceDeltaTokens: number
// Surface nodes in positional order with their individual token counts
// (lets you see how many tokens each message contributes)
readonly nodes: readonly TokenSurfaceNode[]
}
Key concepts:
- surfaceTokens: How many tokens the model actually sees as input for this request. This drives the 'input token' portion of your API bill.
- totalTokens: Input + output combined — the full cost of the request.
- surfaceDeltaTokens: How much the surface grew since the last request. If this keeps increasing, your context is ballooning and you might need a compression strategy.
Usage example:
// Print a token summary at the end of each Turn
ctx.on('session/event', (session, event) => {
// Only care about Turn-end events
if (event.type !== 'turn/end') return
// Call measure to get the current Token measurement for this Session
const measurement = ctx.tokenMeter.measure(session)
console.log(`Turn ${event.data.turn} ended:`)
console.log(` Surface tokens: ${measurement.surfaceTokens}`)
console.log(` Total tokens: ${measurement.totalTokens}`)
// Show delta — positive means context is growing
const delta = measurement.surfaceDeltaTokens
const sign = delta > 0 ? '+' : ''
console.log(` Delta: ${sign}${delta}`)
// Warn if context is growing too fast
if (delta > 2000) {
console.warn(' ⚠ Context growing fast, consider compression')
}
})
The Telemetry Seam: ctx.sessionTelemetry
session/event listeners are great for development, but in production you need to ship data to external systems — Grafana, Datadog, CloudWatch, and so on.
dsh provides a 'Telemetry Seam' for this: ctx.sessionTelemetry.
The word 'seam' is deliberate — it's a standardized interface that lets you plug in any telemetry backend, while the harness itself stays independent of any particular monitoring system.
Structure of each telemetry record:
// SessionTelemetryRecord (from packages/session/session-telemetry/src)
interface SessionTelemetryRecord {
// Two channels:
// 'ledger': full mirror of Session log events, one-to-one mapping
// 'ops': operational signals, only emitted for special situations
channel: 'ledger' | 'ops'
// Timestamp in milliseconds
time: number
// Severity level
severity: 'info' | 'warn' | 'error'
// Identifying attributes (for querying and filtering)
// e.g. session.id, event.type, event.seq
attributes: Record<string, string | number>
// Full payload: a deep copy of event.data
body: unknown
}
What each channel does
ledger channel: A complete mirror of the Session log. Every Session event produces one corresponding ledger record. This is your source of truth for audit and replay.
Includes:
- Every
assistant/message(with full streaming data) - Every
tool/callandtool/result - Failed
assistant/attemptevents (things the model generated but didn't commit) - All lifecycle events:
turn/start,turn/end, etc.
ops channel: Operational signals, just two kinds:
-
agent-error: The Agent failed outside of a Turn (e.g. initialization error) -
shutdown: The Agent shut down cleanly
How severity is determined
-
error: tool results withisError: true,turn/endwith an error reason,agent-errorops events - Everything else:
info
This mapping means you can filter severity === 'error' in your monitoring system to see all failures instantly, without writing custom classification logic.
OpenTelemetry Integration
dsh provides an official OTel Provider plugin: dsh-session-telemetry-otel.
How to wire it in (conceptual):
// Add this plugin to your Bundle configuration (pseudocode)
// This connects ctx.sessionTelemetry to the OTel backend
'@deepseek-ai/dsh-session-telemetry-otel'
// Internally, the plugin will:
// 1. Register an OTel backend implementation for ctx.sessionTelemetry
// 2. Send each SessionTelemetryRecord via the OTel JS SDK's Logger API
// 3. Support configurable Exporters (OTLP, Console, File, etc.)
A few design principles worth knowing:
Boundary axiom: The harness only calls emit(). Batching, retries, and queuing are the OTel SDK's responsibility — the harness doesn't touch them. Both sides can evolve independently.
Best-effort delivery: Telemetry records may be duplicated or lost. Consumers should deduplicate ledger records using the (session.id, format_version, event.seq) tuple, rather than assuming exactly-once delivery.
flush is optional: You can call flush() at the end of each Turn, but the OTel backend doesn't implement it by default (to avoid concurrency conflicts). If you need strong consistency, configure it yourself.
Hands-On: A Simple Debug Observer Plugin
Let's combine everything above into a complete debug observability plugin:
// debug-observer.ts — observability plugin for development use
// Usage: add to your Bundle during development;
// in production, swap in dsh-session-telemetry-otel instead
export const name = 'debug-observer'
// Declare dependency injection for tokenMeter
export const inject = ['tokenMeter']
export function apply(ctx: Context): void {
// ── 1. Watch tool calls ────────────────────────────────────
ctx.on('session/event', (session, event) => {
if (event.type !== 'tool/call') return
console.log(`[Tool Call] ${event.data.name}`)
console.log(` Call ID: ${event.data.callId}`)
// arguments is a raw JSON string — straight from the model, not yet parsed
console.log(` Args: ${event.data.arguments}`)
})
// ── 2. Watch tool results ──────────────────────────────────
ctx.on('session/event', (session, event) => {
if (event.type !== 'tool/result') return
const blocks = event.data.message.content
const isError = blocks.some(b => b.type === 'tool_result' && b.isError)
const icon = isError ? '✗' : '✓'
// toolUseId links this result back to its matching tool/call event
const toolUseId = blocks[0]?.toolUseId ?? 'unknown'
console.log(`[Tool Result] ${icon} (call: ${toolUseId})`)
})
// ── 3. Print token summary at each Turn end ────────────────
ctx.on('session/event', (session, event) => {
if (event.type !== 'turn/end') return
// reason.kind can be 'complete', 'error', 'interrupted', etc.
const reason = event.data.reason.kind
const measurement = ctx.tokenMeter.measure(session)
console.log(`\n[Turn ${event.data.turn}] ended: ${reason}`)
console.log(` Surface: ${measurement.surfaceTokens} tokens`)
console.log(` Total: ${measurement.totalTokens} tokens`)
const delta = measurement.surfaceDeltaTokens
const sign = delta > 0 ? '+' : ''
console.log(` Delta: ${sign}${delta}`)
// If the Turn ended with an error, print the full reason object
if (reason === 'error') {
console.error(` Error: ${JSON.stringify(event.data.reason)}`)
}
})
// ── 4. Watch Session lifecycle ─────────────────────────────
ctx.on('session/created', (session) => {
console.log(`\n[Session] created: ${session.id}`)
})
ctx.on('session/disposed', (session) => {
console.log(`[Session] disposed: ${session.id}`)
})
}
This plugin drops into your Bundle during development and gives you a complete trace of everything the Agent does. In production, swap it for the OTel plugin to route that same data into your monitoring stack.
Debugging Tip: Reading JSONL Log Files
dsh persists Session logs as JSONL files — one JSON object per line, each representing one SessionEvent.
Some useful command-line analysis recipes:
# See all tool calls (extract just the tool names)
cat session.jsonl | grep '"type":"tool/call"' | jq '.data.name'
# See failed assistant attempts (model output that wasn't committed)
cat session.jsonl | grep '"type":"assistant/attempt"' | jq '.'
# Summarize token usage per turn (from assistant/message usage field)
cat session.jsonl | grep '"type":"assistant/message"' | jq '.data.usage'
# See how each Turn ended (complete vs. error)
cat session.jsonl | grep '"type":"turn/end"' | jq '.data.reason.kind'
# Find any failed tool executions
cat session.jsonl | grep '"type":"tool/result"' | jq 'select(.data.message.content[].isError == true)'
These assume you have jq installed. On Windows, PowerShell's ConvertFrom-Json provides equivalent functionality.
The Four Layers of Observability
Real-time observation (development)
└─ session/event listener → each event printed to console immediately
Audit and replay (post-mortem)
└─ JSONL log file → full execution chain reconstruction, analyzable with jq
Token usage analysis
└─ ctx.tokenMeter.measure(session) → per-node token counts across the surface
→ pinpoint which messages are burning your budget
Production monitoring (systems level)
└─ ctx.sessionTelemetry + OTel plugin → ship to Grafana / Datadog / CloudWatch
→ alerts, dashboards, and error tracking all wired up
Wrapping Up
Observability isn't a nice-to-have for Agents — it's the only way to debug them.
dsh is designed with this in mind: Sessions are event logs, and event logs are audit data by definition. ctx.tokenMeter makes token consumption transparent. ctx.sessionTelemetry provides a standardized seam so you can choose any backend.
The core pattern is simple: session/event listeners → JSONL persistence → Telemetry Seam → OTel backend. Which layer you use depends on your context, but all of them can run simultaneously without interfering with each other.
Next up is the final article in the series. We'll bring everything together — tool registration, Session management, error handling, and observability — and build a complete, production-quality plugin from scratch.
Check out PrimeSkills — a curated marketplace of AI agents and skills validated in real-world, enterprise-grade workflows. Not demos — things that actually work in production.
Find more on my Homepage
Top comments (0)