Introduction
Agentic systems can become difficult to debug once a single user request turns into multiple model calls, agent steps, and tool executions.
A final response tells us what the agent produced, but it doesn't necessarily tell us how it got there.
For developers building and debugging agent workflows, useful questions include:
How many agent steps were executed?
How many LLM requests were made?
How many requests failed?
Which tools were called?
Which tool took the longest?
How long did the session run?
What happens to the collected state when the session ends?
This is where observability becomes important.
In this article, I'll walk through how I built dsh-plugin-agent-insights, a community plugin for DeepSeek Harness that provides session-scoped observability without modifying the Harness core packages.
What Is dsh-plugin-agent-insights?
dsh-plugin-agent-insights is a session-scoped observability plugin for DeepSeek Harness.
It tracks:
Area Metrics
Agent Steps, LLM requests, failed LLM requests
Tools Total, successful, failed calls
Performance Execution duration, slowest tool
Session Start time, total duration
Lifecycle Disposal and cleanup
The plugin is implemented as a Cordis service and integrates with Harness lifecycle events.
Source:
dsh-plugin-agent-insights
Session-scoped observability metrics and performance tracking plugin for the DeepSeek Harness ecosystem.
Overview
dsh-plugin-agent-insights provides lightweight, non-invasive, session-scoped observability and performance tracking for DeepSeek Harness agents. It registers as a Cordis service (ctx.agentInsights) and automatically hooks into runtime dispatch pipelines to collect granular metrics without modifying any core packages or interfering with agent execution.
Key Capabilities
-
Tool Execution Metrics: Captures total tool calls, successful calls, failed calls, precise execution duration via
performance.now(), and identifies the slowest tool invocation. - LLM & Agent Metrics: Tracks total agent steps closed, total model requests dispatched, and failed model requests.
- Session Duration Metrics: Records session start timestamps and total elapsed session duration.
- Global Slowest Tool Tracking: Tracks the single slowest tool invocation across all active sessions without retaining references to the sessions.
-
Zero-Leak Memory Architecture: Stores session statistics in a
WeakMap<Session, SessionAgentInsights>, preventing memory retention…
Why Build It as a Plugin?
One reason to use a plugin architecture is to keep specialized functionality separate from core execution logic.
Observability is a good example.
Different applications may require:
basic metrics
performance profiling
tracing
token accounting
dashboards
custom telemetry
Rather than making all of these features part of the core framework, a plugin can observe the framework's lifecycle and maintain its own state.
DeepSeek Harness
│
├── Agent execution
├── Tool execution
├── Sessions
└── Lifecycle events
│
▼
Agent Insights
│
┌───────┼────────┐
▼ ▼ ▼
Agent Tools Sessions
Metrics Metrics Metrics
Understanding the Harness Lifecycle
The plugin integrates with several lifecycle points:
agent/request
Used to observe LLM request attempts.
agent/request-error
Used to record failed model requests.
tools/execute
Used to measure tool execution duration and outcome.
session/event
Used to observe session events and count completed steps.
session/disposed
Used to produce a final session summary and clean up session state.
This separation allows the plugin to observe different parts of the agent lifecycle without owning those operations.
Measuring Tool Execution
One of the most useful metrics is tool latency.
The plugin wraps the tool execution around next():
const startTime = performance.now()
try {
result = await next()
return result
} finally {
const durationMs = performance.now() - startTime
// Record execution metric
}
For each tool call, the plugin records:
{
callId,
name,
durationMs,
isError,
errorMessage
}
This makes it possible to identify slow tools and distinguish successful executions from failures.
Tracking Agent Requests
Every LLM request associated with a session increments the session's request counter.
stats.totalLlmRequests += 1
Failed requests are tracked independently:
stats.failedLlmRequests += 1
This provides a simple distinction between:
Total requests
│
├── Successful
└── Failed
Session-Level Metrics
The plugin aggregates all of this information at the session level.
A session snapshot can contain:
{
sessionId,
sessionStartTime,
sessionDurationMs,
totalSteps,
totalLlmRequests,
failedLlmRequests,
totalToolCalls,
successfulToolCalls,
failedToolCalls,
toolExecutions,
slowestTool
}
This gives developers a compact view of an entire agent session.
Managing Session State
Session-specific metrics are stored using:
WeakMap<Session, SessionAgentInsights>
This allows each session to have independent metrics without maintaining a permanent strong reference to every session.
Conceptually:
Session A → Metrics A
Session B → Metrics B
Session C → Metrics C
Cleaning Up on Session Disposal
Collecting metrics isn't enough.
Long-running agent processes can create many sessions, so lifecycle cleanup is important.
When:
session/disposed
fires, the plugin:
retrieves the session metrics,
calculates the final duration,
optionally logs a summary,
removes the session entry.
For example:
Session completed
8 steps
5 LLM requests
1 failed request
12 tool calls
2 failed tools
14.32s duration
Slowest tool: database-query
Observability Should Not Break the Agent
A key design principle was:
The observability layer should never become another failure point.
If recording a metric fails, the original agent operation should continue normally.
For this reason, internal metric collection is isolated with error handling.
try {
// Record metric
} catch (err) {
ctx.logger.warn(
`[agent-insights] Failed to record metric: ${String(err)}`
)
}
The plugin observes execution rather than becoming responsible for execution.
Configuration
The plugin supports configuration through the Harness plugin system:
- insert:
- id: agent-insights
name: dsh-plugin-agent-insights
config:
maxToolHistoryPerSession: 1000
logSummaryOnDisposed: true
maxToolHistoryPerSession
Controls the number of individual tool execution records retained per session.
logSummaryOnDisposed
Controls whether a summary is logged when the session ends.
Programmatic Usage
The plugin can also be mounted programmatically:
import { Context } from '@deepseek-ai/cordis'
import AgentInsights from 'dsh-plugin-agent-insights'
const ctx = new Context()
await ctx.plugin(AgentInsights, {
maxToolHistoryPerSession: 500,
logSummaryOnDisposed: true,
})
Metrics can then be accessed through:
const metrics = ctx.agentInsights.getMetrics(session)
The plugin exposes three primary APIs:
getMetrics(session)
getSlowestTool(session?)
reset(session?)
Installation
The plugin can be added to a profile using the Harness CLI:
dsh plugin --profile <name> add dsh-plugin-agent-insights
Its cordis.patch.yml provides the corresponding profile integration.
Testing
The plugin includes its own build and testing workflow:
pnpm run build
pnpm run typecheck
pnpm run test
pnpm run prepublishOnly
Keeping the plugin independently testable makes it easier to develop without coupling its validation process to unrelated Harness functionality.
What Could Be Built on Top of This?
These metrics could serve as the foundation for more advanced observability features:
performance dashboards
tool latency analysis
agent failure analysis
session comparisons
regression detection
custom telemetry integrations
distributed tracing
The current plugin is intentionally focused on the fundamentals: collecting useful information while staying out of the agent's execution path.
Lessons Learned
- Lifecycle hooks are powerful extension points
They allow functionality to be added without modifying core packages.
- Instrumentation should observe, not control
The plugin should never become responsible for the operation it is measuring.
- Cleanup is part of observability
Metrics need a lifecycle just like the sessions they describe.
- Telemetry must fail gracefully
A metrics failure should never become an agent failure.
Conclusion
dsh-plugin-agent-insights demonstrates one practical way to extend DeepSeek Harness through its plugin and lifecycle architecture.
It provides visibility into:
agent steps
LLM requests
LLM failures
tool executions
tool failures
tool latency
slowest tools
session duration
session lifecycle
The larger goal is to show how specialized capabilities can be developed around the Harness ecosystem while keeping the core execution path clean.
The plugin is available as a community project for experimentation, feedback, and further development.
Plugin:
dsh-plugin-agent-insights
Session-scoped observability metrics and performance tracking plugin for the DeepSeek Harness ecosystem.
Overview
dsh-plugin-agent-insights provides lightweight, non-invasive, session-scoped observability and performance tracking for DeepSeek Harness agents. It registers as a Cordis service (ctx.agentInsights) and automatically hooks into runtime dispatch pipelines to collect granular metrics without modifying any core packages or interfering with agent execution.
Key Capabilities
-
Tool Execution Metrics: Captures total tool calls, successful calls, failed calls, precise execution duration via
performance.now(), and identifies the slowest tool invocation. - LLM & Agent Metrics: Tracks total agent steps closed, total model requests dispatched, and failed model requests.
- Session Duration Metrics: Records session start timestamps and total elapsed session duration.
- Global Slowest Tool Tracking: Tracks the single slowest tool invocation across all active sessions without retaining references to the sessions.
-
Zero-Leak Memory Architecture: Stores session statistics in a
WeakMap<Session, SessionAgentInsights>, preventing memory retention…
Top comments (0)