Building Session-Level Observability for DeepSeek Harness with a Community Plugin

Omansh Bhatnagar — Mon, 14 Sep 2026 20:19:40 +0000

Introduction

Agentic systems can become difficult to debug once a single user request turns into multiple model calls, agent steps, and tool executions.

A final response tells us what the agent produced, but it doesn't necessarily tell us how it got there.

For developers building and debugging agent workflows, useful questions include:

How many agent steps were executed?
How many LLM requests were made?
How many requests failed?
Which tools were called?
Which tool took the longest?
How long did the session run?
What happens to the collected state when the session ends?

This is where observability becomes important.

In this article, I'll walk through how I built dsh-plugin-agent-insights, a community plugin for DeepSeek Harness that provides session-scoped observability without modifying the Harness core packages.

What Is dsh-plugin-agent-insights?

dsh-plugin-agent-insights is a session-scoped observability plugin for DeepSeek Harness.

It tracks:

Area Metrics
Agent Steps, LLM requests, failed LLM requests
Tools Total, successful, failed calls
Performance Execution duration, slowest tool
Session Start time, total duration
Lifecycle Disposal and cleanup

The plugin is implemented as a Cordis service and integrates with Harness lifecycle events.

Source:

Claven07 / dsh-plugin-agent-insights

dsh-plugin-agent-insights

Session-scoped observability metrics and performance tracking plugin for the DeepSeek Harness ecosystem.

Overview

dsh-plugin-agent-insights provides lightweight, non-invasive, session-scoped observability and performance tracking for DeepSeek Harness agents. It registers as a Cordis service (ctx.agentInsights) and automatically hooks into runtime dispatch pipelines to collect granular metrics without modifying any core packages or interfering with agent execution.

Key Capabilities

Tool Execution Metrics: Captures total tool calls, successful calls, failed calls, precise execution duration via performance.now(), and identifies the slowest tool invocation.
LLM & Agent Metrics: Tracks total agent steps closed, total model requests dispatched, and failed model requests.
Session Duration Metrics: Records session start timestamps and total elapsed session duration.
Global Slowest Tool Tracking: Tracks the single slowest tool invocation across all active sessions without retaining references to the sessions.
Zero-Leak Memory Architecture: Stores session statistics in a WeakMap<Session, SessionAgentInsights>, preventing memory retention…

View on GitHub

Why Build It as a Plugin?

One reason to use a plugin architecture is to keep specialized functionality separate from core execution logic.

Observability is a good example.

Different applications may require:

basic metrics
performance profiling
tracing
token accounting
dashboards
custom telemetry

Rather than making all of these features part of the core framework, a plugin can observe the framework's lifecycle and maintain its own state.

DeepSeek Harness
│
├── Agent execution
├── Tool execution
├── Sessions
└── Lifecycle events
│
▼
Agent Insights
│
┌───────┼────────┐
▼ ▼ ▼
Agent Tools Sessions
Metrics Metrics Metrics

Understanding the Harness Lifecycle

The plugin integrates with several lifecycle points:

agent/request

Used to observe LLM request attempts.

agent/request-error

Used to record failed model requests.

tools/execute

Used to measure tool execution duration and outcome.

session/event

Used to observe session events and count completed steps.

session/disposed

Used to produce a final session summary and clean up session state.

This separation allows the plugin to observe different parts of the agent lifecycle without owning those operations.

Measuring Tool Execution

One of the most useful metrics is tool latency.

The plugin wraps the tool execution around next():

const startTime = performance.now()

try {
  result = await next()
  return result
} finally {
  const durationMs = performance.now() - startTime

  // Record execution metric
}

For each tool call, the plugin records:

{
  callId,
  name,
  durationMs,
  isError,
  errorMessage
}

This makes it possible to identify slow tools and distinguish successful executions from failures.

Tracking Agent Requests

Every LLM request associated with a session increments the session's request counter.

stats.totalLlmRequests += 1

Failed requests are tracked independently:

stats.failedLlmRequests += 1

This provides a simple distinction between:

Total requests
│
├── Successful
└── Failed

Session-Level Metrics

The plugin aggregates all of this information at the session level.

A session snapshot can contain:

{
  sessionId,
  sessionStartTime,
  sessionDurationMs,
  totalSteps,
  totalLlmRequests,
  failedLlmRequests,
  totalToolCalls,
  successfulToolCalls,
  failedToolCalls,
  toolExecutions,
  slowestTool
}

This gives developers a compact view of an entire agent session.

Managing Session State

Session-specific metrics are stored using:

WeakMap<Session, SessionAgentInsights>

This allows each session to have independent metrics without maintaining a permanent strong reference to every session.

Conceptually:

Session A → Metrics A
Session B → Metrics B
Session C → Metrics C

Cleaning Up on Session Disposal

Collecting metrics isn't enough.

Long-running agent processes can create many sessions, so lifecycle cleanup is important.

When:

session/disposed

fires, the plugin:

retrieves the session metrics,
calculates the final duration,
optionally logs a summary,
removes the session entry.

For example:

Session completed

8 steps
5 LLM requests
1 failed request
12 tool calls
2 failed tools
14.32s duration
Slowest tool: database-query

Observability Should Not Break the Agent

A key design principle was:

The observability layer should never become another failure point.

If recording a metric fails, the original agent operation should continue normally.

For this reason, internal metric collection is isolated with error handling.

try {
  // Record metric
} catch (err) {
  ctx.logger.warn(
    `[agent-insights] Failed to record metric: ${String(err)}`
  )
}

The plugin observes execution rather than becoming responsible for execution.

Configuration

The plugin supports configuration through the Harness plugin system:

- insert:
    - id: agent-insights
      name: dsh-plugin-agent-insights
      config:
        maxToolHistoryPerSession: 1000
        logSummaryOnDisposed: true

maxToolHistoryPerSession

Controls the number of individual tool execution records retained per session.

logSummaryOnDisposed

Controls whether a summary is logged when the session ends.

Programmatic Usage

The plugin can also be mounted programmatically:

import { Context } from '@deepseek-ai/cordis'
import AgentInsights from 'dsh-plugin-agent-insights'

const ctx = new Context()

await ctx.plugin(AgentInsights, {
  maxToolHistoryPerSession: 500,
  logSummaryOnDisposed: true,
})

Metrics can then be accessed through:

const metrics = ctx.agentInsights.getMetrics(session)

The plugin exposes three primary APIs:

getMetrics(session)
getSlowestTool(session?)
reset(session?)

Installation

The plugin can be added to a profile using the Harness CLI:

dsh plugin --profile <name> add dsh-plugin-agent-insights

Its cordis.patch.yml provides the corresponding profile integration.

Testing

The plugin includes its own build and testing workflow:

pnpm run build
pnpm run typecheck
pnpm run test
pnpm run prepublishOnly

Keeping the plugin independently testable makes it easier to develop without coupling its validation process to unrelated Harness functionality.

What Could Be Built on Top of This?

These metrics could serve as the foundation for more advanced observability features:

performance dashboards
tool latency analysis
agent failure analysis
session comparisons
regression detection
custom telemetry integrations
distributed tracing

The current plugin is intentionally focused on the fundamentals: collecting useful information while staying out of the agent's execution path.

Lessons Learned

Lifecycle hooks are powerful extension points

They allow functionality to be added without modifying core packages.

Instrumentation should observe, not control

The plugin should never become responsible for the operation it is measuring.

Cleanup is part of observability

Metrics need a lifecycle just like the sessions they describe.

Telemetry must fail gracefully

A metrics failure should never become an agent failure.

Conclusion

dsh-plugin-agent-insights demonstrates one practical way to extend DeepSeek Harness through its plugin and lifecycle architecture.

It provides visibility into:

agent steps
LLM requests
LLM failures
tool executions
tool failures
tool latency
slowest tools
session duration
session lifecycle

The larger goal is to show how specialized capabilities can be developed around the Harness ecosystem while keeping the core execution path clean.

The plugin is available as a community project for experimentation, feedback, and further development.

Plugin:

Claven07 / dsh-plugin-agent-insights

dsh-plugin-agent-insights

Session-scoped observability metrics and performance tracking plugin for the DeepSeek Harness ecosystem.

Overview

Key Capabilities

Tool Execution Metrics: Captures total tool calls, successful calls, failed calls, precise execution duration via performance.now(), and identifies the slowest tool invocation.
LLM & Agent Metrics: Tracks total agent steps closed, total model requests dispatched, and failed model requests.
Session Duration Metrics: Records session start timestamps and total elapsed session duration.
Global Slowest Tool Tracking: Tracks the single slowest tool invocation across all active sessions without retaining references to the sessions.
Zero-Leak Memory Architecture: Stores session statistics in a WeakMap<Session, SessionAgentInsights>, preventing memory retention…

View on GitHub

DEV Community: Omansh Bhatnagar

Building Session-Level Observability for DeepSeek Harness with a Community Plugin

Claven07 / dsh-plugin-agent-insights

dsh-plugin-agent-insights

Overview

Key Capabilities

Claven07 / dsh-plugin-agent-insights

dsh-plugin-agent-insights

Overview

Key Capabilities