DEV Community

kirandeepjassal-crypto
kirandeepjassal-crypto

Posted on • Originally published at prepstack.co.in

MCP Deep Dive, Part 10: When the Agent Feels Off — Debugging and Observability for MCP in Production

#mcp #observability #ai #dotnet

A web API fails loudly: a 500, a stack trace, an alert. An agent fails softly. It doesn't crash — it quietly takes six turns instead of two, calls the wrong tool, spends triple the tokens, and returns an answer that's subtly wrong. None of that throws an exception, and none of it shows up in the logs of any single request.

This is Part 10 of a 15-part deep dive on Model Context Protocol (MCP). Part 3 gave each tool call a span on the server; this part joins those into the whole picture — one agent run across the host, the model, and all three servers, correlated so that when something feels off you have an answer instead of a shrug.

One agent run, as a trace 

agent.run  (tenant, goal)                                       [==================] 1.8s
|
+- agent.turn 0                                                 [======]
|    +- agent.model_call  (plan)                                [===]
|    +- tool.get_campaign_kpis  -> mattrx-analytics             [==]   120ms
|    +- tool.query_events       -> mattrx-analytics             [===]  180ms
|
+- agent.turn 1                                                 [=====]
|    +- agent.model_call  (synthesize)                          [====]
|    +- tool.create_report      -> mattrx-reports (enqueue)     [=]    90ms
|
+- eval gate 0.93 (pass) -> final answer
Enter fullscreen mode Exit fullscreen mode

1. Correlate the whole run with one trace id

Start one root span per agent run and propagate its context across the MCP transport, so every server's spans nest under it.

public async Task<AgentAnswer> RunAsync(AiPrincipal p, string goal, CancellationToken ct)
{
    using var run = ActivitySource.StartActivity("agent.run");   // the root span for the whole run
    run?.SetTag("mattrx.tenant", p.TenantId);
    run?.SetTag("agent.goal", Redact(goal));

    // OTel context propagates over the MCP transport (traceparent) -> every server span nests here.
    return await LoopAsync(p, goal, ct);
}
Enter fullscreen mode Exit fullscreen mode

The unit of observability for an agent is the run, not the request. One trace id, and the whole cross-server run is in front of you — which is why incident triage dropped from hours to minutes.

2. Trace the loop, not just the tool

A span per turn, per model call, and per tool call, nested under the run.

for (var turn = 0; turn < MaxTurns; turn++)
{
    using var turnSpan = ActivitySource.StartActivity("agent.turn");
    turnSpan?.SetTag("agent.turn", turn);

    using (ActivitySource.StartActivity("agent.model_call"))
        reply = await model.ChatAsync(messages, tools, ct);           // one span per model call

    foreach (var call in reply.ToolCalls)
        using (var t = ActivitySource.StartActivity($"tool.{call.Name}"))
        {
            t?.SetTag("mcp.server", Route(call.Name));
            results.Add(await manager.InvokeAsync(call, ct));
        }
}
Enter fullscreen mode Exit fullscreen mode

The tool span answers "was the tool slow?"; the run->turn->call tree answers "why did the agent take six turns?"

3. Structured, redacted protocol logging

Structured logs of the MCP interaction — method, tool, server, tenant, latency, outcome — correlated by the trace id and redacted.

logger.ToolCall(new
{
    TraceId    = Activity.Current?.TraceId.ToString(),
    Tool       = call.Name, Server = Route(call.Name),
    Tenant     = principal.TenantId,
    DurationMs = sw.ElapsedMilliseconds,
    Outcome    = result.IsError ? "error" : "ok",
    // arguments/results: a redacted projection or a hash — never the raw content
});
Enter fullscreen mode Exit fullscreen mode

Traces give order and latency; logs give detail. Correlate them by trace id — and redact the payloads so your observability stack doesn't become your largest unsecured copy of customer data.

4. The metrics that matter for agents

MCP-specific metrics, dimensioned by tool and tenant: latency and error rate per tool, turns per run, tokens and cost per run.

meters.ToolLatency.Record(sw.ElapsedMilliseconds,
    [KeyValuePair.Create("tool", call.Name), KeyValuePair.Create("tenant", principal.TenantId)]);
meters.ToolErrors.Add(result.IsError ? 1 : 0, /* same tags */);
meters.TurnsPerRun.Record(turnCount);
meters.TokensPerRun.Record(usage.TotalTokens);
Enter fullscreen mode Exit fullscreen mode

"Requests per second" tells you nothing about whether your agent is healthy. Per-tool per-tenant latency turns "the assistant is slow" into "query_events on tenant Y regressed at 14:00." Turns-per-run catches thrashing; tokens/cost-per-run catches the agent that quietly got expensive.

5. The audit log is your debugging record

The append-only audit log you already built for security reconstructs exactly what the agent saw and did.

var reconstruction = await audit.ReconstructAsync(traceId, ct);
// -> every tool call, result hash, guardrail decision, and outcome, in sequence
Enter fullscreen mode Exit fullscreen mode

Agents are non-deterministic, so "just reproduce it" usually fails. The audit trail is your reconstruction — an investigation becomes a query, not an archaeology dig.

6. Local debugging: the MCP Inspector and session replay

Poke the server directly, and replay a recorded session to reproduce a bug deterministically.

# Poke a server directly — list tools, call one, see the raw JSON-RPC request/response.
npx @modelcontextprotocol/inspector node ./mattrx-analytics-server
Enter fullscreen mode Exit fullscreen mode 
// Replay a recorded session against a server to reproduce a bug without the model in the loop.
await replayer.RunAsync("session-4821.jsonl", targetServer: "mattrx-analytics", ct);
Enter fullscreen mode Exit fullscreen mode

The Inspector is the fastest way to answer "is it the server or the agent?" — call the tool by hand, no model involved. Replaying a recorded session turns a bug that showed up once into a repeatable test.

What to check when the agent "feels off" 

Symptom                         Look at...
slow                     ->  the RUN trace: which turn/tool span is fat?
wrong answer             ->  the AUDIT log: what did it retrieve + call?
too many turns / cost    ->  turns-per-run + tokens-per-run metrics
intermittent failures    ->  per-tool per-tenant error rate + retries
"did the server change?" ->  the MCP Inspector: call the tool by hand
can't reproduce          ->  REPLAY the recorded session against the server
Enter fullscreen mode Exit fullscreen mode

The model to carry forward

Agents fail softly, so you have to watch softly too. The exception-and-alert model built for web APIs misses the slow drift, the extra turns, the subtly worse answer. Observe the whole run as one trace, keep an audit record you can reconstruct from, dimension your metrics by tool and tenant, and watch quality alongside latency. Then "the agent feels off" has an answer.

Originally published at prepstack.co.in. Part 11 zooms out: rolling MCP across the enterprise.

Top comments (3)

Collapse
 
mads_hansen_27b33ebfee4c9 profile image
Mads Hansen •

Treating the run as the observability unit is exactly right. Two production details matter here. First, tenant is often too high-cardinality for a metric label; it can explode cost and accidentally expose customer identity. Keep low-cardinality dimensions in metrics, put a pseudonymous tenant key in controlled logs/traces, and link outliers with exemplars. Raw goals, arguments, and results should be denied by default rather than relying on a best-effort Redact() call.

Second, a session replay needs a manifest, not only JSON-RPC events: model/provider version, system-prompt digest, tool-schema/version digest, policy version, routing config, and captured external-result hashes. Replay the tool layer with recorded responses first, then separately test the current live dependencies. Otherwise a “failed replay” cannot distinguish agent drift from changed tools or data. I’d also sign/hash-chain the audit sequence so reconstruction evidence is tamper-evident, not merely append-only by convention.

Collapse
 
alexshev profile image
Alex Shev •

Observability for MCP gets interesting because the failure is often not a crash. The agent “feels off” because a tool returned stale context, a permission boundary was too broad, or a retry changed the state. Traces need to show intent, inputs, tool output, and what the agent believed afterward.

Collapse
 
raju_dandigam profile image
Raju Dandigam •

The point about treating the run, not the request, as the unit of observability is exactly right. If you only instrument tool latency, you miss the agent behaviors that matter most in production: thrashing, duplicate turns, tool oscillation, and retries that look like progress.

That is a big part of why we built agent-inspect around local-first execution traces and tool-call receipts for TypeScript agent workflows. Curious whether your audit log and trace model share the same run id end to end, or whether you join them later during incident triage?