A web API fails loudly: a 500, a stack trace, an alert. An agent fails softly. It doesn't crash — it quietly takes six turns instead of two, calls the wrong tool, spends triple the tokens, and returns an answer that's subtly wrong. None of that throws an exception, and none of it shows up in the logs of any single request.
This is Part 10 of a 15-part deep dive on Model Context Protocol (MCP). Part 3 gave each tool call a span on the server; this part joins those into the whole picture — one agent run across the host, the model, and all three servers, correlated so that when something feels off you have an answer instead of a shrug.
One agent run, as a trace
agent.run (tenant, goal) [==================] 1.8s
|
+- agent.turn 0 [======]
| +- agent.model_call (plan) [===]
| +- tool.get_campaign_kpis -> mattrx-analytics [==] 120ms
| +- tool.query_events -> mattrx-analytics [===] 180ms
|
+- agent.turn 1 [=====]
| +- agent.model_call (synthesize) [====]
| +- tool.create_report -> mattrx-reports (enqueue) [=] 90ms
|
+- eval gate 0.93 (pass) -> final answer
1. Correlate the whole run with one trace id
Start one root span per agent run and propagate its context across the MCP transport, so every server's spans nest under it.
public async Task<AgentAnswer> RunAsync(AiPrincipal p, string goal, CancellationToken ct)
{
using var run = ActivitySource.StartActivity("agent.run"); // the root span for the whole run
run?.SetTag("mattrx.tenant", p.TenantId);
run?.SetTag("agent.goal", Redact(goal));
// OTel context propagates over the MCP transport (traceparent) -> every server span nests here.
return await LoopAsync(p, goal, ct);
}
The unit of observability for an agent is the run, not the request. One trace id, and the whole cross-server run is in front of you — which is why incident triage dropped from hours to minutes.
2. Trace the loop, not just the tool
A span per turn, per model call, and per tool call, nested under the run.
for (var turn = 0; turn < MaxTurns; turn++)
{
using var turnSpan = ActivitySource.StartActivity("agent.turn");
turnSpan?.SetTag("agent.turn", turn);
using (ActivitySource.StartActivity("agent.model_call"))
reply = await model.ChatAsync(messages, tools, ct); // one span per model call
foreach (var call in reply.ToolCalls)
using (var t = ActivitySource.StartActivity($"tool.{call.Name}"))
{
t?.SetTag("mcp.server", Route(call.Name));
results.Add(await manager.InvokeAsync(call, ct));
}
}
The tool span answers "was the tool slow?"; the run->turn->call tree answers "why did the agent take six turns?"
3. Structured, redacted protocol logging
Structured logs of the MCP interaction — method, tool, server, tenant, latency, outcome — correlated by the trace id and redacted.
logger.ToolCall(new
{
TraceId = Activity.Current?.TraceId.ToString(),
Tool = call.Name, Server = Route(call.Name),
Tenant = principal.TenantId,
DurationMs = sw.ElapsedMilliseconds,
Outcome = result.IsError ? "error" : "ok",
// arguments/results: a redacted projection or a hash — never the raw content
});
Traces give order and latency; logs give detail. Correlate them by trace id — and redact the payloads so your observability stack doesn't become your largest unsecured copy of customer data.
4. The metrics that matter for agents
MCP-specific metrics, dimensioned by tool and tenant: latency and error rate per tool, turns per run, tokens and cost per run.
meters.ToolLatency.Record(sw.ElapsedMilliseconds,
[KeyValuePair.Create("tool", call.Name), KeyValuePair.Create("tenant", principal.TenantId)]);
meters.ToolErrors.Add(result.IsError ? 1 : 0, /* same tags */);
meters.TurnsPerRun.Record(turnCount);
meters.TokensPerRun.Record(usage.TotalTokens);
"Requests per second" tells you nothing about whether your agent is healthy. Per-tool per-tenant latency turns "the assistant is slow" into "query_events on tenant Y regressed at 14:00." Turns-per-run catches thrashing; tokens/cost-per-run catches the agent that quietly got expensive.
5. The audit log is your debugging record
The append-only audit log you already built for security reconstructs exactly what the agent saw and did.
var reconstruction = await audit.ReconstructAsync(traceId, ct);
// -> every tool call, result hash, guardrail decision, and outcome, in sequence
Agents are non-deterministic, so "just reproduce it" usually fails. The audit trail is your reconstruction — an investigation becomes a query, not an archaeology dig.
6. Local debugging: the MCP Inspector and session replay
Poke the server directly, and replay a recorded session to reproduce a bug deterministically.
# Poke a server directly — list tools, call one, see the raw JSON-RPC request/response.
npx @modelcontextprotocol/inspector node ./mattrx-analytics-server
// Replay a recorded session against a server to reproduce a bug without the model in the loop.
await replayer.RunAsync("session-4821.jsonl", targetServer: "mattrx-analytics", ct);
The Inspector is the fastest way to answer "is it the server or the agent?" — call the tool by hand, no model involved. Replaying a recorded session turns a bug that showed up once into a repeatable test.
What to check when the agent "feels off"
Symptom Look at...
slow -> the RUN trace: which turn/tool span is fat?
wrong answer -> the AUDIT log: what did it retrieve + call?
too many turns / cost -> turns-per-run + tokens-per-run metrics
intermittent failures -> per-tool per-tenant error rate + retries
"did the server change?" -> the MCP Inspector: call the tool by hand
can't reproduce -> REPLAY the recorded session against the server
The model to carry forward
Agents fail softly, so you have to watch softly too. The exception-and-alert model built for web APIs misses the slow drift, the extra turns, the subtly worse answer. Observe the whole run as one trace, keep an audit record you can reconstruct from, dimension your metrics by tool and tenant, and watch quality alongside latency. Then "the agent feels off" has an answer.
Originally published at prepstack.co.in. Part 11 zooms out: rolling MCP across the enterprise.
Top comments (12)
Observability for MCP gets interesting because the failure is often not a crash. The agent “feels off” because a tool returned stale context, a permission boundary was too broad, or a retry changed the state. Traces need to show intent, inputs, tool output, and what the agent believed afterward.
Exactly — the crash is the easy case, because at least it announces itself. The failure that actually hurts is the one where every span is green and the answer is still wrong: a tool returned stale context, a boundary was a little too broad, a retry mutated state between attempts. Nothing "failed," so nothing pages you.
The line I'd underline is your last one — "what the agent believed afterward." That's the field most observability setups miss. We log intent, inputs, and tool output, and then stop right before the interesting part: what the model concluded from that output and carried into the next step. When an agent goes off, the divergence almost always lives there — the tool returned correct data and the model drew the wrong inference, or it silently down-weighted a result. Without capturing the post-tool belief state, you can see that the inputs were fine and the output was fine and still have no idea why the run went sideways.
The retry point is the other quiet one. A retry isn't idempotent at the belief level even when it's idempotent at the tool level — the model now has two observations of the "same" call, and if they differ at all, that difference becomes reasoning it acts on. Traces that only show the final attempt hide exactly the state change that explains the weird behavior.
So the unit that actually debugs these isn't request → response, it's intent → inputs → tool output → resulting belief, per step, across the whole run. Green spans tell you the plumbing held; only that last field tells you what the agent was actually thinking.
That "all spans green but the answer is wrong" case is the one that matters in local ranking work. A Maps agent can successfully fetch a grid, categories, reviews, and competitors, then still reason from yesterday's centroid or an old business name. I would log the evidence the agent believed, not just the tool calls it made.
That local-ranking example is the cleanest illustration of the whole problem I've seen. Every tool call succeeds — grid fetched, categories fetched, reviews fetched, competitors fetched, all green — and the agent still reasons from yesterday's centroid or a renamed business. The output is confidently wrong with a spotless trace behind it, and there's no span you can point at because no span failed.
"Log the evidence the agent believed, not just the tool calls it made" is exactly the fix, and your domain makes the why obvious: the tool returning fresh data and the agent actually using the fresh data are two separate events, and the gap between them is where the stale centroid lives. If you only capture the call, that gap is invisible — you can prove the data was current and still not explain the wrong ranking. Recording which centroid or business name the agent carried forward is what turns "fetched correctly but reasoned from stale state" into something you can see instead of infer.
The bit I'd add for the Maps case specifically: freshness has to be part of the believed-evidence record, not just the fetch. "Reviews fetched at T" and "agent ranked using reviews it treated as current" can diverge by a full refresh cycle, and that delta is the bug. Log the belief with its as-of timestamp and stale reasoning stops being a mystery.
That is exactly the failure class. The tool layer can be perfectly green while the model's working state is wrong. I would rather debug the evidence bundle the agent accepted than stare at a successful trace and pretend success means correctness.
Yes, "what the agent believed afterward" is the missing column. For MCP I would log it as a small derived state, not a giant transcript: tool selected, fact accepted, confidence/source, and whether the answer/action depended on it. Then an incident can ask whether the bad output came from the tool, the interpretation, or a stale belief that survived after the call.
The "small derived state, not a giant transcript" framing is what makes this actually shippable. Logging the full reasoning trace is what everyone reaches for and then quietly turns off — expensive, noisy, and it buries the one field that matters. Your four-field shape is basically the schema I'd want: tool selected, fact accepted, confidence/source, and — the load-bearing one — whether the answer or action depended on it.
That last field is what turns the log from interesting into triageable. A stale belief the model held but didn't use is noise; a stale belief the output depended on is the incident. Capturing the dependency link is what lets an investigation go straight to your three-way split — bad tool, bad interpretation, or a stale belief that survived past the call it came from. Three different bugs, three different fixes, and today most setups can't tell them apart because they logged the call and not the derived state.
The only thing I'd add is that "fact accepted" and "confidence/source" together give you a second signal for free: when the source was fresh and confidence was high but the output was still wrong, you've isolated it to interpretation; when confidence was low and the agent acted anyway, that's a guardrail gap, not a reasoning one. Going to use this as the concrete format for the belief field — it's the most practical version of it I've seen.
That load-bearing field is the whole difference. If the answer did not depend on the stale belief, it is noise. If it did, it becomes the shortest path to the bug. That is why I would rather log a small belief ledger than a giant trace nobody reads.
Yes. The belief-afterward field is where the debugging becomes useful. If the trace says every call worked but the agent carried forward the wrong conclusion, the trace is only half the story.
Treating the run as the observability unit is exactly right. Two production details matter here. First,
tenantis often too high-cardinality for a metric label; it can explode cost and accidentally expose customer identity. Keep low-cardinality dimensions in metrics, put a pseudonymous tenant key in controlled logs/traces, and link outliers with exemplars. Raw goals, arguments, and results should be denied by default rather than relying on a best-effortRedact()call.Second, a session replay needs a manifest, not only JSON-RPC events: model/provider version, system-prompt digest, tool-schema/version digest, policy version, routing config, and captured external-result hashes. Replay the tool layer with recorded responses first, then separately test the current live dependencies. Otherwise a “failed replay” cannot distinguish agent drift from changed tools or data. I’d also sign/hash-chain the audit sequence so reconstruction evidence is tamper-evident, not merely append-only by convention.
Both of these are corrections I'll take, and the second one changes how I'd build the replay entirely.
On tenant as a metric label — you're right, and I conflated two stores that should have different rules. Tenant is high-cardinality and identifying, so as a metric dimension it's the worst of both: it blows up your time-series cardinality (and bill), and it quietly turns a metrics backend into a place customer identity leaks. The split you're describing is the correct one: metrics carry only low-cardinality dimensions (tool, status, model, maybe tenant tier), and the tenant key lives as a pseudonymous id in access-controlled logs/traces, with exemplars linking a spiking metric to the specific traces behind it. You get the outlier's fingerprint without putting the identity in the metric. And the deeper point — deny raw goals/arguments/results by default rather than leaning on a best-effort Redact() — is the one I under-weighted. A redactor you have to remember to call is a redactor that eventually doesn't get called. Default-deny with an explicit allowlist of what's safe to capture is the only version that survives contact with a tired engineer at 2am.
On replay, the manifest is the part I got wrong. JSON-RPC events alone reconstruct what the agent did, but not the world it did it in — and without that, a failed replay is uninterpretable. Your manifest list is exactly the missing context: model/provider version, system-prompt digest, tool-schema/version digest, policy version, routing config, and hashes of the external results. And the two-phase idea is the key insight: replay the tool layer with recorded responses first (does the agent still behave the same against a frozen world?), then separately exercise the live dependencies (did the tools or data change underneath us?). Collapse those and you can't tell agent drift from a changed downstream — which is precisely the "the agent feels off" ambiguity the whole post was trying to resolve. I described one-trace-per-run as the unit but didn't give it enough context to be deterministically replayable; the manifest is what closes that.
And hash-chaining the audit sequence is the right upgrade over "append-only by convention." Append-only is a property of how you intend to write; a signed hash chain is a property you can verify. During an incident — especially a security one — "this reconstruction is tamper-evident" is a materially stronger statement than "we don't have a delete path." It's the same theme a couple of other commenters have pushed me on across this series: stop trusting conventions, make the guarantee something a test or a signature can prove.
Going to fold both into a revision — the metrics/logs cardinality split, and the replay-manifest-plus-two-phase model with a hash-chained trail. Genuinely some of the most useful feedback the series has gotten.
The point about treating the run, not the request, as the unit of observability is exactly right. If you only instrument tool latency, you miss the agent behaviors that matter most in production: thrashing, duplicate turns, tool oscillation, and retries that look like progress.
That is a big part of why we built agent-inspect around local-first execution traces and tool-call receipts for TypeScript agent workflows. Curious whether your audit log and trace model share the same run id end to end, or whether you join them later during incident triage?